roachtest: backup-restore/mixed-version failed about cockroach HOT 11 OPEN

23.1: could not find valid split key

from cockroach.

Looks like the replica that returned the error to AddSSTable was on n2, which was on 22.2 at the time (this test stresses upgrades, so since this was testing 23.1 it started on 22.2 to then do the upgrade and the node that returned this error was on 22.2 at the time).

The range that returned it is in table/183 or something, which isn't the jobs table.

08:14:16 mixed_version_backup.go:1419: not setting any custom cluster settings (using defaults) means this was off-the-shelf and not some randomly weird config.

from cockroach.

r634 isn't looking pretty:

 "stats": {
          "contains_estimates": 0,
          "last_update_nanos": 1710062613391780670,
          "intent_age": 0,
          "gc_bytes_age": 426629484084,
          "live_bytes": 16415,
          "live_count": 1,
          "key_bytes": 98166,
          "key_count": 1,
          "val_bytes": 134125670,
          "val_count": 8180,

from cockroach.

this is 22.2, which is EOL'ed, and is only being tested here since 23.1 needs to remain compatible with it during upgrades, but we're not going to go dig into and fix 22.2 bugs, so I might suggest we just close this as unactionable? But I'll let KV weigh in too.

from cockroach.

We're backpressuring an AddSSTable when the range is only 134MB large. Is this because 22.2 had a 64MB max_range_size? Assuming this is the case (I'll check), it doesn't look like anything is going wrong in KV.

The backpressure is throwing an error because there's only a single key in the range, which has 8180 versions. @dt do you know what's in this /Table/186 and whether this is expected?

from cockroach.

Is this because 22.2 had a 64MB max_range_size?

Yes, this is the case: range_max_bytes: 67108864.

from cockroach.

I think we can close this as unactionable since this is a known 22.2 behavior we wont' be changing ?

from cockroach.

So the remaining question is whether the composition of r643 (/Table/18{3/1/99-6/1/4}) is expected:

          "key_bytes": 98166,
          "key_count": 1,
          "val_bytes": 134125670,
          "val_count": 8180,

Is the single-key, many-16KB-version surprising?

from cockroach.

The init command is cockroach workload init bank {pgurl:1-4} --payload-bytes 16384 --ranges 0 --rows 100, so that explains the version size.

Do we expect this to write 8180 versions to the same key?

from cockroach.

Oh, I suspect that this is because we ran the bank workload after, which updates these rows and writes new versions. We see errors like:

E240310 09:23:29.618860 1 workload/cli/run.go:550  [-] 2614  pq: split failed while applying backpressure to Put [/Table/183/1/4/0,/Min), EndTxn(parallel commit) [/Table/183/1/4/0], [txn: 9904f87b] on range r690:/Table/183/1/{4-5} [(n3,s3):4, (n4,s4):2, (n1,s1):3, next=5, gen=94]: could not find valid split key

We have 16KB sized rows (the maximum option in bankPossiblePayloadBytes), only 100 of them (the minimum option in bankPossibleRows), then run a workload that continually updates these rows. I would expect us to hit this in never versions of CRDB as well, as this will quickly fill up a range ((512MB*2) / (16KB * 3500qps / (100/2)) = 15m).

Removing the release-blocker label and handing this to OwnerDisasterRecovery to determine how they want to deflake the test.

from cockroach.

Oooff, that random combo of row count and size is sneaky. Thanks for the thorough investigation! I've opened #120462

from cockroach.