messaging_service: add streaming/maintenance tenant to statement tenants about scylladb HOT 4 OPEN

Don't we already have an open issue on the Alternator TTL scanning losing its scheduling group? This looks like a dup.

from scylladb.

Don't we already have an open issue on the Alternator TTL scanning losing its scheduling group? This looks like a dup.

I guess we didn't, so let me explain this now:

As explained in #17609, when we implemented Alternator's TTL feature, which has a background thread scanning the database for expired rows, we did not implement any controller on how fast this scanning should proceed. Instead, we let the scanner run in the maintenance scheduling group, and the theory is that the scheduler will then ensure that this work cannot hurt the latency of the main workload (which runs in the query scheduling group), and also its throughput reduction will be limited to the amount of shares configured for the maintenance scheduling group.

However, experimentation showed that not all scanning work was done in the maintenance scheduling group. When node A runs the expiration scanner on node A on maintenance scheduling-group, and performs a read request to remote node B, that read request was supposed to also be run on maintenance scheduling-group. But it turns out that it didn't, and this is the bug reported in this issue: The node B ran the read request on the default request scheduling group, not the inherited one.

from scylladb.

I added an assignment to myself to write a test for this issue (@denesb has already provided a fix, #18729, but we also need a test). I think this is an important issue to fix (we now believe that users who used TTL encountered it almost immediately, and some performance problems we saw in tests were also caused by this), so I want this fix to go in.

from scylladb.

I wrote a test reproducing the bug and validating @denesb 's patch: #18757

from scylladb.