Performance degradation after upgrade 1.93.6 => 1.97.1 about victoriametrics HOT 6 CLOSED

Hello @evkuzin!
It is likely related to more agressive background merge policy change in 1.97:

FEATURE: make background merge more responsive and scalable. This should help the following issues: 5190, 3425, 648.

Can you check whether number of background merges or their speed increased for big parts?

from victoriametrics.

@hagen1778
Oddly enough its 0 for last 24 hours on the new instance comparing to old version

from victoriametrics.

I can't correlate this graph with previous ones as they have different X axis. Could you correlate and see of performance degradation is connected to increased merge activities?

Oddly enough its 0 for last 24 hours on the new instance comparing to old version

Apparently, the mentioned change unblocked some of the merges which were postponed for historical data. Newer instances have less historical data, so they have less data to merge or no data at all.

from victoriametrics.

Same timeline for better visibility.
But yeah, regarding

Could you correlate and see of performance degradation is connected to increased merge activities?

See first post 1 and third picture.
It does correlate - when pending datapoints are bursting to 20 millions merge speed is 3 mil on the same node and when merge speed is down to 2 mil, pending datapoint's are down to a base threshold (by base I mean similar to old version instances)

from victoriametrics.

In the telegram chat it was mentioned that its okay to have ingestion rate <= x2 pending datapoints. But here we have x10 and this is bothering me.

from victoriametrics.

In the telegram chat it was mentioned that its okay to have ingestion rate <= x2 pending datapoints. But here we have x10 and this is bothering me.

I think the metric of pending datapoints became misleading with time. It doesn't illustrate if system is overloaded or not. Although, changes to background merges logic obsoleted the comment about x2 of ingestion rate. If you check the playground it shows pending rows 1Mil and ingestion rate 200k/s, so x5.

Let's sum up the issue:

1. It is expected that after update from older versions to 1.97+ VM will run additional merges. These merges should finish shortly and won't bother you again.
2. Pending samples metric is misleading and will be likely removed from the Grafana dashboard. If there are no other signs of performance degradation, you can ignore pending samples metric.

from victoriametrics.