High CPU，once per the configured -retentionPeriod about victoriametrics HOT 6 OPEN

Hello, could you please plot graph with following expression for the problematic period:
rate(vm_missing_metric_names_for_metric_id_total) > 0

from victoriametrics.

@f41gh7 OK，Here are the results. From the chart, there is no data returned

from victoriametrics.

Please help to review this preheating issue. One hour before our expiration date, we encountered the same problem again

from victoriametrics.

from victoriametrics.

from victoriametrics.

Hello @mmkkoo !

From the "The vmagent pending starts appearing at 11 o'clock and disappears at exactly 12 o'clock！！！" screenshot it looks like during normal operation the persistent queue isn't empty as well. It is a sign of vmagent not being able to forward data to remote destination fast enough. So it is very likely that even before indexdb rotation your cluster doesn't have enough resources for processing the current workload. I'd suggest to pay attention to this and read https://docs.victoriametrics.com/troubleshooting/#slow-data-ingestion

Update.
Sorry, didn't notice that persistent queue started to accumulate only hour before the rotation.
Could you provide screenshots of the following panels on the 24h time range including the rotation period:

1. vmstorage disk writes/reads and write/read calls
2. vmstorage cpu&memory usage
3. vmstorage merge speed
4. slow inserts

Thanks!

from victoriametrics.