The memory usage keeps increasing after starting and finally results in an OOM error about victoriametrics HOT 3 OPEN

Hello @escapekyg

Could you elaborate on the deployment architecture you're using?
Based on IP addresses in command-line flags it seems like there are 3 machines which host all three components, is it correct?

Also, seems like you're using quite outdated release, could you try upgrading to more recent release and check if the issue persists?
Recent releases include a lot of bugfixes and improvements including ones related to memory usage.

from victoriametrics.

Hello @escapekyg

Could you elaborate on the deployment architecture you're using? Based on IP addresses in command-line flags it seems like there are 3 machines which host all three components, is it correct?

Also, seems like you're using quite outdated release, could you try upgrading to more recent release and check if the issue persists? Recent releases include a lot of bugfixes and improvements including ones related to memory usage.

Hello @zekker6
Thanks for you reply.
Yes, you're right, it's a cluster with 3 nodes.
Which version do you recommend upgrading to? Can version 1.99.0 be directly compatible with 1.85.3?

from victoriametrics.

Yes, you're right, it's a cluster with 3 nodes.

Such deployment is usually not recommended because each component tries to use up to full memory size for the operations.
In case all components are running on the same machine please make sure to adjust memory.allowedPercent value.
By default the value is 60% which means that component will try to fit into 60% leaving 40% for OS level caches.
In case all components are hosted on single machine each component will try to use up to 60% percent of overall memory which can lead to OOM issues.
Depending on the deployment type it might be possible to limit amount of memory available to the processes which will also work.

Which version do you recommend upgrading to? Can version 1.99.0 be directly compatible with 1.85.3?

Ideally I would recommend to wait for 1.100.0 to be released(expected early this week) as 1.99.0 had a bug which could lead to inconsistent data read(see #5959).

It is possible to upgrade directly to the latest releases, please note that there were releases with backward incompatible changes which means that it will not be possible to roll back to versions older than these.
The releases are:

• v1.90.0 - https://docs.victoriametrics.com/changelog_2023/#v1900
• v1.92.0 - https://docs.victoriametrics.com/changelog_2023/#v1920

Also note that there were other deprecation warning in betwee v1.85.3 and current releases which deprected metrics and flags used by the VictoriaMetrics and its' components, you can find all changes in the changelog - https://docs.victoriametrics.com/changelog/

from victoriametrics.