Parca's own memory usage is close to node's capacity about parca HOT 4 CLOSED

Thank you, @asubiotto, for the detailed explanation. This should give us enough details for our testing. I will close the issue as I realized it probably belong to discussions.

Are you running the Parca Agent at all?

For the moment, we can't run parca-agent, since it requires having privileged access to the node (refer to the "details" section of the issue). In our context, to deploy a pod that is allowed privilege escalation, will require some extra "paper-work", thus we're delaying this part of testing for now.

from parca.

Hi @narqo,

The memory usage is linear with the number of targets. The memory usage you observe is expected, as it reflects the "active memory" size of the database, which grows up until a preconfigured limit before rotating the data and starting over (what you see at 16:30 in your graph). The spikes are compactions of this data since we need to decompress data at compaction time.

You can use --storage-active-memory=<number of bytes> to tune the maximum memory here, which will trade off how long you can view data for (if you don't have object storage enabled) which in your case is currently 3 hours for lower memory usage. The default active memory size is 512MiB, but there is likely memory we're not accounting for given that it looks like the block grew to 2.5GiB (memory that was rotated out). If you want to be thorough, feel free to grab some memory profiles (parca can be set up to scrape itself) during the block growth.

Parca can't be scaled out horizontally and there are currently no plans to add support for this. If you're looking for a more comprehensive continuous profiling setup (with horizontal scaling etc...) we do offer Polar Signals Cloud.

from parca.

Are you running the Parca Agent at all?

from parca.

My bad, sorry for missing that piece.

FYI another piece that uses memory is the metastore (where function names, etc. are stored), you can offload this to disk though by enabling persistence.

from parca.