High requests and churn rate when writing metrics for /api/v1/import/prometheus in Pushgateway format about victoriametrics HOT 5 OPEN

Also, during the investigation, I've found out that Prometheus was scraping metrics from Pushgateway with an interval of 30secs, but I was unable to find a way to set some kind of interval for /insert/{}/prometheus/api/v1/import/prometheus in the docs. Is this not possible with vminsert?

You are a bit confusing push/pull approaches in this case, you should read this docs about difference between push model and pull model: https://docs.victoriametrics.com/keyconcepts/#write-data

In brief

There are two models of data collection for monitoring:

1. push model - when the application sends data to the monitoring system on its own with the frequency it requires
2. pull model - when the monitoring system polls (scrape) applications at a specified interval, writing data to DB

Prometheus supports pull model, but Pushgateway can adapt push model to pull model for prometheus:

graph LR;
    app1["App 1 with /metrics endpoint"]
    app2["App 2 with /metrics endpoint"]
    app3["App 3 with push"]
    app4["App 2 with push"]
    prom[(Prometheus)]
    pgw[[Pushgateway]]
    prom-- scrape app1 metrics -->app1;
    prom-- scrape app2 metrics -->app2;
    prom-- scrape app1 and app2 metrics -->pgw;
    app3-- push app3 metrics -->pgw;
    app4-- push app4 metrics -->pgw;

VictoriaMetrics (single or cluster) supports push model, but vmagent can adapt push model to pull model for VM:

---
title: First option
---
graph LR;
    app1["App 1 with /metrics endpoint"]
    app2["App 2 with /metrics endpoint"]
    app3["App 3 with push"]
    app4["App 2 with push"]
    vmdb[(VMDB)]
    vmagent[[vmagent]]
    vmagent-- scrape app1 metrics -->app1;
    vmagent-- scrape app2 metrics -->app2;
    app3-- push app3 metrics -->vmdb;
    app4-- push app4 metrics -->vmdb;
    vmagent -- push app1 and app2 metrics -->vmdb;

---
title: Second option
---
graph LR;
    app1["App 1 with /metrics endpoint"]
    app2["App 2 with /metrics endpoint"]
    app3["App 3 with push"]
    app4["App 2 with push"]
    vmdb[(VMDB)]
    vmagent[[vmagent]]
    vmagent-- scrape app1 metrics -->app1;
    vmagent-- scrape app2 metrics -->app2;
    app3-- push app3 metrics -->vmagent;
    app4-- push app4 metrics -->vmagent;
    vmagent -- push all metrics -->vmdb;

As you can see, vminsert (VMDB on scheme) does not control how often data is sent to it, this is done on the application side - as you wrote it should be done in the application/library parameters.

from victoriametrics.

Hey, @urupaud.
Did you check data in Cardinality explorer, can you investigate with it what leads high churn rate?
Maybe you had some relabelings on Prometheus side and didn't setup them on VM.

from victoriametrics.

Hey @Amper

Thanks for the info, I will check it.

Also, during the investigation, I've found out that Prometheus was scraping metrics from Pushgateway with an interval of 30secs, but I was unable to find a way to set some kind of interval for /insert/{}/prometheus/api/v1/import/prometheus in the docs. Is this not possible with vminsert?

Prometheus-net libarary is writing metrics with an interval of 1 second, so I guess now the applications that use this library write to vminsert every second instead of with an interval of 30 seconds just like with Prometheus.

Thanks

from victoriametrics.

we can try to override the interval in Prometheus-net libarary, but just wanted to check with you guys before going with it

from victoriametrics.

Thanks a lot for the detailed explanation @Amper 🙏

from victoriametrics.