vmagent job flapping up/down with no errors about victoriametrics HOT 9 OPEN

Hello,
how do you scrape vmagent here, could you attach the scrape job config?
And how many vmagent instances showed in http://vmagent:8429/targets page, do them changed a lot?

from victoriametrics.

@Haleygo thanks for you response! as I posted vmagent has 6 shards, unfortunately it's not possible to post all the targets from /targets endpoint, since each vmagent only scrapes its part of targets. Overall count of targets is about 9000 (from vmagent dashboard) and they really do change a lot. It is a dev cluster where any developer can create his own test environment. I'm ready to post anything you need to be able to help

from victoriametrics.

No problem, I'm only concern about the vmagent scrape job.
How do you scrape vmagent, using service discovery or static configs? How many instances shown under the vmagent scrape job, are them all healthy? If not, there should be reason on /targets page as well.

from victoriametrics.

we use vm-stack chart for out setup

vmagent via VMServiceScrape (kubernetes sd)

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
...
  labels:
....
    managed-by: vm-operator
  name: vmagent-vm-stack
  namespace: monitoring
  ownerReferences:
  - apiVersion: operator.victoriametrics.com/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: VMAgent
    name: vm-stack
    uid: dff72d61-3e36-4968-bd25-9f48cac8f339
  resourceVersion: "1104257"
  uid: 642c3ef7-fca1-4cee-b146-ad54260bab09
spec:
  endpoints:
  - attach_metadata: {}
    path: /metrics
    port: http
  namespaceSelector: {}
  selector:
    matchExpressions:
    - key: operator.victoriametrics.com/additional-service
      operator: DoesNotExist
    matchLabels:
      app.kubernetes.io/component: monitoring
      app.kubernetes.io/instance: vm-stack
      app.kubernetes.io/name: vmagent
      managed-by: vm-operator

I can't get what should I post here. Can you provide exact command or query, pls?

UI targets page is really huge due to targets count near 9000

we definitely have unhealthy targets, actually hundreds of them, is it related to vmagent's job up/down flapping?

from victoriametrics.

UI targets page is really huge due to targets count near 9000

Targets can be filtered on ui like this, please share for vmagent job

from victoriametrics.

Thanks! I can see now. Sometimes it is down

sometime it is up

no idea why it's happening

from victoriametrics.

@Haleygo can you please tell is there any recommendation to fix this? I've tried to increase scrape timeout with no luck, what else context deadline exceeded could mean?

from victoriametrics.

I can see now. Sometimes it is down

Looks like you have a lot of scrape failures(3062/4783; 2733/4846), they could be caused by resources pressure or slow network, could you also check vmagent's cpu usage?

from victoriametrics.

@Haleygo thanks! VMagent shows extremely high CPU usage. Nodes almost for 100% CPU usage. I didn't expect scrape failures could be caused due to high CPU usage. How to determine a scrape failure is exactly because of lack of resources?

from victoriametrics.