Comments (7)
Haleygo
commented on May 28, 2024
Hello,
From the error logs, the caFile provided for those scrape jobs are invalid, could you check if you mounted the right files to vmagent?
2024-04-10T10:28:07.401Z warn VictoriaMetrics/lib/promscrape/scrapework.go:382 cannot scrape target "https://172.16.0.74:2379/metrics" ({endpoint="http-metrics",instance="172.16.0.74:2379",job="kube-etcd",namespace="kube-system",pod="etcd-sealos-run-master0000",service="victoria-metrics-k8s-stack-kube-etcd"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot perform request to "https://172.16.0.74:2379/metrics": Get "https://172.16.0.74:2379/metrics": tls: failed to verify certificate: x509: certificate signed by unknown authority
2024-04-10T10:28:09.071Z warn VictoriaMetrics/lib/promscrape/scrapework.go:382 cannot scrape target "https://172.16.0.74:10257/metrics" ({endpoint="http-metrics",instance="172.16.0.74:10257",job="kube-controller-manager",namespace="kube-system",pod="kube-controller-manager-sealos-run-master0000",service="victoria-metrics-k8s-stack-kube-controller-manager"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot perform request to "https://172.16.0.74:10257/metrics": Get "https://172.16.0.74:10257/metrics": tls: failed to verify certificate: x509: certificate is valid for localhost, localhost, not kubernetes
from victoriametrics.
bxy4543
commented on May 28, 2024
kubectl exec -n vm -it vmagent-victoria-metrics-k8s-stack-554cd7779b-6ch5n -c vmagent -- sh
/ # ls -LR /etc/vmagent*
/etc/vmagent:
config config_out
/etc/vmagent/config:
vmagent.yaml.gz
/etc/vmagent/config_out:
vmagent.env.yaml
/etc/vmagent-tls:
certs
/etc/vmagent-tls/certs:
is it here? It looks like the tls certificate is empty.
from victoriametrics.
plutocholia
commented on May 28, 2024
kubectl exec -n vm -it vmagent-victoria-metrics-k8s-stack-554cd7779b-6ch5n -c vmagent -- sh / # ls -LR /etc/vmagent* /etc/vmagent: config config_out /etc/vmagent/config: vmagent.yaml.gz /etc/vmagent/config_out: vmagent.env.yaml /etc/vmagent-tls: certs /etc/vmagent-tls/certs:is it here? It looks like the tls certificate is empty.
I think she meant /var/run/secrets/kubernetes.io/serviceaccount/ca.crt which is provided in the spec.endpoints.[*].tlsConfig scope of VMServiceScrape resource.
from victoriametrics.
bxy4543
commented on May 28, 2024
kubectl describe clusterrole monitoring:vmagent-cluster-access-victoria-metrics-k8s-stack
Name: monitoring:vmagent-cluster-access-victoria-metrics-k8s-stack
Labels: app.kubernetes.io/component=monitoring
app.kubernetes.io/instance=victoria-metrics-k8s-stack
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=vmagent
app.kubernetes.io/version=v1.96.0
helm.sh/chart=victoria-metrics-k8s-stack-0.18.11
managed-by=vm-operator
Annotations: meta.helm.sh/release-name: victoria-metrics-k8s-stack
meta.helm.sh/release-namespace: vm
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
configmaps [] [] [get list watch]
endpoints [] [] [get list watch]
namespaces [] [] [get list watch]
nodes/metrics [] [] [get list watch]
nodes/proxy [] [] [get list watch]
nodes [] [] [get list watch]
pods [] [] [get list watch]
secrets [] [] [get list watch]
services [] [] [get list watch]
endpointslices.discovery.k8s.io [] [] [get list watch]
ingresses.extensions [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
[/metrics/resources] [] [get]
[/metrics] [] [get]
registry.image.openshift.io/metrics [] [] [get]
routers.image.openshift.io/metrics [] [] [get]
registry.route.openshift.io/metrics [] [] [get]
routers.route.openshift.io/metrics [] [] [get]
[/metrics/resources] [] [list]
[/metrics] [] [list]
[/metrics/resources] [] [watch]
[/metrics] [] [watch]
Yes, thanks for the reply, it is a vm scrape job automatically created by installing vm stack through helm. Why is there tls: bad certificate, and how can I fix this problem now?
from victoriametrics.
Haleygo
commented on May 28, 2024
Why is there tls: bad certificate, and how can I fix this problem now?
Because this cert is invaild for etcd, /var/run/secrets/kubernetes.io/serviceaccount/ca.crt is just the default path of SA token, it's just there to be an example i guess(maybe we should remove it if it caused misunderstanding)).
If you want to access etcd using https, you need to create etcd client certificates, mount them to vmagent and change the cert path in VMServiceScrape here:
spec:
endpoints:
- port: http-metrics
scheme: https
tlsConfig:
caFile: <you-etcd-cert-path>
from victoriametrics.
bxy4543
commented on May 28, 2024
Thanks for the reply, I'll try creating and mounting the certificate manually. @Haleygo
from victoriametrics.
bxy4543
commented on May 28, 2024
it's just there to be an example i guess(maybe we should remove it if it caused misunderstanding)).
Indeed, this caused some misunderstandings for me.
from victoriametrics.
Related Issues (20)
- vmauth per-user metrics can cause high memory usage in the long term HOT 2
- vmbackup cannot open a snapshot that it just created HOT 7
- How to drop selected metrics received at victoriametrics (single) from api/v1/write remoteWrite vmagents? HOT 4
- Scrape CRD's support by single-node VictoriaMetrics HOT 2
- victorialogs crash HOT 4
- ERROR: 422 on query with binary operation and keep_metric_names HOT 4
- vmui requests are not automatically quoted HOT 1
- Graphite queries not calculated correctly HOT 1
- how to set custom tag in filebeat output.elasticsearch _stream_fields HOT 2
- vmagent k8s target discovery is too slow HOT 4
- Data integrity problem after the vmstorage breaks down HOT 3
- victoria logsQL sort unexpected result. HOT 3
- How to search log as quickly as possible while writing log in client side HOT 2
- Attempts to configure VM for small memory footprint don't yield expected results HOT 2
- Maximum advised storage capacity for a vmstorage instance? HOT 1
- VictoriaMetrics Datadog APM/Trace Agent URL integration HOT 3
- MetricsQL: Document stalness marker differences between `default_rollup` and `last_over_time` HOT 2
- How to properly setup remoteWrite credentials in vmagent HOT 8
- VictoriaLogs UI sometimes loses log records in Firefox HOT 8
- Teach -httpAuth.username to read content of a file HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from victoriametrics.