Comments (5)
DirectXMan12
commented on August 23, 2024
2
Basically, both the relist interval and the rate interval have to be at least the Prometheus scrape interval. More specifically, the metrics relist interval only checks for metrics that are present in the time between now and the last relist time. I'll add something to the docs.
from prometheus-adapter.
DirectXMan12
commented on August 23, 2024
This usually happens if your Prometheus collection interval is longer than the rate or discovery intervals, or something similar.
from prometheus-adapter.
jsturtevant
commented on August 23, 2024
I went through the tutorial at https://github.com/stefanprodan/k8s-prom-hpa to double check the work and I am seeing the same errors, though the HPA is able to read the targets:
k get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
podinfo Deployment/podinfo 850m / 10 2 10 2 7d
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "podinfo-6b86c8ccc9-67t27",
"apiVersion": "/__internal"
},
"metricName": "http_requests",
"timestamp": "2018-06-26T22:57:28Z",
"value": "850m"
},
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "podinfo-6b86c8ccc9-nm7dd",
"apiVersion": "/__internal"
},
"metricName": "http_requests",
"timestamp": "2018-06-26T22:57:28Z",
"value": "850m"
}
]
}
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
Error from server (NotFound): the server could not find the metric http_requests for pods
k describe hpa podinfo
#truncated output
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetPodsMetric 24m (x43 over 45m) horizontal-pod-autoscaler unable to get metric http_requests: unable to fetch metrics from API: the server could not find the requested resource (get pods.custom.metrics.k8s.io *)
Warning FailedGetPodsMetric 4m (x8650 over 7d) horizontal-pod-autoscaler unable to get metric http_requests: unable to fetch metrics from API: the server could not find the metric http_requests for pods
My Prometheus collection interval:
- job_name: kubernetes-pods
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
and the custom-metrics-apiserver-deployment config:
- name: custom-metrics-apiserver
image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.2.0
args:
- /adapter
- --secure-port=6443
- --tls-cert-file=/var/run/serving-cert/serving.crt
- --tls-private-key-file=/var/run/serving-cert/serving.key
- --logtostderr=true
- --prometheus-url=http://running-ocelot-prometheus-server/
- --metrics-relist-interval=30s
- --rate-interval=5m
- --v=10
Do have something misconfigured?
from prometheus-adapter.
jsturtevant
commented on August 23, 2024
So I bumped my metrics-relist-interval to 90s and no longer seeing the error. I was using the Prometheus Helm Chart which has the pod scrape interval set to 1m and making sure both metrics-relist-interval and rate-interval were above the Prometheus scrape interval resolved the issue.
- --metrics-relist-interval=90s
- --rate-interval=5m
Is there any guidance on what are good values for metrics-relist-interval and rate-interval?
Would it be valuable to give an error message that might point folks in the right direction for this type of misconfiguration, if they believe the value should be listed?
from prometheus-adapter.
navathag
commented on August 23, 2024
@DirectXMan12 : i have my relist interval set to same as the prometheus scrape interval i still face the issue(metrics keep disappearing) sometimes i get the metrics while sometimes i get empty resources at "kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[]}"
`custom-metrics-apiserver-deployment ` config:
args:
- --secure-port=6443
- --tls-cert-file=/var/run/serving-cert/serving.crt
- --tls-private-key-file=/var/run/serving-cert/serving.key
- --logtostderr=true
- --prometheus-url=http://prom-global-svc.monitor:9090/prom/global
- --metrics-relist-interval=15s
- --v=10
- --config=/etc/adapter/config.yaml
And prometheus config:
global:
evaluation_interval: 1m
scrape_interval: 15s
scrape_timeout: 5s
Could you please suggest me on this
from prometheus-adapter.
Related Issues (20)
- How do i get the v0.11.2 docker image? HOT 2
- Documentation on how to directly query the adapter HOT 2
- Panic slice out of bands when querying namespace metric HOT 1
- Monitoring of prometheus-adapter metrics HOT 3
- Error while disabling TLS in server, setting secure-port = 0 throws error HOT 3
- Improve security hygiene and documentation HOT 3
- Unable to see Node Metrics - Error Metrics Missing CPU for node "XXX", skipping HOT 3
- Kubectl --raw reporting an unknown metric even though it shows up in the list of known metrics HOT 1
- Issue fetching external metric HOT 1
- Failed to get pods metric value: unable to get metric DCGM_FI_DEV_FB_USED_AVG: no metrics returned from custom metrics API HOT 5
- Prometheus adapter support for cross cluster autoscaling HOT 5
- Failed to pull image "quay.io/coreos/k8s-prometheus-adapter-amd64:v0.4.1" HOT 3
- Adding custom CRD for Prometheus Adapter rules HOT 1
- Correct Configuration Fails to Provide Expected Custom Metrics in EKS HOT 2
- Why the custom-metric-api yaml file lost after release-0.11? HOT 4
- external.metrics api does not goes up. only custom.metrics api. ='( HOT 2
- Custom metric does not get loaded into k8s/api not available HOT 3
- Clarify 'window' usage in configuration HOT 1
- Query too long HOT 1
- Kubernetes HPA unable to fetch custom metrics from Prometheus Adapter HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prometheus-adapter.