Comments (12)
Thank you Ruiwen, for the cardinality issue I have some comments on it:
This is very different from the existing implementation of pod_start_total_duration_seconds
. Waiting for @dashpole or others from sig-instrumentation to give some advice on the best way to record one-time per-pod metrics like this.
from kubernetes.
cc: @ruiwen-zhao for review.
from kubernetes.
/sig instrumentation
from kubernetes.
/sig node
from kubernetes.
Just to bring up previous discussion around metric cardinality, adding both pod name and node name to metric labels might be too much cardinality. We need to come up with a way to address this.
cc @SergeyKanzhelev @logicalhan @dashpole
from kubernetes.
Thank you Ruiwen, for the cardinality issue I have some comments on it:
-
Kubernetes already has some metrics from scheduler that include the pod name, namespace, and node name as metrics label:
-
kubernetes/pkg/kubelet/metrics/collectors/log_metrics.go
Lines 32 to 34 in 06b813f
Kubenetes has another metric kubelet_container_log_filesystem_used_bytes
that also use pod name and namespace as metrics labels.
- KSM also exports pod metrics in prometheus format and some metrics have pod name and namespace as metrics labels: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md.
from kubernetes.
from kubernetes.
@yujuhong Yes. The pod_start_total_duration_seconds
is a Distribution over all pods in the node, but newly added metric in this feature proposed to add a gauge metric that provides the exact startup time for a single pod to become ready.
@dashpole Hi David, could you please provide some insights here? Thanks!
from kubernetes.
A few questions to get the discussion started:
- Why a gauge instead of a histogram? A gauge is OK when looking at a single stream, or if you want to graph the average. But durations are often best represented by a histogram, as you can graph percentiles, or show a distribution. But if you graph a bunch of gauges, you will just see lots of lines on the graph, which isn't that helpful.
- Does this need to be in the kubelet? IIUC, this metric is produced by watching pods, and emitting a metric when it becomes ready for the first time. It doesn't need any special knowledge that the kubelet has, right?
- How long would the metric exist for? The startup will occur at the very beginning of the pod's life (in a single instant). Most pod-level metrics exist for the lifetime of the pod, but doing that would mean any aggregation would be less meaningful. Averaging the startup time of all currently-running pods in the cluster won't tell you if pod startup is currently slow. We could emit the metric for an arbitrary amount of time (e.g. 5 minutes), but that risks a scraper missing a pod entirely.
Bikeshedding: From the names, kubelet_pod_full_startup_duration_seconds
vs pod_start_total_duration_seconds
, I wouldn't know what the difference is. Would pod_ready_duration_seconds
or pod_first_ready_duration_seconds
be better?
from kubernetes.
@dashpole Hi David thank you for the comment.
Why a gauge instead of a histogram? A gauge is OK when looking at a single stream, or if you want to graph the average. But durations are often best represented by a histogram, as you can graph percentiles, or show a distribution. But if you graph a bunch of gauges, you will just see lots of lines on the graph, which isn't that helpful.
I want to use a gauge because I want to record the exact startup time of the pod, and it will allow users to know the exact time it takes for their pods to become ready to serve. With the pod-level metric, users could also group them together under the workload (e.g. deployment).
Does this need to be in the kubelet? IIUC, this metric is produced by watching pods, and emitting a metric when it becomes ready for the first time. It doesn't need any special knowledge that the kubelet has, right?
I use kubelet as kubelet will track the status of each pod in pod_startup_latency_tracker, and kubelet will watch for the status change of each pod. Also, kubelet is usually the first layer to process the pod status and it's a stable component (compared to other components in the cluster like kube-state-metrics which I usually see out-of-memory issue..) Do you have any recommendation for other places to add such metric?
How long would the metric exist for? The startup will occur at the very beginning of the pod's life (in a single instant). Most pod-level metrics exist for the lifetime of the pod, but doing that would mean any aggregation would be less meaningful. Averaging the startup time of all currently-running pods in the cluster won't tell you if pod startup is currently slow. We could emit the metric for an arbitrary amount of time (e.g. 5 minutes), but that risks a scraper missing a pod entirely.
For "Most pod-level metrics exist for the lifetime of the pod, but doing that would mean any aggregation would be less meaningful", can you provide more context here to help me understand? Thanks!
From the names, kubelet_pod_full_startup_duration_seconds vs pod_start_total_duration_seconds, I wouldn't know what the difference is. Would pod_ready_duration_seconds or pod_first_ready_duration_seconds be better?
pod_first_ready_duration_seconds
looks good to me!
from kubernetes.
Does this need to be in the kubelet? IIUC, this metric is produced by watching pods, and emitting a metric when it becomes ready for the first time. It doesn't need any special knowledge that the kubelet has, right?
Would something like kube-state-metrics more suitable for this?
https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/
from kubernetes.
/assign @JeffLuoo
/assign
/triage accepted
from kubernetes.
Related Issues (20)
- A DaemonSet pod environment variable did not inject service information HOT 9
- topologySpreadConstraints for availability zone in aws is not working as expected HOT 2
- API Server responds 200 OK, but not properly handling the request HOT 3
- Versioned feature gate lint error need to provide clear directions on what to do HOT 5
- Need emulation version guidance on how to perserve disabled feature gate tests for LockToDefault features HOT 6
- Failure cluster [c85e0cdc...] `Cluster size autoscaling` HOT 3
- setHostnameAsFQDN: true but the hostname is just pod name HOT 4
- Getting Unknown Field error for "podLogsDir" HOT 5
- Upgrade failed when using patches directory HOT 3
- `kubectl` handling of `node-role.kubernetes.io/control-plane` label encourages poor practice HOT 7
- PodScheduled status.conditions field does not have an entry in `managedFields` for Pod HOT 3
- Add validation that `--emulation-version=<version>` >= `DefaultKubeBinaryVersion`-3 HOT 7
- [Flaking Tests] Kubernetes e2e suite.[It] [sig-node] [Feature:GPUDevicePlugin] [Serial] Test using a Pod should run gpu HOT 14
- Failure cluster [00391c9f...] `ci-kubernetes-e2e-gke-prod-*-conformance` leaked to public bucket? HOT 3
- [Flaky Test] Kubernetes e2e suite.[It] [sig-cli] Kubectl client Kubectl validation should create/apply an invalid/valid CR with arbitrary-extra properties for CRD with partially-specified validation schema HOT 4
- [Flaky Test] k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.reconciler HOT 4
- Pods Not Scaling Up with HPA Despite CPU Utilization Exceeding Target HOT 3
- /sig <API Machinery> The kubeadm init can't start a healthy API server HOT 5
- terminationMessagePolicy: "File" not effective HOT 9
- WithRoutine loses stack information HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes.