Giter Club home page Giter Club logo

k8s-ephemeral-storage-metrics's Introduction

K8s Ephemeral Storage Metrics

License: MIT Actions Status Artifact Hub

A prometheus ephemeral storage metric exporter for pods, containers, nodes, and volumes.

This project was created to address lack of monitoring in Kubernetes

This project does not monitor CSI backed ephemeral storage ex. Generic ephemeral volumes

main image

Helm Install

helm repo add k8s-ephemeral-storage-metrics https://jmcgrath207.github.io/k8s-ephemeral-storage-metrics/chart
helm repo update
helm upgrade --install my-deployment k8s-ephemeral-storage-metrics/k8s-ephemeral-storage-metrics

Values

Key Type Default Description
affinity object {}
deploy_type string "Deployment" Set as Deployment for single controller to query all nodes or Daemonset
dev object {"enabled":false,"grow":{"image":"ghcr.io/jmcgrath207/k8s-ephemeral-storage-grow-test:latest","imagePullPolicy":"IfNotPresent"},"shrink":{"image":"ghcr.io/jmcgrath207/k8s-ephemeral-storage-shrink-test:latest","imagePullPolicy":"IfNotPresent"}} For local development or testing that will deploy grow and shrink pods and debug service
image.imagePullPolicy string "IfNotPresent"
image.repository string "ghcr.io/jmcgrath207/k8s-ephemeral-storage-metrics"
image.tag string "1.11.2-rc01"
interval int 15 Polling node rate for exporter
kubelet object {"insecure":false,"readOnlyPort":0,"scrape":false} Scrape metrics through kubelet instead of kube api
log_level string "info"
max_node_concurrency int 10 Max number of concurrent query requests to the kubernetes API.
metrics object {"adjusted_polling_rate":false,"ephemeral_storage_container_limit_percentage":true,"ephemeral_storage_container_volume_limit_percentage":true,"ephemeral_storage_container_volume_usage":true,"ephemeral_storage_node_available":true,"ephemeral_storage_node_capacity":true,"ephemeral_storage_node_percentage":true,"ephemeral_storage_pod_usage":true} Set metrics you want to enable
metrics.adjusted_polling_rate bool false Create the ephemeral_storage_adjusted_polling_rate metrics to report Adjusted Poll Rate in milliseconds. Typically used for testing.
metrics.ephemeral_storage_container_limit_percentage bool true Percentage of ephemeral storage used by a container in a pod
metrics.ephemeral_storage_container_volume_limit_percentage bool true Percentage of ephemeral storage used by a container's volume in a pod
metrics.ephemeral_storage_container_volume_usage bool true Current ephemeral storage used by a container's volume in a pod
metrics.ephemeral_storage_node_available bool true Available ephemeral storage for a node
metrics.ephemeral_storage_node_capacity bool true Capacity of ephemeral storage for a node
metrics.ephemeral_storage_node_percentage bool true Percentage of ephemeral storage used on a node
metrics.ephemeral_storage_pod_usage bool true Current ephemeral byte usage of pod
nodeSelector object {}
podAnnotations object {}
pprof bool false Enable Pprof
prometheus.enable bool true
prometheus.release string "kube-prometheus-stack"
prometheus.rules.enable bool false Create PrometheusRules firing alerts when out of ephemeral storage
prometheus.rules.predictFilledHours int 12 How many hours in the future to predict filling up of a volume
serviceMonitor object {"additionalLabels":{},"metricRelabelings":[],"podTargetLabels":[],"relabelings":[],"targetLabels":[]} Configure the Service Monitor
serviceMonitor.additionalLabels object {} Add labels to the ServiceMonitor.Spec
serviceMonitor.metricRelabelings list [] Set metricRelabelings as per https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.RelabelConfig
serviceMonitor.podTargetLabels list [] Set podTargetLabels as per https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitorSpec
serviceMonitor.relabelings list [] Set relabelings as per https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.RelabelConfig
serviceMonitor.targetLabels list [] Set targetLabels as per https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitorSpec
tolerations list []

Prometheus alert rules

To prevent from multiple kind of alerts being fired for a single container or emptyDir volume when both prometheus.enable and prometheus.rules.enable are on, add the following inhibition rules to your Alert Manager config:

- source_matchers:
    - alertname="EphemeralStorageVolumeFilledUp"
  target_matchers:
    - severity="warning"
    - alertname="EphemeralStorageVolumeFillingUp"
  equal:
    - pod_namespace
    - pod_name
    - volume_name
- source_matchers:
    - alertname="ContainerEphemeralStorageUsageAtLimit"
  target_matchers:
    - severity="warning"
    - alertname="ContainerEphemeralStorageUsageReachingLimit"
  equal:
    - pod_namespace
    - pod_name
    - exported_container

Contribute

Start minikube

make new_minikube

Run locally

make deploy_local

Run locally with Delve Debug

make deploy_debug

Then connect to localhost:30002 with delve or your IDE.

Run e2e Test

make deploy_e2e

Debug e2e

make deploy_e2e_debug

Then run a debug against deployment_test.go

License

This project is licensed under the MIT License. See the LICENSE file for more details.

k8s-ephemeral-storage-metrics's People

Contributors

jmcgrath207 avatar fusakla avatar miminar avatar bhuism avatar gcuisinier avatar janlo avatar matzegebbe avatar mcanevet avatar tobmad avatar trashadewan avatar canidam avatar lynxstarshine avatar mchtech avatar jicki avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.