Local monitoring

If for example there is an agent running on each node, this agent can simply fetch the local healthz port: http://localhost:10248/healthz.

e.g with curl: curl -i http://localhost:10248/healthz

Remote monitoring

There is two way to fetch endpoints (metrics, healthz...), the both methods use HTTPS and a token (1.) and are exectued against the APISERVER or directly to Kubelet on each nodes.

The first one is mostly use with Prometheus and Kubernetes discovery (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config), it allows one to automatically discover the nodes and hence avoid the task of defining monitoring for each node.

The second one can be used in more traditional monitoring where one must configure each node to be checked.

Configuration

1. Create service account and permissions

Create a Service Account (monitoring) with a secondary Token (monitoring-secret-token) associated, it is best practice to not use the default created Token. Using a secondary token is easier for management. This Service Account can only fetch information about nodes and pods. The token will be used in https requests to Authenticate against the APIserver

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: monitoring
  namespace: kube-system
secrets:
- name: monitoring-secret-token
---
apiVersion: v1
kind: Secret
metadata:
  name: monitoring-secret-token
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: monitoring
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-clusterrole
  namespace: kube-system
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - pods
  verbs: ["get", "list"]
- nonResourceURLs: ["/metrics", "/healthz", "/healthz/*"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: monitoring-clusterrole-binding
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: monitoring-clusterrole
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: monitoring
  namespace: kube-system

e.g, how to export the token:

kubectl -n kube-system get secrets monitoring-secret-token -ojsonpath='{.data.token}' | base64 -d

This token can now be passed in headers in the form: "Authorization: Bearer $TOKEN"

2. Test token remotely

--> Choose a Kubernetes node

NODE="vm154162"

--> Get TOKEN with kubectl and APISERVER from configuration file

TOKEN=$(kubectl -n kube-system get secrets monitoring-secret-token -ojsonpath='{.data.token}' | base64 -d)
APISERVER=$(kubectl config view | grep server | cut -f 2- -d ":" | tr -d " ")

--> Fetch the Kubelet endpoints

curl -k $APISERVER/api/v1/nodes/$NODE/proxy/metrics --header "Authorization: Bearer $TOKEN"
curl -k $APISERVER/api/v1/nodes/$NODE/proxy/metrics/cadvisor --header "Authorization: Bearer $TOKEN"
curl -k $APISERVER/api/v1/nodes/$NODE/proxy/healthz --header "Authorization: Bearer $TOKEN"

--> Fetch the endpoints directly on Kubelet

curl -k https://$NODE:10250/metrics --header "Authorization: Bearer $TOKEN"
curl -k https://$NODE:10250/metrics/cadvisor --header "Authorization: Bearer $TOKEN"
curl -k https://$NODE:10250/heathz --header "Authorization: Bearer $TOKEN"

from doc-caasp.