After an outage, i've opened an issue with @angelbarrera92 on the node-exporter project, to implements the machine-id collector also : prometheus/node_exporter#1546 .
Long story short, basically already exists a metacollector that can be adapter for our usecase , and this is called the textfile collector.
As said in the issue referred above, it can be configured for this purpose by following these steps:

• add the following parameter to the node-exporter binary : ---collector.textfile.directory=/var/textfile-dir where /var/textfile-dir can be whatever dir you want

• create inside the directory used (in our example /var/textfile-dir) a file machine-id.prom with the following content: systemd_machine_id{id="abc"} 1 where i guess "abc" will be the label printed in the prometheus time-series db.

I think it just work out-of-the-box, probably is necessary to adjust the id key, but nothing else.
After this will be mandatory add a ServiceMonitor to identify that every machine-id must be unique in the cluster

Testing the upgrade to v2.0.0 we found that ServiceMonitors definition for kube-scheduler and kube-controller-manager are using the wrong name for the port. They point to the port https-metrics when the service defines the port as just metrics.

Hi!
during a setup with module version 2.0.1 we've found out that node-exporter 1.3.1 have some issues when node metrics are requested (grafana dashboards that ask for them and kubectl top nodes).
Once node-exporter is upgraded to 1.5.0 (at least), everything has back to work.

We are facing the same problem in different cluster, every single one of the has begun having this alert after the kubernetes upgrade to 1.21, and going even to the 1.23 the alert is still there.

error faced and symptoms
the log of the pod are full of this: node-exporter-gsg5p kube-rbac-proxy http: proxy error: context canceled
prometheus see pod as down and start alerting for targetdown: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job, namespace, service)) > 10`

module version
the version for the monitoring is usually the v1.14.3.

kubenertes version
from 1.21 upwards

test done for solution
removed the runasnonroot true,
removed any limits on the resources
checked if there were throttling problems
edited scrapeTimeout and interval

solution that has worked
remove the container kube-rbac-proxy from the node-exporter pod

https://github.com/sighup-io/fury-kubernetes-monitoring/blob/master/katalog/kube-state-metrics/README.md needs to reflect the image that is running

Since the grafana service changes, we need to delete it before upgrading. We could add it in the notes, in addition to delete the grafana deployment also delete the service before apply the new version.

Add E2E for K8s v1.24.0 in CI

Evaluate adding Node Problem Detector to the monitoring module. References:

https://kubernetes.io/docs/tasks/debug/debug-cluster/monitor-node-health/
https://github.com/kubernetes/node-problem-detector

The CPUThrottlingHigh alert is used to notify if the Kubelet is throttling the pod's CPU usage for more than 25% of time in the last 5 minutes. While this might indicate wrong CPU limits there are particular class of pods (e.g. node-exporter) that are more subject to throttling than other.

My proposal is either to drop this alert or to increase the threshold to, at least, 50-75% given the narrow time window.

What's your opinion about this?

I am also attaching some issues from upstream projects.

Refs:

• prometheus-operator/kube-prometheus#72
• kubernetes-monitoring/kubernetes-mixin#108
• kubernetes/kubernetes#67577

we spent some day working on a prometheus installation and we found that we "leave" too many labels for the metrics, some are even duplicates.

Proposal
by adding the following configuration we can remove some of the label which have high cardinality and aren't used in dashboards or rules from the various FURY modules

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
spec:
  writeRelabelConfigs:
    - job_name: id-label
        metric_relabel_configs:
        - source_labels: [__name__]
          regex: 'id'
          action: drop
    - job_name: container_id-label
        metric_relabel_configs:
        - source_labels: [__name__]
          regex: 'container_id'
          action: drop
    - job_name: le-label
        metric_relabel_configs:
        - source_labels: [__name__]
          regex: 'le'
          action: drop
    - job_name: service-name-label
        metric_relabel_configs:
        - source_labels: [__name__]
          regex: 'service-name'
          action: drop
    - job_name: address-label
        metric_relabel_configs:
        - source_labels: [__name__]
          regex: 'address'
          action: drop

In order to further save memory it will be a good idea to choose between the service and job label, since those two are replicated but somewhere its used job, somewhere else is used service.
By choosing which one to use everywhere we can drastically reduce the cardinality by removing the other as per the example posted before.

What do you think about this?

We need to test the roles on an ubuntu 22 scenario

Please considering adding Pixie as a package to the monitoring module:
https://github.com/pixie-io/pixie

Starting from 1.17, kubeadm is not exposing anymore kube-controller-manager and kube-scheduler metrics over the insecure port (respectively TCP/10252 and TCP/10251) this makes our current ServiceMonitor configuration unable to properly scrape such targets.

Refs:

• kubernetes/kubernetes#92720
• kubernetes/kubeadm#2202

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#no-really-you-must-read-this-before-you-upgrade

Any Prometheus queries that match pod_name and container_name labels (e.g. cadvisor or kubelet probe metrics) should be updated to use pod and container instead. pod_name and container_name labels will be present alongside pod and container labels for one transitional release and removed in the future. (#69099, @ehashman)

We had some good experiences in one of our customers using Karma as a centralized AlertManager dashboard for several clusters.

Let's consider adding it to out monitoring module

The prometheus adapter currently does not create external and custom metrics for HPA

In the implementation, we only define the possibility of using the most common metrics based on CPU and mem. Still, in the readme, we describe the possibility of using aggregate metrics based on other resource types Kubernetes based(custom) and external or not of type kubernetes(external) :
The Prometheus adapter provides an implementation of Kubernetes
resource metrics,
custom metrics, and
external metrics APIs.

Kubernetes Version: v1.24.8
fury-kubernetes-monitoring: v2.0.0
furyctl: Furyctl version 0.8.0
Furyfile.yml

versions:
 monitoring: v2.0.0
bases:
    - name: monitoring/prometheus-operator
    - name: monitoring/prometheus-operated
    - name: monitoring/alertmanager-operated
    - name: monitoring/kube-state-metrics
    - name: monitoring/grafana
    - name: monitoring/node-exporter
    - name: monitoring/kubeadm-sm

In the default configuration prometheus-operated has insufficient permissions in its ClusterRole.
https://github.com/sighupio/fury-kubernetes-monitoring/blob/v2.0.0/katalog/prometheus-operated/clusterRole.yaml

Logs from prometheus pod:

ts=2022-12-01T12:50:47.605Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"monitoring\""
ts=2022-12-01T12:50:47.605Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"monitoring\""
ts=2022-12-01T12:50:47.605Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"monitoring\""
ts=2022-12-01T12:50:47.605Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"monitoring\""
ts=2022-12-01T12:50:47.605Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"monitoring\""
ts=2022-12-01T12:50:47.605Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"monitoring\""
ts=2022-12-01T12:50:47.605Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"monitoring\""
ts=2022-12-01T12:50:47.605Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"monitoring\""
ts=2022-12-01T12:50:47.607Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Pod: unknown (get pods)"
ts=2022-12-01T12:50:47.607Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Pod: unknown (get pods)"

After updating the prometheus-operated/clusterRole.yaml with below file it started working corectly

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - services
  - pods
  - endpoints
  verbs:
  - get
  - list
  - watch

We could start unit testing our prometheus' rules using promtool as described here

Prometheus 2.7 introduced a new feature that enables size-based retention alongside the usual time-based one. This option is supported by Prometheus Operator since v0.31.0 through the retentionSize field in the Prometheus CR. Enabling this option would allow to avoid having the Prometheus storage filling-up thus causing downtime of the monitoring subsystem.

Ref:

• https://www.robustperception.io/configuring-prometheus-storage-retention

The examples provided in the examples folder are outdated, they won't work with the latest versions of Furyctl and of the module.

At least those for Alert Manager (the ones I checked).

I noticed that kube-state-metrics 1.9.5 is supporting only Kubernetes v1.16.x.

On a 1.15.x cluster the only incompatibility is about ValidatingWebhookConfiguration, to something major but I think we should update the compatibility matrix accordingly.

Refs:

• kubernetes/kube-state-metrics#1093
• https://github.com/kubernetes/kube-state-metrics#compatibility-matrix

The KubeletDown alert is always firing.

The alert's expression is absent(up{job="kubelet",metrics_path="/metrics"} == 1) but since the up time series is lacking the metrics_path label, this makes the absent function always return a 1-element vector thus making the alert trigger.

I will provide a PR to fix this.

Currently, all dashboards deployed by default end up inside the "Default" folder in Grafana's UI, ending up being a little messy to discover the dashboards or find a specific one.

It would be better if we organize the dashboards in folders. We could, for example, have a folder for each KFD module or group them with some other criteria.

Hello guys! During a KFD upgrade, I noticed this weird behavior using the current AlertManagerConfig resources.

Inspecting the /etc/alertmanager/config/alertmanager.yaml inside alertmanager pods I get this:

route:
  routes:
  - receiver: monitoring/deadmanswitch/healthchecks
    matchers:
    - alertname="DeadMansSwitch"
    - namespace="monitoring"
    continue: true
  - receiver: monitoring/infra/infra
    group_by:
    - alertname
    matchers:
    - namespace=~"calico-api|calico-system|cert-manager|gatekeeper-system|ingress-nginx|kube-system|logging|monitoring|pomerium|tigera-operator|vmware-system-csi"
    - alertname=~"TargetDown|KubePersistentVolumeFillingUp|NginxIngressCertificateExpiration|NginxIngressLatencyTooHigh|NginxIngressFailedReload|NginxIngressFailureRate|NginxIngressDown"
    - namespace="monitoring"
    continue: true
  - receiver: monitoring/k8s/k8s
    group_by:
    - alertname
    matchers:
    - alertname=~"NodeNetworkInterfaceFlapping|NodeClockSkewDetected|KubeAPIDown|KubeletDown|KubeletTooManyPods|KubeNodeNotReady|KubeClientCertificateExpiration|NodeFileDescriptorLimit|NodeFilesystemAlmostOutOfFiles|NodeFilesystemAlmostOutOfSpace|NodeFilesystemFilesFillingUp|NodeFilesystemSpaceFillingUp|NodeRAIDDegraded|NodeRAIDDiskFailure|etcdInsufficientMembers|etcdMembersDown|etcdNoLeader|CoreDNSPanic|CoreDNSHealthRequestsLatency|KubeControllerManagerDown|KubeSchedulerDown|PrometheusRuleFailures|AlertmanagerClusterDown|AlertmanagerConfigInconsistent|AlertmanagerFailedReload|X509CertificateExpiration|X509CertificateRenewal"
    - namespace="monitoring"
    continue: true

As you can see, alertmanager adds a namespace = monitoring label, so the result is that I don't get the alerts delivered correctly to the target URL, because they don't have that label set.

My solution, for now, is to add the spec.enforcedNamespaceLabel = "namespace" in Prometheus resource, so every alert generated by Prometheus is injected with a label namespace = <namespace>.

However, this solves only the DeadMansSwitch issue, at the moment I'm not sure if the other alert groups are working fine. I will give you more details whenever possible.

Hi team, we previously removed this alert, since it's only noise on clusters. 23ef846

But in the last release is back https://github.com/sighupio/fury-kubernetes-monitoring/blob/master/katalog/prometheus-operated/kubernetes-monitoring-rules.yml#L344

Can we remove it again? And also, let's put somewhere the list on our changes, this way on the next update we don't occur in the same problem.

Since we are using common labels, the prometheus generated instance from prometheus operator is without required labels to be selected from the prometheus-k8s service

Prometheus-operator: v0.28.0
prometheus: v2.7.1
alertmanager: v0.16.0

With the latest release, we lost an important (IMHO) alert: NodeMachineIDCollision. It was introduced in the v1.5.0 version.

https://github.com/sighupio/fury-kubernetes-monitoring/blob/v1.5.0/katalog/prometheus-operated/rules.yml#L412-L420

Thanks!

To improve monitoring on packages, we need to add some prometheus alerts on Grafana package

Currently the aforementioned dashboard is showing some metrics specific to Weave-net in one of the main dashboards ("Kubernetes Cluster) which have no data point if it's not deployed.

Given that Calico is now our default standard CNI we should move this graphs to a separate dashboard if these are still relevant and add Calico related relevant metrics if necessary to the main dashboard.