node_exporter goes into crashLoopBackoff after node reboot and never recovers about node_exporter HOT 2 CLOSED

This does not sound like a node_exporter issue, but a Kubernetes issue.

There are no "detected prerequisites" in the node_exporter. Every scrape is dynamic.

The node_exporter logs any errors returned, the only way to exit with no errors is for the webserver function to exit with no errors, usually by a SIGTERM or SIGKILL.

from node_exporter.

IMO, I don't think it's k8s issue as the other pods are running without any issue. This issue is seen only when we reboot the node and NE tries to come up. NE is moved to CrashLoopBackoff state and is rendered in that state until we edit the daemonset/pod spec. Now the moment I edit the spec (even with minimal changes like reducing the probe time only to cause the pod to restart ), NE comes up. So I suspect that after the sandbox creation when the container is coming up it just exits possibly because it is getting into some condition which is not handled. I am mounting the volumes as below:

volumeMounts:
- mountPath: /host
  mountPropagation: HostToContainer
  name: rootfs
  readOnly: true
- mountPath: /host/proc
  name: proc
  readOnly: true
- mountPath: /host/sys
  name: sys
  readOnly: true

volumes:

• hostPath:
  path: /
  type: ""
  name: rootfs
• hostPath:
  path: /proc
  type: ""
  name: proc
• hostPath:
  path: /sys
  type: ""
  name: sys

Also I have a query in this context. When it crashes, do we get any trace in the log? Let me know if any information is needed which I am missing to provide.

from node_exporter.