Node Failure detection mechanism about metal3-docs HOT 7 CLOSED

/assign @dhellmann @russellb @maelk

from metal3-docs.

We looked at this for OpenShift, and came to the conclusion that the turn-around time for reporting hardware issues for running hosts through the baremetal operator would be slower than if we added features to tools like node_exporter. I'm not sure if all of the classes of issues listed above could be detected by data node_exporter collects today, so we may either need to add to node_exporter or at the very least ensure it is configured to report metal-specific data like SMART status for disks.

The BMO should report issues with ironic operations, since if inspection or provisioning fail then the host won't ever become a node and won't be able to run node_exporter and the BMO knows about the problem. I think we do put the host into an error state for those cases, although metal3-io/baremetal-operator#610 will mean the host doesn't stay in that state and we continuously retry the operation, so that might affect how the error is detected by a monitoring tool. We should ensure that the right prometheus metric for each failure is still updated as part of that work.

from metal3-docs.

https://github.com/metal3-io/ironic-prometheus-exporter may also be a useful way to report some of the stats that Ironic can collect out of band via the BMC.

from metal3-docs.

I have gone through the node-exporter previously, most of the error node-exporter doesn't reproduce. I will certainly look into Ironic-prometheus-exporter in detail. Most probably we will have to implement this feature in one of these project. I will investigate more in both of these project and will discuss with you later for further implementation.

from metal3-docs.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

from metal3-docs.

Stale issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle stale.

/close

from metal3-docs.

@metal3-io-bot: Closing this issue.

from metal3-docs.