[Bug]: Node binary should quit with status code 1 when a fatal error occurs. It currently does not. about gaia HOT 7 CLOSED

Thank you for opening the issue.

I don't think this is the case for v17 only but for all versions using cosmos-sdk v47.

Does the node respond when you query /status or /block?

from gaia.

https://forum.cosmos.network/t/cosmos-hub-v17-1-chain-halt-post-mortem/13899

During the chain halt the node was still running and peering and responding to /status and /block

Maybe this issue should be raised in the cosmos-sdk repository.

from gaia.

I am pretty sure this is a feature of the upstream SDK yea. Peer Exchange continues despite a non 0 exit code (baseapp recovers from it to keep the instances alive) since it could recover in some exit 1 cases (not this one, but others).

from gaia.

I am pretty sure this is a feature of the upstream SDK yea. Peer Exchange continues despite a non 0 exit code (baseapp recovers from it to keep the instances alive) since it could recover in some exit 1 cases (not this one, but others).

From a devops standpoint, when running nodes, I believe nodes that halt and do not progress from irrecoverable errors should exit with 1 code. For example, should one run a container on either Docker or Kubernetes, when the main app exits from an error there could be automatic attempts to restart and also trigger error logs to Docker/Kubernetes monitoring systems.

When you keep a node alive from errors that cannot be recovered there are a ton of funny edge cases you need to think about.

A node that is in error and connected to peers and responds to API calls, I have seen catching_up = false on status calls. Even if we monitor latest_block_time how do we differentiate between a legitimate chain halt from an upgrade where latest_block_time is stale, or a fatal error that crashes a node and continues to give a stale latest_block_time?

Furthermore, is there really a point in staying connected to peers during a fatal error that crashes a node?

Surely there needs to be more work on identifying errors that should immediately terminate the node and cause a cascade in warnings and error in monitoring systems and errors that are recoverable.

from gaia.

I agree with the points above.

Can you open this issue on cosmos-sdk? This is out of scope for a chain repository.

Please tag me on the new issue if you need further assistance from us.

from gaia.

How was the fatal halt conveyed here? apphash, lastresulthash mismatches? it seems the framework acted as intended.

from gaia.

Discussion continues on cosmos-sdk

Closing.

from gaia.