[investigate]: Prevoting nil err, wrong Block.Header.AppHash. about gaia HOT 9 CLOSED

Thank you for opening the issue!

Could you let us know if you used a snapshot to sync after the upgrade or you upgraded your existing nodes? This would help us with triage.

Please let us know if this happens again. If you have further issues try running gaia rollback on the node that failed and re-starting.

Besides that, what usually works is re-syncing from a snapshot.

from gaia.

Hi

Could you let us know if you used a snapshot to sync after the upgrade or you upgraded your existing nodes?

Both current node were running from a snapshot

from gaia.

Hi

Could you let us know if you used a snapshot to sync after the upgrade or you upgraded your existing nodes?

Both current node were running from a snapshot

Thanks for the clarification. Apphash errors show more frequently on some of the snapshot providers.

Let's leave this issue opened in case it happens again. We'll keep an eye on the reports here and in other places.

from gaia.

Hi @MSalopek
It happened again, last friday both of my nodes went down at the same time

7:54AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{80.71.51.52:26656} 8acc5a62ad8eeb2140cff79a13dff0f993ab2354 out}"
7:54AM INF service stop impl="Peer{MConn{80.71.51.52:26656} 8acc5a62ad8eeb2140cff79a13dff0f993ab2354 out}" module=p2p msg="Stopping Peer service" [email protected]:26656
7:54AM ERR Error in validation err="wrong Block.Header.AppHash.  Expected 4D92317C4522B1812AEA119D76A23ABA554F18D7F1875C1426C5EAC3CC97A425, got 53651095E88C911B3D0F0B5D163A4F1B95E11A7ED4D4142C3C91BF2BFEB98F09" module=blockchain
7:54AM ERR Stopping peer for error err="Reactor validation error: wrong Block.Header.AppHash.  Expected 4D92317C4522B1812AEA119D76A23ABA554F18D7F1875C1426C5EAC3CC97A425, got 53651095E88C911B3D0F0B5D163A4F1B95E11A7ED4D4142C3C91BF2BFEB98F09" module=p2p peer="Peer{MConn{65.109.32.148:26726} 6ecca845883e9273062ee515d2657080e6539d9e out}"
7:54AM INF service stop impl="Peer{MConn{65.109.32.148:26726} 6ecca845883e9273062ee515d2657080e6539d9e out}" module=p2p msg="Stopping Peer service" [email protected]:26726
7:54AM INF service stop impl=MConn{65.109.32.148:26726} module=p2p msg="Stopping MConnection service" [email protected]:26726
7:54AM ERR Stopping peer for error err="Reactor validation error: wrong Block.Header.AppHash.  Expected 4D92317C4522B1812AEA119D76A23ABA554F18D7F1875C1426C5EAC3CC97A425, got 53651095E88C911B3D0F0B5D163A4F1B95E11A7ED4D4142C3C91BF2BFEB98F09" module=p2p peer="Peer{MConn{89.149.218.64:26656} 4cb59a50805f0646111412cf5d189e805e3e0c3d out}"

So you're guess is that the issue come from the snapshot I'm using? Can you explain more?
What should I investigate in my node settings to hope to resolve the problem?

Thks

from gaia.

Hi @MSalopek It happened again, last friday both of my nodes went down at the same time

7:54AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{80.71.51.52:26656} 8acc5a62ad8eeb2140cff79a13dff0f993ab2354 out}"
7:54AM INF service stop impl="Peer{MConn{80.71.51.52:26656} 8acc5a62ad8eeb2140cff79a13dff0f993ab2354 out}" module=p2p msg="Stopping Peer service" [email protected]:26656
7:54AM ERR Error in validation err="wrong Block.Header.AppHash.  Expected 4D92317C4522B1812AEA119D76A23ABA554F18D7F1875C1426C5EAC3CC97A425, got 53651095E88C911B3D0F0B5D163A4F1B95E11A7ED4D4142C3C91BF2BFEB98F09" module=blockchain
7:54AM ERR Stopping peer for error err="Reactor validation error: wrong Block.Header.AppHash.  Expected 4D92317C4522B1812AEA119D76A23ABA554F18D7F1875C1426C5EAC3CC97A425, got 53651095E88C911B3D0F0B5D163A4F1B95E11A7ED4D4142C3C91BF2BFEB98F09" module=p2p peer="Peer{MConn{65.109.32.148:26726} 6ecca845883e9273062ee515d2657080e6539d9e out}"
7:54AM INF service stop impl="Peer{MConn{65.109.32.148:26726} 6ecca845883e9273062ee515d2657080e6539d9e out}" module=p2p msg="Stopping Peer service" [email protected]:26726
7:54AM INF service stop impl=MConn{65.109.32.148:26726} module=p2p msg="Stopping MConnection service" [email protected]:26726
7:54AM ERR Stopping peer for error err="Reactor validation error: wrong Block.Header.AppHash.  Expected 4D92317C4522B1812AEA119D76A23ABA554F18D7F1875C1426C5EAC3CC97A425, got 53651095E88C911B3D0F0B5D163A4F1B95E11A7ED4D4142C3C91BF2BFEB98F09" module=p2p peer="Peer{MConn{89.149.218.64:26656} 4cb59a50805f0646111412cf5d189e805e3e0c3d out}"

So you're guess is that the issue come from the snapshot I'm using? Can you explain more? What should I investigate in my node settings to hope to resolve the problem?

Thks

I think the easiest would be to statesync instead of using a node snapshot using v15.2.0 binary. Try using this as a starting point: https://polkachu.com/state_sync/cosmos. From where are you getting your node snapshots?

Besides that, try purging your addressbook and download a new one and check that your persistent peers are upgraded correctly (if you use persistent peers).

Are you running a validator node or an RPC node? Are you accepting Txs or is it just used for reading state?

Could you share your config/app.toml and config/config.toml? Please take note to redact any sensitive information such as IP addresses, keys etc.

from gaia.

Could you share your config/app.toml and config/config.toml?

There is already the two config files you've asked for. I'll have a look for state_sync in the mean time.
config.toml.txt
app.toml.txt

Are you running a validator node or an RPC node? Are you accepting Txs or is it just used for reading state?

A RPC node that allow tx

from gaia.

Hello @MSalopek

From where are you getting your node snapshots?

They come from the following source: https://quicksync.io/cosmos.json
I'm using the cosmoshub-4-pruned file

from gaia.

Have you had any luck with other node snapshot providers or statesync?

Has the issue been resolved?
We don't have any other reports around this issue.

from gaia.

Network upgraded to v16.0.0 and there was no recent activity on this issue.

I'm closing it for now. Feel free to reopen if the issue happens for v16.x.

from gaia.