Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.' about subtensor HOT 11 CLOSED

Confirming a workaround is to use an old node version, it is fully compatible with the latest runtime. We're using a625834e on a test machine and it syncs fine.

We're continuing to investigate this, thanks for bearing with us.

from subtensor.

This is fixed in #714

A node will attempt to use its native runtime in substitute for the on-chain Wasm runtime if all of spec_name, spec_version, and authoring_version are the same between the Wasm and native binaries. This was causing nodes compiled from main to use a runtime slightly different compared to what is on-chain, and write invalid stuff to storage.

In the future, CI should prevent us from ever pushing to main a spec version that matches what is on-chain (cc @sam0x17)

from subtensor.

The past week that's what was happening for me as well, but yesterday it wouldn't even work for a while, it immediately started spamming the same errors.

With the old Release, no issues for 24 hours so far, and the node is syncing fine.

from subtensor.

same issue here +1

from subtensor.

Facing the same problem

from subtensor.

This is a huge problem for us. We've been seeing this on our own subtensor node, and across many validators on the subnet, i.e. this is a bittensor-breaking issue which should be addressed ASAP to maintain the integrity of the entire chain.

So with that said, let's try to be constructive here:

1. Is there a known workaround?
2. Is there a known working commit?

Currently, latest main is not runnable on our machines, that's using docker or directly compiling from main...

@distributedstatemachine any ideas?

from subtensor.

With the old Release, no issues for 24 hours so far, and the node is syncing fine.

Yep can also confirm 1.1.1 is working for me:

cd subtensor
git fetch --all
git reset --hard 703bdbd178c5d6e64edd4ffcf69aa49d63349290
cargo build --release --features pow-faucet

... Restart

703bdbd

from subtensor.

Similar to #635

from subtensor.

This issue came back again after wiping my folders and using the rust based subtensor instead of docker 😦
Node breaks after a certain point 😦

I got the problem with a ryzen 7950x Debian using Docker subtensor (original issue) AND then wiped out disk to reinstall with Rust based subtensor also.

Just also found two other servers on different providers local subtensor out of sync.

All same cause

from subtensor.

For anyone still experiencing this issue, I pulled Release v1.1.1, cleaned blockchain data, rebuilt cargo and started the subtensor again and it works fine.

I am using the subtensor only to fetch data from the chain and for no other reason. If you are using the subtensor for any other reason, I can't guarantee this release is gonna work smoothly for you.

Workaround until this issue is fixed.

from subtensor.

For anyone still experiencing this issue, I pulled Release v1.1.1, cleaned blockchain data, rebuilt cargo and started the subtensor again and it works fine.

I am using the subtensor only to fetch data from the chain and for no other reason. If you are using the subtensor for any other reason, I can't guarantee this release is gonna work smoothly for you.

Workaround until this issue is fixed.

My node will run for about 24h normally before it encounters the error again when starting from scratch with a clean install.
Have yours been working for longer than this?

from subtensor.