Comments (11)
argakiig
commented on July 20, 2024
reth.log
full logs from latest run attached
from reth.
emhane
commented on July 20, 2024
Pipeline sync progress is inconsistent first_stage_checkpoint=19990817 inconsistent_stage_id=Execution inconsistent_stage_checkpoint=19983346 yeah ok seems like a bug
from reth.
argakiig
commented on July 20, 2024
latest log snippet
2024-06-01T17:44:00.369880Z INFO reth::cli: Status connected_peers=100 freelist=7661 stage=Execution checkpoint=19983346 target=19990817
2024-06-01T17:44:00.892994Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:00.897560Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-06-01T17:44:00.897926Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:00.921484Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:01.047626Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:01.048123Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:09.014534Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:09.312716Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:09.452077Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:09.611671Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:09.714275Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:12.922680Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:13.337590Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:13.341201Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-06-01T17:44:13.341563Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:13.576735Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:13.577466Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:24.923907Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:25.046618Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:25.053320Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-06-01T17:44:25.053765Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:25.300012Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:25.301049Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:25.369752Z INFO reth::cli: Status connected_peers=100 freelist=7661 stage=Execution checkpoint=19983346 target=19990817
2024-06-01T17:44:32.066220Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x752e5b4d61cf58d62165b96113cd334f98cb99e25fe3696ec3fcea2a97fb4cc3b3152b2d3f99aaa39d2381b395d50637aecd7f9fbe6e31bd455d1ab6e98b9e93
2024-06-01T17:44:34.492948Z DEBUG net: Session established remote_addr=217.31.52.180:30304 client_version=Nethermind/v1.25.4+20b10b35/linux-x64/dotnet8.0.2 peer_id=0xa25bb36d01ca8a36aefd6d657f14edb95ec1b04a13b4231a0e49212513d3ae436fe35ae961745fd81a2e363181dd0403e8354c7c2a94b5493c14c8950d4214cb total_active=100 kind=outgoing peer_enode=enode://a25bb36d01ca8a36aefd6d657f14edb95ec1b04a13b4231a0e49212513d3ae436fe35ae961745fd81a2e363181dd0403e8354c7c2a94b5493c14c8950d4214cb@217.31.52.180:30304
2024-06-01T17:44:34.541003Z DEBUG net: Session established remote_addr=209.159.151.82:30304 client_version=reth/v0.2.0-beta.4-7dcdfcc/x86_64-unknown-linux-gnu peer_id=0x4b2bd9bb79cf1e46da47527f679965050af9b305d19f85478eb8f1d034a9a466953c0fbfd0b7974f109abe862ec954291c909260feb5ad6802a9e9f9655a10fa total_active=101 kind=outgoing peer_enode=enode://4b2bd9bb79cf1e46da47527f679965050af9b305d19f85478eb8f1d034a9a466953c0fbfd0b7974f109abe862ec954291c909260feb5ad6802a9e9f9655a10fa@209.159.151.82:30304
2024-06-01T17:44:34.541645Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x4b2bd9bb79cf1e46da47527f679965050af9b305d19f85478eb8f1d034a9a466953c0fbfd0b7974f109abe862ec954291c909260feb5ad6802a9e9f9655a10fa
2024-06-01T17:44:34.736885Z DEBUG net: Session established remote_addr=194.110.169.134:50303 client_version=Nethermind/v1.25.4+20b10b35/linux-x64/dotnet8.0.2 peer_id=0x4c1225077ed50243fc990c0015b38ed4ec84ee3231972ee1466e95a1eaf7479d6dcf5de710134ce26f7ea3a2ce565d3c2c04cb2b9cbf9c2250fa2958c587e753 total_active=101 kind=outgoing peer_enode=enode://4c1225077ed50243fc990c0015b38ed4ec84ee3231972ee1466e95a1eaf7479d6dcf5de710134ce26f7ea3a2ce565d3c2c04cb2b9cbf9c2250fa2958c587e753@194.110.169.134:50303
2024-06-01T17:44:36.925307Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:37.943878Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:37.947510Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-06-01T17:44:37.947844Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:38.080914Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:38.081527Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-06-01T17:44:38.769218Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x4c1225077ed50243fc990c0015b38ed4ec84ee3231972ee1466e95a1eaf7479d6dcf5de710134ce26f7ea3a2ce565d3c2c04cb2b9cbf9c2250fa2958c587e753
from reth.
argakiig
commented on July 20, 2024
Another thing i find odd, I restarted the node and its exactly the same freelist checkpoint target combo as before
2024-06-03T03:17:34.849209Z INFO Starting reth version="0.2.0-beta.6 (ac29b4b)"
2024-06-03T03:17:34.849901Z INFO Opening database path="/data/db"
2024-06-03T03:17:34.856377Z INFO Configuration loaded path="/data/config.toml"
2024-06-03T03:17:34.860379Z INFO Database opened
2024-06-03T03:17:34.860399Z INFO Starting metrics endpoint addr=0.0.0.0:9001
2024-06-03T03:17:34.860568Z INFO
Pre-merge hard forks (block based):
- Frontier @0
- Homestead @1150000
- Dao @1920000
- Tangerine @2463000
- SpuriousDragon @2675000
- Byzantium @4370000
- Constantinople @7280000
- Petersburg @7280000
- Istanbul @9069000
- MuirGlacier @9200000
- Berlin @12244000
- London @12965000
- ArrowGlacier @13773000
- GrayGlacier @15050000
Merge hard forks:
- Paris @58750000000000000000000 (network is known to be merged)
Post-merge hard forks (timestamp based):
- Shanghai @1681338455
- Cancun @1710338135
2024-06-03T03:17:34.937439Z INFO Transaction pool initialized
2024-06-03T03:17:34.937545Z INFO Connecting to P2P network
2024-06-03T03:17:34.937728Z INFO Loading saved peers file=/data/known-peers.json
2024-06-03T03:17:34.965191Z INFO StaticFileProducer initialized
2024-06-03T03:17:34.965378Z INFO Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: None, transaction_lookup: None, receipts: None, account_history: None, storage_history: None, receipts_log_filter: ReceiptsLogPruneConfig({}) } }
2024-06-03T03:17:34.965471Z INFO Consensus engine initialized
2024-06-03T03:17:34.965530Z INFO Engine API handler initialized
2024-06-03T03:17:34.966873Z INFO RPC auth server started url=0.0.0.0:8551
2024-06-03T03:17:34.967085Z INFO RPC IPC server started url=/tmp/reth.ipc
2024-06-03T03:17:34.967096Z INFO RPC HTTP server started url=0.0.0.0:8545
2024-06-03T03:17:34.967099Z INFO RPC WS server started url=0.0.0.0:8546
2024-06-03T03:17:34.967106Z INFO Starting consensus engine
2024-06-03T03:17:34.967767Z INFO Preparing stage pipeline_stages=1/12 stage=Headers checkpoint=19990817 target=None
2024-06-03T03:17:34.967788Z INFO Target block already reached checkpoint=19990817 target=Hash(0x2099c7f6fb55870e0c49597237f4f27a508b1647a35a437808ce2442d3d1d874)
2024-06-03T03:17:34.967871Z INFO Executing stage pipeline_stages=1/12 stage=Headers checkpoint=19990817 target=None
2024-06-03T03:17:34.967938Z INFO Finished stage pipeline_stages=1/12 stage=Headers checkpoint=19990817 target=None stage_progress=100.00%
2024-06-03T03:17:34.968058Z INFO Preparing stage pipeline_stages=2/12 stage=Bodies checkpoint=19990817 target=19990817
2024-06-03T03:17:34.968077Z INFO Executing stage pipeline_stages=2/12 stage=Bodies checkpoint=19990817 target=19990817
2024-06-03T03:17:34.968084Z INFO Finished stage pipeline_stages=2/12 stage=Bodies checkpoint=19990817 target=19990817 stage_progress=100.00%
2024-06-03T03:17:34.968104Z INFO Preparing stage pipeline_stages=3/12 stage=SenderRecovery checkpoint=19990817 target=19990817
2024-06-03T03:17:34.968119Z INFO Executing stage pipeline_stages=3/12 stage=SenderRecovery checkpoint=19990817 target=19990817
2024-06-03T03:17:34.968161Z INFO Finished stage pipeline_stages=3/12 stage=SenderRecovery checkpoint=19990817 target=19990817 stage_progress=100.00%
2024-06-03T03:17:34.968224Z INFO Preparing stage pipeline_stages=4/12 stage=Execution checkpoint=19983346 target=19990817
2024-06-03T03:17:34.968259Z INFO Executing stage pipeline_stages=4/12 stage=Execution checkpoint=19983346 target=19990817
2024-06-03T03:17:37.967556Z INFO Status connected_peers=0 freelist=7661 stage=Execution checkpoint=19983346 target=19990817
2024-06-03T03:18:02.967355Z INFO Status connected_peers=1 freelist=7661 stage=Execution checkpoint=19983346 target=19990817
2024-06-03T03:18:04.423292Z INFO Received forkchoice updated message when syncing head_block_hash=0xdd873daca6f23fc22c2f3251062b57e370cc43735c3f4c8f36c6e92173baabcf safe_block_hash=0x7e877bb0568265a3f7db984457e340785d18e7ce1640d454e1162f3e16c969f7 finalized_block_hash=0x37c8be7cb8c86bac54a5b7d9f3fd5394d71700d4fea460d5b2cefa0785950ac8
from reth.
emhane
commented on July 20, 2024
Another thing i find odd, I restarted the node and its exactly the same freelist checkpoint target combo as before
that's normal, it's the measure of the number of pages on the mdbx freelist, which basically measures how well memory is used, i.e. how compact the db is (less free pages, more compact and vv). more about the freelist here if you're curious #5228.
from reth.
argakiig
commented on July 20, 2024
Another thing i find odd, I restarted the node and its exactly the same freelist checkpoint target combo as before
that's normal, it's the measure of the number of pages on the mdbx freelist, which basically measures how well memory is used, i.e. how compact the db is (less free pages, more compact and vv). more about the freelist here if you're curious #5228.
Thank you I understand what the freelist is, what I dont quite understand is why it would remain static for going on 4 days now despite restarts it always returns to the same point. Does this mean nothing is being written at all and if so why?
from reth.
bislly
commented on July 20, 2024
i am using docker and had the same issue, instead stuck on the MerkleExecute stage (8/12).
(log also showed Pipeline sync progress is inconsistent)
tried fixing it and i did 2 thing that one of them worked
when i looked at the lighthouse logs, 1 log showed:
INFO Refusing to checkpoint sync, msg: database already exists, use --purge-db to force checkpoint sync, service: beacon, module: client::builder:243
so i added --purge-db one time
The other thing was:
some time ago, i put some limits on the dockers in the docker compose file:
deploy:
resources:
limits:
cpus: '0.50'
memory: 2G
after i removed it and run the docker again, it seems to work and continue with the stage.
one of these fixed the case for me
from reth.
argakiig
commented on July 20, 2024
i am using docker and had the same issue, instead stuck on the MerkleExecute stage (8/12). (log also showed Pipeline sync progress is inconsistent)
tried fixing it and i did 2 thing that one of them worked when i looked at the lighthouse logs, 1 log showed: INFO Refusing to checkpoint sync, msg: database already exists, use --purge-db to force checkpoint sync, service: beacon, module: client::builder:243 so i added --purge-db one time
The other thing was: some time ago, i put some limits on the dockers in the docker compose file: deploy: resources: limits: cpus: '0.50' memory: 2G
after i removed it and run the docker again, it seems to work and continue with the stage.
one of these fixed the case for me
thanks for suggestions. it is running in a kubernetes context with 16cpu and 64gb ram for reth and 4/16 for lighthouse.
I started lighthouse from an empty disk this last time, but will purge db and report back as well
from reth.
argakiig
commented on July 20, 2024
well, quick update. with 0.8.0, the freelist has ballooned but the node seems unable to keep going. I will keep that running for a bit and provide further debug logs
2024-06-05T01:39:21.902605Z INFO Status connected_peers=10 freelist=7661 latest_block=19983346
2024-06-05T01:39:34.525810Z INFO Received forkchoice updated message when syncing head_block_hash=0x15205573c72afe22a54d9189ba98834f997da5aecc9ea5aba1a382394efd3db8 safe_block_hash=0x4a23f9338a4658c8efaa33b9b0d55f2c1c21b9ab963edb6ac804eb4becdad2b0 finalized_block_hash=0x172e10d85746b20e0e334344a0a1de1aca22222e4754d4547752f603a571501c
2024-06-05T01:39:46.902645Z INFO Status connected_peers=11 freelist=7661 latest_block=19983346
2024-06-05T01:40:11.286909Z INFO Target block already reached checkpoint=19990817 target=Hash(0x2099c7f6fb55870e0c49597237f4f27a508b1647a35a437808ce2442d3d1d874)
2024-06-05T01:40:11.286948Z INFO Preparing stage pipeline_stages=1/12 stage=Headers checkpoint=19990817 target=None
2024-06-05T01:40:11.286961Z INFO Executing stage pipeline_stages=1/12 stage=Headers checkpoint=19990817 target=None
2024-06-05T01:40:11.287052Z INFO Finished stage pipeline_stages=1/12 stage=Headers checkpoint=19990817 target=None stage_progress=100.00%
2024-06-05T01:40:11.288350Z INFO Preparing stage pipeline_stages=2/12 stage=Bodies checkpoint=19990817 target=19990817
2024-06-05T01:40:11.288387Z INFO Executing stage pipeline_stages=2/12 stage=Bodies checkpoint=19990817 target=19990817
2024-06-05T01:40:11.288400Z INFO Finished stage pipeline_stages=2/12 stage=Bodies checkpoint=19990817 target=19990817 stage_progress=100.00%
2024-06-05T01:40:11.293640Z INFO Preparing stage pipeline_stages=3/12 stage=SenderRecovery checkpoint=19990817 target=19990817
2024-06-05T01:40:11.293662Z INFO Executing stage pipeline_stages=3/12 stage=SenderRecovery checkpoint=19990817 target=19990817
2024-06-05T01:40:11.293669Z INFO Finished stage pipeline_stages=3/12 stage=SenderRecovery checkpoint=19990817 target=19990817 stage_progress=100.00%
2024-06-05T01:40:11.294494Z INFO Preparing stage pipeline_stages=4/12 stage=Execution checkpoint=19983346 target=19990817
2024-06-05T01:40:11.294517Z INFO Executing stage pipeline_stages=4/12 stage=Execution checkpoint=19983346 target=19990817
2024-06-05T01:40:11.903339Z INFO Status connected_peers=14 freelist=808150 stage=Execution checkpoint=19983346 target=19990817
2024-06-05T01:40:36.903087Z INFO Status connected_peers=14 freelist=808150 stage=Execution checkpoint=19983346 target=19990817
from reth.
argakiig
commented on July 20, 2024
I have another node syncing with 0.8.0 and it is progressing with an almost identical helm deployment so it doesnt seem likely its a configuration issue but more likely somehow I got multiple bad snapshot dbs?
from reth.
argakiig
commented on July 20, 2024
ok, so after upgrading to beta 9 it seems I am finally able to get past the point I have been stalled at for 2 days 👍 This can be closed unless someone else needs it
from reth.
Related Issues (20)
- Add new BlockchainProvider type
- handling additional trace_filter fields HOT 1
- Move `CanonStateNotification` out of `reth-provider` crate
- Segmentation fault HOT 5
- Integrate memory overlay in canonical state
- Add more docs for tree engine
- Rename BlockSource::Database variant
- TxHash received by Exex is zero for a lot of transactions HOT 4
- Add support for eth_getAccount HOT 8
- op-reth: import receipts issue
- Import CLI fails on archive node because Prune stages aren't in the set
- op-reth: block import issue HOT 1
- Use correct rpc account type for eth_getAccount
- Remove InMemoryState trait
- Implement compact for eof bytecode
- Add storage_root provider function for account
- Integrate chainorchestrator events into node events
- Add provider function to return a list of all stages and their progress HOT 1
- create a RateLimit middleware layer for rpcservice
- Enable GZIP compression for JSON-RPC calls
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from reth.