Comments (15)
from node-binary.
Does two times crash you see "Open too many files" both?
If that's the case could you please paste your machine's 'ulimit -a' result
here? If the open files in result is less than 1024 please change system
config:vim /etc/security/limits.conf
# add in the end
* soft nofile 4096 * hard nofile 4096
We did not see any file handle related errors, but I have increased the limit anyways - it still crashes on the same block, with the same log file and stderr messages.
from node-binary.
We did not see any file handle related errors, but I have increased the limit anyways - it still crashes on the same block, with the same log file and stderr messages.
The file handle related errors comes from here in your log:
I[2019-06-15|23:20:12.573] does not recover invalid tx module=dex txHash=3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE err="{\"codespace\":1,\"code\":1,\"abci_code\":65537,\"message\":\"recovered: open /home/ubuntu/.bnbchaind/data/application.db/151355.ldb: too many open files\\nstack:\\ngoroutine 836 [running]:\\nruntime/debug.Stack(0xc000de86b8, 0xfa79c0, 0xc0a7209410)\\n\\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx.func1(0xc000de9aa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:810 +0x6e\\npanic(0xfa79c0, 0xc0a7209410)\\n\\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*GoLevelDB).Get(0xc00000e488, 0xc05bc756b0, 0x29, 0x30, 0x29, 0xc05bc756b0, 0x8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*prefixDB).Get(0xc00fe58210, 0xc05bc75650, 0x21, 0x21, 0x0, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*nodeDB).GetNode(0xc0000bf900, 0xc11848d8a0, 0x20, 0x20, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:76 +0x29e\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).getRightNode(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc14cdce0b0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de89e8, 0xc000de8aa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:176 +0x319\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0a7d79d90, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8a48, 0xc000de8b08)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0a7d79ce0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8aa8, 0xc000de8b68)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a1130, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b08, 0xc000de8bc8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a1080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b68, 0xc000de8c28)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a0fd0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x249249249249, 0x249249249249)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc03413cb00, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x10000000042d1a1, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc02fa55080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc031fa3e20, 0xc000de8ce0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc011520210, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x20, 0xf483c0, 0xc11848d820, 0xc11848d820)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*ImmutableTree).Get(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0xc01102b920, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xc11848d820, 0x8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:131 +0x86\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountStoreCache).GetAccount(0xc053124140, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:236 +0x13d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc12b08e980, 0xc11848d3c0, 0x14, 0x14, 0xf34460, 0xc000de8eb8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc
12b08e980, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0x1042900, 0x13f2840)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0xc12b08ec00, 0x13a9d78)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetAccount(0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:87 +0xcb\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.getCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:159 +0xb1\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.addCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:199 +0xc2\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.inputOutputCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc0a7208510, 0x1, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:239 +0x43b\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.BaseKeeper.InputOutputCoins(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:73\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.handleMsgSend(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f4ca0, 0xc000d6a1e0, 0xc0a7208510, 0x1, 0x1, 0xc0a7208570, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:24 +0xed\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.NewHandler.func1(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f31a0, 0xc0a72085a0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:12 +0x226\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(0xc000891590, 0x13e9e80, 0xc0a7209080, 0xc12b08ea00, 0x11, 0xc031fa3840, 0x1, 0x1, 0xc1178339c0, 0x40, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:743 +0x2f3\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx(0xc000891590, 0x1100105, 0xc03ce4d2b0, 0xc8, 0xc8, 0x13cda80, 0xc12e319680, 0xc1178339c0, 0x40, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:833 +0x24c\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc000891590, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:651 +0x297\\ngithub.com/binance-chain/node/app.(*BinanceChain).DeliverTx(0xc0000b6840, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/app/app.go:400 +0x9b\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker.func1(0xc003117970, 0xc000de9fa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:142 +0x1b6\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker(0xc003117970)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 +0x8c\\ncreated by github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).OnStart\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93 +0x64\\n\"}" height=7402603
Your local database has already inconsistent with other peers as the first time you execute this tx 3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE
your execution result is error: open too many files
while all other nodes on chain is a good transaction: https://explorer.binance.org/tx/3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE.
To recover your database you'd better clean your $HOME/data
folder, and increase the open file handle limit before you resync.
from node-binary.
Right, that makes sense - I've started a fresh sync, but is there any way to rollback just the broken chain state? The chain is already at 13m+ blocks, and resyncing from scratch after errors like this, which might crop up in other scenarios (running out of disk space, for instance) is not usually feasible.
from node-binary.
Right, that makes sense - I've started a fresh sync, but is there any way to rollback just the broken chain state? The chain is already at 13m+ blocks, and resyncing from scratch after errors like this, which might crop up in other scenarios (running out of disk space, for instance) is not usually feasible.
I think you have to restart from the beginning.
from node-binary.
we can release some data fixing tools to cover such case soon. but its not rollback, anyone cannot rollback blockchain.
if you doesn't care about historical data, there is a state_sync option https://docs.binance.org/fullnode.html#state-sync can help you directly sync to latest block.
from node-binary.
we can release some data fixing tools to cover such case soon. but its not rollback, anyone cannot rollback blockchain.
if you doesn't care about historical data, there is a state_sync option https://docs.binance.org/fullnode.html#state-sync can help you directly sync to latest block.
Right, of course, I'm not talking about an actual rollback of blocks, just the local state to a known-good point so that we don't need to resync from scratch if the chain head fails.
We unfortunately require the full history, so need to sync without pruning
from node-binary.
The issue still persists - A fresh resync from scratch after increasing the open file limit to 4096 synced up to the exact same block (7402603) before crashing again, with the same stack trace in the logs.
I increased the limit to 35000, and it is still failing on that specific block with the same error message.
Any more ideas?
Running on:
$ uname -a
Linux binance-mainnet-s 4.15.0-1033-gcp #35-Ubuntu SMP Fri May 17 13:27:54 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
from node-binary.
The main process at least does not seem to be using up all that many descriptors:
$ while sleep 1; do ls /proc/2000/fd | wc -l | tee -a temp.out; done
321
324
327
331
334
338
338
339
339
339
339
342
361
366
371
376
379
384
387
392
397
401
406
410
414
418
345
# crashed here
The node also doesn't seem to be using up the full limit:
$ ulimit -n
35000
$ cat /proc/2957/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 39869 39869 processes
Max open files 1024 4096 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 39869 39869 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
from node-binary.
How much memory does your machine have?
Can you share the error log here please?
from node-binary.
I've attached the latest bnc.log, and the stderr and stdout logs.
The machine is a GCP VM with 4 cores/10 GB memory - we are using a 500 GB SSD for the blockchain data.
from node-binary.
@RaghavSood I saw in the log that your node is already synced to height 8762856.
from node-binary.
@huangsuyu It was, that was the initial crash we received 2 days ago - we resynced from scratch after that, and then started crashing on block 7402603
, which is when I opened this ticket.
from node-binary.
Apologies, I grabbed the logs from the wrong node (we have two, both are crashing with the same error).
Here's the updated ones:
bnc-s.zip
from node-binary.
Apologies, I grabbed the logs from the wrong node (we have two, both are crashing with the same error).
Here's the updated ones:
bnc-s.zip
Thanks!
Is bnbchaind running as a systemd?
why it restarted after crash 3mins later?
from node-binary.
Related Issues (20)
- Request Blocked While Connecting to the Binance Chain public Node
- .
- Sync full node failed! How to solve? HOT 3
- testnet Couldn't connect to any seeds HOT 2
- Get Testnet BNB HOT 1
- Requires testnet BNB
- PLEASE MAKE SNAPSHOT FOR BC NODE (NOT BSC)
- snapshot cannot download on page https://docs.binance.org/guides/node/snapshot.html HOT 1
- Node cannot run, apphash inconsistent error HOT 5
- ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected HOT 4
- No latest snapshots can not be downloaded
- Please release 0.9.0 in this repo HOT 1
- document spell error
- tss tools. ./tss init can not create p2p info?
- wallet hacked
- prof port should be 9060
- TestNet Node ERROR
- error during handshake: error on replay: Wrong Block.Header.AppHash HOT 2
- ERROR: Block.Header.AppHash on mainnet with binary version 0.9.0 HOT 2
- Unable to sync mainnet, failed to process committed block 285461516, state_recover do not work HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-binary.