Giter Club home page Giter Club logo

Comments (15)

ackratos avatar ackratos commented on August 20, 2024

from node-binary.

RaghavSood avatar RaghavSood commented on August 20, 2024

Does two times crash you see "Open too many files" both?

If that's the case could you please paste your machine's 'ulimit -a' result
here? If the open files in result is less than 1024 please change system
config:

vim /etc/security/limits.conf
# add in the end
* soft nofile 4096 * hard nofile 4096

We did not see any file handle related errors, but I have increased the limit anyways - it still crashes on the same block, with the same log file and stderr messages.

from node-binary.

ackratos avatar ackratos commented on August 20, 2024

We did not see any file handle related errors, but I have increased the limit anyways - it still crashes on the same block, with the same log file and stderr messages.

The file handle related errors comes from here in your log:

I[2019-06-15|23:20:12.573] does not recover invalid tx                  module=dex txHash=3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE err="{\"codespace\":1,\"code\":1,\"abci_code\":65537,\"message\":\"recovered: open /home/ubuntu/.bnbchaind/data/application.db/151355.ldb: too many open files\\nstack:\\ngoroutine 836 [running]:\\nruntime/debug.Stack(0xc000de86b8, 0xfa79c0, 0xc0a7209410)\\n\\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx.func1(0xc000de9aa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:810 +0x6e\\npanic(0xfa79c0, 0xc0a7209410)\\n\\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*GoLevelDB).Get(0xc00000e488, 0xc05bc756b0, 0x29, 0x30, 0x29, 0xc05bc756b0, 0x8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*prefixDB).Get(0xc00fe58210, 0xc05bc75650, 0x21, 0x21, 0x0, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*nodeDB).GetNode(0xc0000bf900, 0xc11848d8a0, 0x20, 0x20, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:76 +0x29e\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).getRightNode(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc14cdce0b0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de89e8, 0xc000de8aa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:176 +0x319\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0a7d79d90, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8a48, 0xc000de8b08)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0a7d79ce0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8aa8, 0xc000de8b68)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a1130, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b08, 0xc000de8bc8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a1080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b68, 0xc000de8c28)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a0fd0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x249249249249, 0x249249249249)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc03413cb00, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x10000000042d1a1, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc02fa55080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc031fa3e20, 0xc000de8ce0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc011520210, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x20, 0xf483c0, 0xc11848d820, 0xc11848d820)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*ImmutableTree).Get(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0xc01102b920, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xc11848d820, 0x8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:131 +0x86\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountStoreCache).GetAccount(0xc053124140, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:236 +0x13d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc12b08e980, 0xc11848d3c0, 0x14, 0x14, 0xf34460, 0xc000de8eb8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc
12b08e980, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0x1042900, 0x13f2840)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0xc12b08ec00, 0x13a9d78)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetAccount(0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:87 +0xcb\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.getCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:159 +0xb1\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.addCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:199 +0xc2\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.inputOutputCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc0a7208510, 0x1, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:239 +0x43b\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.BaseKeeper.InputOutputCoins(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:73\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.handleMsgSend(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f4ca0, 0xc000d6a1e0, 0xc0a7208510, 0x1, 0x1, 0xc0a7208570, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:24 +0xed\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.NewHandler.func1(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f31a0, 0xc0a72085a0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:12 +0x226\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(0xc000891590, 0x13e9e80, 0xc0a7209080, 0xc12b08ea00, 0x11, 0xc031fa3840, 0x1, 0x1, 0xc1178339c0, 0x40, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:743 +0x2f3\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx(0xc000891590, 0x1100105, 0xc03ce4d2b0, 0xc8, 0xc8, 0x13cda80, 0xc12e319680, 0xc1178339c0, 0x40, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:833 +0x24c\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc000891590, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:651 +0x297\\ngithub.com/binance-chain/node/app.(*BinanceChain).DeliverTx(0xc0000b6840, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/app/app.go:400 +0x9b\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker.func1(0xc003117970, 0xc000de9fa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:142 +0x1b6\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker(0xc003117970)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 +0x8c\\ncreated by github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).OnStart\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93 +0x64\\n\"}" height=7402603

Your local database has already inconsistent with other peers as the first time you execute this tx 3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE your execution result is error: open too many files while all other nodes on chain is a good transaction: https://explorer.binance.org/tx/3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE.

To recover your database you'd better clean your $HOME/data folder, and increase the open file handle limit before you resync.

from node-binary.

RaghavSood avatar RaghavSood commented on August 20, 2024

Right, that makes sense - I've started a fresh sync, but is there any way to rollback just the broken chain state? The chain is already at 13m+ blocks, and resyncing from scratch after errors like this, which might crop up in other scenarios (running out of disk space, for instance) is not usually feasible.

from node-binary.

chainwhisper avatar chainwhisper commented on August 20, 2024

Right, that makes sense - I've started a fresh sync, but is there any way to rollback just the broken chain state? The chain is already at 13m+ blocks, and resyncing from scratch after errors like this, which might crop up in other scenarios (running out of disk space, for instance) is not usually feasible.

I think you have to restart from the beginning.

from node-binary.

ackratos avatar ackratos commented on August 20, 2024

we can release some data fixing tools to cover such case soon. but its not rollback, anyone cannot rollback blockchain.

if you doesn't care about historical data, there is a state_sync option https://docs.binance.org/fullnode.html#state-sync can help you directly sync to latest block.

from node-binary.

RaghavSood avatar RaghavSood commented on August 20, 2024

we can release some data fixing tools to cover such case soon. but its not rollback, anyone cannot rollback blockchain.

if you doesn't care about historical data, there is a state_sync option https://docs.binance.org/fullnode.html#state-sync can help you directly sync to latest block.

Right, of course, I'm not talking about an actual rollback of blocks, just the local state to a known-good point so that we don't need to resync from scratch if the chain head fails.

We unfortunately require the full history, so need to sync without pruning

from node-binary.

RaghavSood avatar RaghavSood commented on August 20, 2024

The issue still persists - A fresh resync from scratch after increasing the open file limit to 4096 synced up to the exact same block (7402603) before crashing again, with the same stack trace in the logs.

I increased the limit to 35000, and it is still failing on that specific block with the same error message.

Any more ideas?

Running on:

$ uname -a
Linux binance-mainnet-s 4.15.0-1033-gcp #35-Ubuntu SMP Fri May 17 13:27:54 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

from node-binary.

RaghavSood avatar RaghavSood commented on August 20, 2024

The main process at least does not seem to be using up all that many descriptors:

$ while sleep 1; do ls /proc/2000/fd | wc -l | tee -a temp.out; done
321
324
327
331
334
338
338
339
339
339
339
342
361
366
371
376
379
384
387
392
397
401
406
410
414
418
345
# crashed here

The node also doesn't seem to be using up the full limit:


$ ulimit -n
35000
$ cat /proc/2957/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             39869                39869                processes
Max open files            1024                 4096                 files
Max locked memory         16777216             16777216             bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       39869                39869                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

from node-binary.

chainwhisper avatar chainwhisper commented on August 20, 2024

How much memory does your machine have?

Can you share the error log here please?

from node-binary.

RaghavSood avatar RaghavSood commented on August 20, 2024

I've attached the latest bnc.log, and the stderr and stdout logs.

The machine is a GCP VM with 4 cores/10 GB memory - we are using a 500 GB SSD for the blockchain data.

logs.zip

from node-binary.

chainwhisper avatar chainwhisper commented on August 20, 2024

@RaghavSood I saw in the log that your node is already synced to height 8762856.

from node-binary.

RaghavSood avatar RaghavSood commented on August 20, 2024

@huangsuyu It was, that was the initial crash we received 2 days ago - we resynced from scratch after that, and then started crashing on block 7402603, which is when I opened this ticket.

from node-binary.

RaghavSood avatar RaghavSood commented on August 20, 2024

Apologies, I grabbed the logs from the wrong node (we have two, both are crashing with the same error).

Here's the updated ones:
bnc-s.zip

from node-binary.

chainwhisper avatar chainwhisper commented on August 20, 2024

Apologies, I grabbed the logs from the wrong node (we have two, both are crashing with the same error).

Here's the updated ones:
bnc-s.zip

Thanks!

Is bnbchaind running as a systemd?

why it restarted after crash 3mins later?

from node-binary.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.