harmony-one / pangaea-community Goto Github PK

Describe the bug
Concurrent wallet transfers are not possible - one of the wallet processes will crash and complain about:

panic: cannot open Badger datastore at .dht---c2: Cannot acquire directory lock on ".dht---c2". 

Another process is using this Badger database.: resource temporarily unavailable

To Reproduce
Start two concurrent transfers:

• ./wallet.sh -t transfer --from one1kv8r6mwy9lp9tf39vrajqr96k983hv5e0l98wg --to one1xvqpwgpkvf3xtvj692ncmch2fx0thqwgv8dymd --amount=0.0001 --shardID=0 --toShardID=0 --pass pass:
• ./wallet.sh -t transfer --from one1kv8r6mwy9lp9tf39vrajqr96k983hv5e0l98wg --to one1xvqpwgpkvf3xtvj692ncmch2fx0thqwgv8dymd --amount=0.0001 --shardID=0 --toShardID=0 --pass pass:

Expected behavior
If concurrent transfers should be possible according to design:

• none of the wallet binaries should crash
• both transactions should be attempted

If concurrent transfers shouldn't be possible according to design:

• output an error message for the wallet process that didn't manage to acquire a lock on the Badger database instead of crashing/panicing

Details
The reason this bug happens is because of a combination of missing variables and the naming of the .dht/Badger database:

https://github.com/harmony-one/harmony/blob/pangaea/node/service_setup.go#L113-L119

func (node *Node) networkInfoDHTPath() string {
  return fmt.Sprintf(".dht-%s-%s-c%s",
    node.NodeConfig.SelfPeer.IP,
    node.NodeConfig.SelfPeer.Port,
    node.chainConfig.ChainID,
  )
}

Somehow node.NodeConfig.SelfPeer.IP and node.NodeConfig.SelfPeer.Port are not set which results in the .dht/Badger db to be generated with the exact same name for all wallet processes, e.g: .dht---c2.

Because of the naming issue all wallet processes will try to acquire locks on the exact same .dht/Badger db (.dht---c2) whereas the proper functionality should be to generate a Badger db using an unique ip and port combination for each wallet process, eg:

• .dht-127.0.0.1-9100-c2
• .dht-127.0.0.1-9011-c2
• .dht-127.0.0.1-9107-c2
• etc...

Interesting enough I haven't been able to reproduce this error using the local development/testnet enviroment because node.NodeConfig.SelfPeer.IP and node.NodeConfig.SelfPeer.Port are properly assigned.

I added the following debugging/output code to func (node *Node) networkInfoDHTPath() in node/service_setup.go:

func (node *Node) networkInfoDHTPath() string {
  fmt.Println("\n\n\nDebugging networkInfoDHTPath:")
  fmt.Println(fmt.Sprintf("node.SelfPeer.IP: %s", node.SelfPeer.IP))
  fmt.Println(fmt.Sprintf("node.SelfPeer.Port: %s", node.SelfPeer.Port))
  fmt.Println(fmt.Sprintf("node.chainConfig.ChainID: %s", node.chainConfig.ChainID))
  fmt.Println(fmt.Sprintf(".dht file will be named: .dht-%s-%s-c%s", node.SelfPeer.IP, node.SelfPeer.Port, node.chainConfig.ChainID))

  return fmt.Sprintf(".dht-%s-%s-c%s",
    node.NodeConfig.SelfPeer.IP,
    node.NodeConfig.SelfPeer.Port,
    node.chainConfig.ChainID,
  )
}

And I get the following output:

Debugging networkInfoDHTPath:
node.SelfPeer.IP: 127.0.0.1
node.SelfPeer.Port: 9100
node.chainConfig.ChainID: 2
.dht file will be named: .dht-127.0.0.1-9100-c2

Debugging networkInfoDHTPath:
node.SelfPeer.IP: 127.0.0.1
node.SelfPeer.Port: 9011
node.chainConfig.ChainID: 2
.dht file will be named: .dht-127.0.0.1-9011-c2

Debugging networkInfoDHTPath:
node.SelfPeer.IP: 127.0.0.1
node.SelfPeer.Port: 9107
node.chainConfig.ChainID: 2
.dht file will be named: .dht-127.0.0.1-9107-c2

Which means that node.NodeConfig.SelfPeer.IP / node.NodeConfig.SelfPeer.Port are properly assigned in the local test network but are not getting properly assigned for my actual Pangaea node.

Environment (please complete the following information):
Pangaea node:

• OS: Linux 4.15.0-62-generic #69-Ubuntu SMP Wed Sep 4 20:55:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Instance type: OVH VPS (8gb memory, 2 CPU, 80gb SSD)
• Wallet binary version: Harmony (C) 2019. wallet, version v4696-pangaea-20190924.0-0-ge3030c50 (jenkins@ 2019-09-25T01:31:41+0000)

Local development environment:

• OS: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
• Wallet binary version: Harmony (C) 2019. wallet, version v4803-v1-20190924.0-99-g1297623c-dirty (2019-10-04T11:40:04+0200)

Describe the bug
Tx spam using large transactions (~100kb) results in excessive node memory usage.

This could happen because of:

• Unbounded/non-pooled Go subroutines eventually spinning out of control and crashing the nodes, and/or
• Excessive in-memory allocation of certain arrays/slices/queues (the pendingTransactions queue is especially interesting).

Issue has already been reported to @harmony-ek in the P-OPS Telegram channel but after discussion with @AndyBoWu today on Discord I was asked to open an issue for this.

There's already a related open issue for the unbounded/non pooled subroutines here: harmony-one/harmony#1645

It's also very possible that the slices/arrays that keep track of pending transactions, cx receipts etc. could be a part of the problem. Some slices/arrays (especially pendingTransactions) store all of the tx data in memory, and if people routinely spam ~100kb transactions the in-memory consumption of the node process increases quite fast.

Assuming a 20k tx pool limit for pending transactions (I've routinely seen Pangaea node operators with 15-17k pendings transactions in their queues) some nodes might end up storing gigabytes of data in memory for that queue alone. 15k pending transactions (including all of the tx data/embedded base64-data) could theoretically end up storing 1.5gb of pending transaction data in memory, given all transactions are ~100kb.

Add the other slices/arrays/queues to the mix coupled with the unbounded Go subroutines and it's probably no surprise that even the explorer m5.large instances (with 7.5gb available memory) experience the OOM reaping.

After restarting the harmony process on a m5.large explorer node, the process typically only manages to stay alive for 15-20 minutes before getting OOM-reaped by the OS. The process usually caps out at 7.1-7.2 gb of memory before the OS reaps the process and a tracelog is outputted, e.g: https://gist.github.com/SebastianJ/ad569b1ce48742b2a06117d6c273fa3a

(Tracelog seems to indicate unbounded subroutines being a major issue)

To Reproduce
Steps to reproduce the behavior:

1. Spam the network with transactions with a lot of base64 embedded data (90-100kb), e.g: https://gist.githubusercontent.com/SebastianJ/50c1405109d64651e13958d82eae112c/raw/fbbbdf598dd1bfd533f4d944f10f0176f71cb8c2/HugeTxExample (just put it in a loop or something to make sure it constantly spams the network)
2. Let the network start processing these transactions
3. Wait for nodes to start getting OOM reaped (seems explorer nodes get OOM-reaped way earlier than regular nodes with lesser memory - guessing this is because explorer nodes perform more processing and memory intensive tasks)

Expected behavior
Network should be able to cope with a massive amount of transactions, both in terms of number of transactions as well as the size of transactions in terms of bytes.

Environment (please complete the following information):
Explorer nodes:

• OS: Linux 4.14.77-81.59.amzn2.x86_64 #1 SMP Mon Nov 12 21:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Instance type: m5.large (7.5gb available memory)
• Harmony binary version: Harmony (C) 2019. harmony, version v4696-pangaea-20190924.0-0-ge3030c50 (ec2-user@ 2019-09-24T23:30:06+0000)

Additional info
Just as an experiment I upgraded all explorer nodes to use Systemd units to start the harmony binary so that they would auto-restart upon getting OOM-reaped.

So far that has only worked for the shard 1 explorer. None of the other explorer nodes manage to sync or display blocks properly on the explorer Web UI - they seem to get stuck in a perpetual state of trying to sync and then losing the sync status when they get OOM-reaped. Shard 1 somehow manages to get past this state.

There's also a related issue regarding large transactions and the Web UI here: harmony-one/harmony#1676

Tech Ambassador are willing to be more involved in operating Pangaea. However for that, we'll need more privilege.

This issue is opened so we can discuss solution. (interim or long term)

using node.sh binary - harmony, version v4662-pangaea-20190920.0-0-gc2324c41
using wallet.sh binary - wallet, version v4662-pangaea-20190920.0-0-gc2324c41

CPU useage can rise from 30% to 70%+ from a single transaction. Using 2 CPU, 4 GB ram.
e.g.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28441 root 20 0 763760 29500 23864 R 72.1 0.7 0:02.99 wallet
28422 root 20 0 3414356 317060 26896 S 42.2 7.8 0:04.85 wallet
1154 root 20 0 3781312 315688 28760 S 29.9 7.8 190:04.37 harmony
28458 root 20 0 757344 22504 19032 R 18.9 0.6 0:00.57 wallet

Running a test w:ith XC and have the following finding:
Ok, I run in to the following scenario:
I have two one1-address accounts,
1 is original and comes from Harmony during signup proces.
2 is created by myself for testing

When sending with XC from 2 to 1 it is working correctly and I also receive the amount on account 1.
When sending with XC from 1 to 2 it seems to be working, but amount is not received on account 2 and tokens are lost (in space 🤣😂)

Transactions can both be found in Pangaea Explorer!

Can someone test and confirm this?

Not sure if this is ok but this block is more than 90x the average block size. This could be an issue for disseminate the block, and increase latency.
https://explorer.pangaea.harmony.one/#/block/0x1c4439693842a3b0c19372ae5ad456268f28a7f12643a5db5edde945a6cbfd53

In Pangaea Network several people is flooding the network with tx with a lot of MB data input, for example with base64 text, base 64 images, etc...
This is not an issue itself, but this will lead to a memory/block size problems and an increase of pending transactions due to block propagation.
Solution 1: the "--input data" command should allow much less size (bytes) base 64 input, to improve propagation and avoid pending transactions increasing.
For example blocks with MAX ~2000 bytes, not 99999 bytes(currently).
Solution 2: implement a gas mechanism to protect the network, the more the tx input data the more gas payment
Current pending transactions in my node: 10108

when searching zerolog

found a strange error
{"level":"info","port":"9000","ip":"13.94.186.14","error":"leveldb: not found","number":4920,"dataLen":0,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/core/rawdb/accessors_chain.go:545","time":"2019-09-25T10:50:15.637918871Z","message":"ReadCXReceipts"}

We have the possibility to have several wallets addresses, and when we check the wallet balance using for example a script, the Putty explode literally. please see the video/zip
How to recreate the issue:

• Create 1000 wallets:
  for n in {1..1000}; do ./wallet.sh -t new -nopass ; done ;

• Check balances:
  ./wallet.sh -t balances

and now wait until jump and print goroutine messages

Please check the zip file --> video
video_Bruno Marshall-wallet issue.zip

The command "wallet.sh -t getFreeToken --address does not work correctly.
Just created new wallet and tried to get the 100 tokens for every shard (faucet?), but this doesn't work at the moment.
shard1 doesn't even show up as being requested!

./wallet.sh -t getFreeToken --address one1k5f0gj3j5kwnczeq7zt5g44hrwp8rrsey69u04
Using pangaea profile for wallet
Transaction Id requesting free token in shard 0: 0xe962c3a508a0497cff3397d41e735870b3a105a2801b532be165b883985f0348
Transaction Id requesting free token in shard 2: 0x6962d91accb3f5a3322869f4fa18bebf54a25fbe4c154d00b497137c789d9b7e
Transaction Id requesting free token in shard 3: 0x512d057c9e64b9a0c7f1d17b5a87fcbe257a5375f2e32255efaefdea9dd250f2

here is the log:

release/harmony/node/node.go:276","time":"2019-09-23T09:22:39.392095567Z","message":"Got more transactions"}
{"level":"info","port":"9000","ip":"209.97.180.175","numPeersNow":0,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node_handler.go:580","time":"2019-09-23T09:22:41.369054045Z","message":"No peers, continue"}
{"level":"warn","port":"9000","ip":"209.97.180.175","error":"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 34.228.143.91:6000: connect: connection refused"","peerIP":"34.228.143.91","peerPort":"6000","caller":"/mnt/jenkins/workspace/harmony-release/harmony/api/service/syncing/syncing.go:709","time":"2019-09-23T09:22:43.692263741Z","message":"[Sync]GetBlockChainHeight failed"}
{"level":"debug","port":"9000","ip":"209.97.180.175","OtherHeight":15177,"MyHeight":15177,"IsOutOfSync":false,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/api/service/syncing/syncing.go:739","time":"2019-09-23T09:22:43.923313548Z","message":"[SYNC] Checking sync status"}
{"level":"info","port":"9000","ip":"209.97.180.175","numPeersNow":0,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node_handler.go:580","time":"2019-09-23T09:22:46.368978629Z","message":"No peers, continue"}
{"level":"info","port":"9000","ip":"209.97.180.175","numPeersNow":0,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node_handler.go:580","time":"2019-09-23T09:22:51.369081986Z","message":"No peers, continue"}
{"level":"debug","port":"9000","ip":"209.97.180.175","caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node_handler.go:152","time":"2019-09-23T09:22:51.555345883Z","message":"NET: received message: Node/Transaction"}
{"level":"info","port":"9000","ip":"209.97.180.175","length of newTxs":1,"totalPending":15,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node.go:276","time":"2019-09-23T09:22:51

Ambassador can eventually help if they have access to the network

Hello ! Pangaea faucet it does not work anymore ...

The command used:
./wallet.sh -t getFreeToken --address=one13ncg4atrsw96y685fcw402rhqgh80zden4t54n

After restarted the node i see several error messages in my log.
i am receiving bingo's and also reached consesus.
i did not send any tx.

below the errors

{"level":"error","port":"9000","ip":"13.94.186.14","caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node_handler.go:408","time":"2019-09-25T10:12:09.789070563Z","message":"Error when parsing tx into message"}

and

{"level":"error","port":"9000","ip":"13.94.186.14","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","target":"52.68.207.80:6000","response":null,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/api/service/syncing/downloader/client.go:103","time":"2019-09-25T10:12:03.231451749Z","message":"[SYNC] client.go:Register failed"}

Can do this endlessly.....
Send tokens, get a txhash.
Nonce wont increase
Balance wont reduce
txHash wont appear on explorer

Transactions appears double or/with same ID number
How to recreate this issue:

• Run 1000 TX:

for n in {1..1000}; do ./wallet.sh -t transfer --from one1kkznllwcpcqakvmpqrll5yzp9tdjnf4ufnxcl6 --to one1nqkv7j2tzvqfu684cxdtudvx3k6nmgfml5wng2 --amount 0.01 --pass pass: ; done ;

after some hours the output will be this:

I will perform more test to check:

• If effectively there are 2 TX with changing balance
• if only one is succeed and the other rejected

Expected outcome was an error response explaining that shards ID's are incorrectly entered.

./wallet.sh -t transfer --from one1v6vflk65tzpdlf8p5ntpeglv3tf2d20q96k3cj --to one196vgv5zjlhv5s25hhyjfy86hg3fg5ws4pug00u --shardID 4 --toShardID 4 --amount 0.018442 --pass pass:
Using pangaea profile for wallet
panic: runtime error: index out of range

goroutine 1 [running]:
main.processTransferCommand()
        /mnt/jenkins/workspace/harmony-release/harmony/cmd/client/wallet/main.go:716 +0x13bb
main.main()
        /mnt/jenkins/workspace/harmony-release/harmony/cmd/client/wallet/main.go:247 +0x55f

transfer is complaining about shard id, even though shard id is provided

root@JSKPangaeaNode:~# ./wallet.sh -t transfer --to one1yvxwzsj7gh36774puwtw9tgfg58cfz3gkr7cfk --from one18ndk75w6nea5x7nxjhvv0xj6zfdsc7nks3f778 --amount 1.0 --shardID 0 --pass pass: Using pangaea profile for wallet Please specify the shard ID for the transfer (e.g. --shardID=0) root@JSKPangaeaNode:~# date Thu 26 Sep 2019 04:04:45 PM UTC root@JSKPangaeaNode:~# LD_LIBRARY_PATH=$(pwd) ./harmony -version Harmony (C) 2019. harmony, version v4696-pangaea-20190924.0-0-ge3030c50 (jenkins@ 2019-09-25T01:31:41+0000)

./wallet.sh -t transfer --from one1ujuzvzyfq85fhq20m5me75vjmdqcmrleql7k00 --shardID 1 --to one1kv8r6mwy9lp9tf39vrajqr96k983hv5e0l98wg --toShardID 1 --amount 1 --pass pass: Using pangaea profile for wallet Unlock account succeeded! 'pass:' Transaction Id for shard 1: 0x07e17baad490655bc84da3bcbbd99b7533a67393c03ed549e15e70b533de2d26

Always get this transaction hash 0x07e17baad490655bc84da3bcbbd99b7533a67393c03ed549e15e70b533de2d26

`
./wallet.sh -t balances
Using pangaea profile for wallet
Account 0:
Address: one1ujuzvzyfq85fhq20m5me75vjmdqcmrleql7k00
Balance in Shard 0: connection failed Balance in Shard 1: connection failed Balance in Shard 2: 162.071233617, nonce: 0
Balance in Shard 3: connection failedAccount 1:

`

there are more of those higher up, but those are in a span of at least 2 hours, still the same txid, on top of the fact that i cannot send anything out

https://explorer.pangaea.harmony.one/#/address/one155ne9v72hq4jp37vw0d0ml7e8yfxhgz6yfj377

root@pangaea:~# ./wallet.sh -t balances
Using pangaea profile for wallet
Account 0:
Address: one155ne9v72hq4jp37vw0d0ml7e8yfxhgz6yfj377
Balance in Shard 0: 98.17, nonce: 5
Balance in Shard 1: 103.0000, nonce: 0
Balance in Shard 2: 98.0000, nonce: 2
Balance in Shard 3: 1451.657669109, nonce: 1

root@pangaea:~# ./wallet.sh -t balances
Using pangaea profile for wallet
Account 0:
Address: one155ne9v72hq4jp37vw0d0ml7e8yfxhgz6yfj377
Balance in Shard 0: 98.18, nonce: 5
Balance in Shard 1: 103.0000, nonce: 0
Balance in Shard 2: 98.0000, nonce: 2
Balance in Shard 3: 1457.488167427, nonce: 1

below should have a tx count for each shard as well

root@pangaea: echo curl --silent https://explorer.pangaea.harmony.one:8888/address?id=one155ne9v72hq4jp37vw0d0ml7e8yfxhgz6yfj377 | jq
{
"address": {
"id": "one155ne9v72hq4jp37vw0d0ml7e8yfxhgz6yfj377",
"balance": 1757303956792468300000,
"txCount": 0,
"shardData": [
{
"id": "one155ne9v72hq4jp37vw0d0ml7e8yfxhgz6yfj377",
"balance": 9.818e+19,
"txs": []
},
{
"id": "one155ne9v72hq4jp37vw0d0ml7e8yfxhgz6yfj377",
"balance": 1.03e+20,
"txs": []
},
{
"id": "one155ne9v72hq4jp37vw0d0ml7e8yfxhgz6yfj377",
"balance": 9.8e+19,
"txs": []
},
{
"id": "one155ne9v72hq4jp37vw0d0ml7e8yfxhgz6yfj377",
"balance": 1458123956792468200000,
"txs": []
}
]
}
}
root@pangaea:

Using:
Build: Harmony (C) 2019. harmony, version v4696-pangaea-20190924.0-0-ge3030c50 (jenkins@ 2019-09-25T01:31:41+0000).
Build: Harmony (C) 2019. wallet, version v4662-pangaea-20190920.0-0-gc2324c41 (jenkins@ 2019-09-21T06:31:36+0000).
Panic alert stopped node from running.
goroutine 1 [running]:
github.com/harmony-one/harmony/api/service/networkinfo.New(0x14bc220, 0xc0001f4d80, 0x113aa42, 0x19, 0x0, 0x0, 0x0, 0x0, 0x10c7e80)
/mnt/jenkins/workspace/harmony-release/harmony/api/service/networkinfo/service.go:66 +0x503
github.com/harmony-one/harmony/node.(*Node).setupForClientNode(0xc000826400)
/mnt/jenkins/workspace/harmony-release/harmony/node/service_setup.go:63 +0x70
github.com/harmony-one/harmony/node.(*Node).ServiceManagerSetup(0xc000826400)
/mnt/jenkins/workspace/harmony-release/harmony/node/service_setup.go:86 +0xc6
main.createWalletNode(0xc000401f20)
/mnt/jenkins/workspace/harmony-release/harmony/cmd/client/wallet/main.go:309 +0x2b4
main.processTransferCommand()
/mnt/jenkins/workspace/harmony-release/harmony/cmd/client/wallet/main.go:729 +0x61a
main.main()
/mnt/jenkins/workspace/harmony-release/harmony/cmd/client/wallet/main.go:247 +0x55f