Recover funds from crashed core lightning node. about lightning HOT 11 OPEN

Can you share the logs when calling emergencyrecover? That might show us where it fails. Also adding @adi2011 as the author of the recovery plugin.

from lightning.

Please find the details below :
I ran emergencyrecover and after that the getlog commands. Below is the info from getlog :

{
"created_at": "1710693901.278881343",
"bytes_used": 3846881,
"bytes_max": 10485760,
"log": [
{
"type": "SKIPPED",
"num_skipped": 376
},
{
"type": "INFO",
"time": "13.808906673",
"source": "lightningd",
"log": "--------------------------------------------------"
},
{
"type": "SKIPPED",
"num_skipped": 937
},
{
"type": "UNUSUAL",
"time": "19.311173752",
"source": "plugin-bookkeeper",
"log": "Snapshot balance does not equal ondisk reported 0msat, off by (+0msat/-0msat) (account wallet) Logging journal entry."
},
{
"type": "SKIPPED",
"num_skipped": 9
}
]
}

from lightning.

Additional Info for getlog debug :

{
"created_at": "1710694205.592952807",
"bytes_used": 4241774,
"bytes_max": 10485760,
"log": [
{
"type": "SKIPPED",
"num_skipped": 161
},
{
"type": "INFO",
"time": "5.488185299",
"source": "plugin-bcli",
"log": "bitcoin-cli initialized and connected to bitcoind."
},
{
"type": "SKIPPED",
"num_skipped": 120
},
{
"type": "DEBUG",
"time": "5.994206501",
"source": "plugin-bookkeeper",
"log": "Setting up database at sqlite3://accounts.sqlite3"
},
{
"type": "SKIPPED",
"num_skipped": 576
},
{
"type": "DEBUG",
"time": "8.896661729",
"source": "lightningd",
"log": "Adding block 835094: 000000000000000000020a59df695df10af0f29757c7221c4b1faf5fe1510ecf"
},
{
"type": "SKIPPED",
"num_skipped": 269
},
{
"type": "DEBUG",
"time": "10.607919907",
"source": "plugin-bcli",
"log": "Log pruned 134 entries (mem 10536455 -> 2427548)"
},
{
"type": "SKIPPED",
"num_skipped": 43
},
{
"type": "DEBUG",
"time": "10.808193457",
"source": "lightningd",
"log": "Adding block 835099: 00000000000000000000182aa974d676e8e578eee4882a6f1c5dc2cc30e5005e"
},
{
"type": "SKIPPED",
"num_skipped": 84
},
{
"type": "DEBUG",
"time": "11.336942878",
"source": "plugin-bcli",
"log": "Log pruned 135 entries (mem 10493227 -> 2033985)"
},
{
"type": "SKIPPED",
"num_skipped": 34
},
{
"type": "DEBUG",
"time": "11.521482020",
"source": "lightningd",
"log": "Adding block 835101: 00000000000000000002d17bfbc3bbaa7a2e0ea1152d8bbe9ea6fec410ec82a1"
},
{
"type": "SKIPPED",
"num_skipped": 2
},
{
"type": "DEBUG",
"time": "11.593491937",
"node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1",
"source": "chan#4377",
"log": "Got depth change 0->117554 for 20a2f38f5cc37c877a0ead2b52c4a90965a6fecb95ddef8d632bf6fdf4a1d62a"
},
{
"type": "DEBUG",
"time": "11.593510787",
"node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1",
"source": "chan#4377",
"log": "Funding tx 20a2f38f5cc37c877a0ead2b52c4a90965a6fecb95ddef8d632bf6fdf4a1d62a depth 117554 of 3"
},
{
"type": "DEBUG",
"time": "11.593568348",
"node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1",
"source": "chan#4377",
"log": "Funding tx 20a2f38f5cc37c877a0ead2b52c4a90965a6fecb95ddef8d632bf6fdf4a1d62a confirmed, but peer disconnected"
},
{
"type": "DEBUG",
"time": "11.593580868",
"node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1",
"source": "chan#4377",
"log": "attempting update blockheight 2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221"
},
{
"type": "DEBUG",
"time": "11.593582144",
"node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1",
"source": "chan#4377",
"log": "chain not synced, not updating blockheight"
},
{
"type": "DEBUG",
"time": "11.593609273",
"source": "lightningd",
"log": "sendrawtransaction: 020000000001012ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a22001000000000c46af80028f62000000000000160014c02712a2d3f09e7efadfce3d4c76501ba53ed561650426000000000022002059cfd4bad3f6c8d6899b5bae169de4e3e3ff4a1e2a158a1bfd2f038e2ac50896040047304402206c3afe012479a046fe380785a64c0cccc2e928915a979ba9cb5453daa58acd8402200150871c063b95a625b9c887890d4308a617798f8bcba64e4896398780bf726c0147304402203bffba3835de33c37ae26d3b496e6625b54dbd39194c566b4b3138289604ef1602207f6469897e7971961d165280a2c4cd14bfde84849d5697b1aa207db96bcc8b7c014752210207c263af1717e8df4790d5dc3d433aefa98c0169286ad78ff46b17430d4bcdfa210239b38299018f11e10ec3deb11534da31d8cfc650b056fc70afc92299963a5fa452ae5835c120"
},
{
"type": "SKIPPED",
"num_skipped": 1
},
{
"type": "DEBUG",
"time": "11.597354270",
"source": "gossipd",
"log": "REPLY WIRE_GOSSIPD_NEW_BLOCKHEIGHT_REPLY with 0 fds"
},
{
"type": "SKIPPED",
"num_skipped": 1
},
{
"type": "DEBUG",
"time": "11.599591378",
"source": "plugin-bcli",
"log": "sendrawtx exit 0 (bitcoin-cli -rpcconnect=10.21.21.8 -rpcport=8332 -rpcuser=... -stdinrpcpass sendrawtransaction 020000000001012ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a22001000000000c46af80028f62000000000000160014c02712a2d3f09e7efadfce3d4c76501ba53ed561650426000000000022002059cfd4bad3f6c8d6899b5bae169de4e3e3ff4a1e2a158a1bfd2f038e2ac50896040047304402206c3afe012479a046fe380785a64c0cccc2e928915a979ba9cb5453daa58acd8402200150871c063b95a625b9c887890d4308a617798f8bcba64e4896398780bf726c0147304402203bffba3835de33c37ae26d3b496e6625b54dbd39194c566b4b3138289604ef1602207f6469897e7971961d165280a2c4cd14bfde84849d5697b1aa207db96bcc8b7c014752210207c263af1717e8df4790d5dc3d433aefa98c0169286ad78ff46b17430d4bcdfa210239b38299018f11e10ec3deb11534da31d8cfc650b056fc70afc92299963a5fa452ae5835c120) "
},
{
"type": "SKIPPED",
"num_skipped": 1
},
{
"type": "DEBUG",
"time": "11.600410526",
"source": "plugin-bookkeeper",
"log": "account wallet has balance 0msat"
},
{
"type": "UNUSUAL",
"time": "11.600419530",
"source": "plugin-bookkeeper",
"log": "Snapshot balance does not equal ondisk reported 0msat, off by (+0msat/-0msat) (account wallet) Logging journal entry."
},
{
"type": "DEBUG",
"time": "11.600458255",
"source": "plugin-bookkeeper",
"log": "account 2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221 has balance 2491765413msat"
},
{
"type": "DEBUG",
"time": "11.600461788",
"source": "plugin-bookkeeper",
"log": "Snapshot balances updated"
},
{
"type": "SKIPPED",
"num_skipped": 15
}
]
}

from lightning.

It seems your channel with 02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1 is already in AWAITING_UNILATERAL. Does it stub more channels in listpeers after running emergencyrecover?... @pratapc65

from lightning.

@adi2011 : I have closed all the channels and do not have other peer connections before the node recovery.. The channel with 2aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1 is in "AWAITING_UNILATERAL" for the past 2 months.
The logs shows the raw transaction and how can I resend the transaction to mempool via bitcoin core .

"sendrawtransaction: 020000000001012ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a22001000000000c46af80028f62000000000000160014c02712a2d3f09e7efadfce3d4c76501ba53ed561650426000000000022002059cfd4bad3f6c8d6899b5bae169de4e3e3ff4a1e2a158a1bfd2f038e2ac50896040047304402206c3afe012479a046fe380785a64c0cccc2e928915a979ba9cb5453daa58acd8402200150871c063b95a625b9c887890d4308a617798f8bcba64e4896398780bf726c0147304402203bffba3835de33c37ae26d3b496e6625b54dbd39194c566b4b3138289604ef1602207f6469897e7971961d165280a2c4cd14bfde84849d5697b1aa207db96bcc8b7c014752210207c263af1717e8df4790d5dc3d433aefa98c0169286ad78ff46b17430d4bcdfa210239b38299018f11e10ec3deb11534da31d8cfc650b056fc70afc92299963a5fa452ae5835c120"

from lightning.

https://mempool.space/tx/a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24

Your transaction is already in the mempool, it hasn't closed for so long because the feerate seems very low.

from lightning.

@pratapc65 in the past when I had a forced closed chan with too low of a fee I had to use a "transaction accelerator" to pay miners out of band to include the transaction. There is viaBTC tx accelerator (they are expensive) and mempool.space has a tx accelerator service but I don't know if their service is active yet.

I have fell victim to stuck transactions before and even tried to replace unconfirmed transactions. Despite the popular myth, you cannot double spend transactions because nodes will reject the conflicting transaction. And transactions do not drop from the mempool. This claim is based off my experience with stuck transactions and experimentation from such scenarios. If I'm wrong, then someone should correct me on this. I think your best bet is to try to pay a miner out of band to include this transaction. viaBTC does accept a limited number of free entries for their accelerator service each hour. You could try with that first and if no luck then you can try the paid service.

from lightning.

Maybe this transaction can be RBF'ed or CPFP? But if itis a unilateral close that might not be an option.

from lightning.

Hello @nakoshi-satamoto : Thanks for the inputs. I tried to accelerate the transaction a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24 viaBTC free and it mentioned the fee was low.
When I tried the paid service it mentioned the transaction was not recognized.

@cdecker ; @adi2011 : Is there a way to bump up the transaction fee via the btc core and rebroadcast .

from lightning.

@pratapc65 you can use withdraw RPC to initiate CPFP for this txn.

Please set the utxo argument appropriately.

from lightning.

@adi2011 ; Can you help me with the command. How do I know my nodes utxo that is stuck in mempool.

My nodes listfunds
{
"outputs": [],
"channels": [
{
"peer_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1",
"connected": false,
"state": "AWAITING_UNILATERAL",
"channel_id": "2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221",
"short_channel_id": "717548x1017x1",
"our_amount_msat": 2491765413,
"amount_msat": 2516997000,
"funding_txid": "20a2f38f5cc37c877a0ead2b52c4a90965a6fecb95ddef8d632bf6fdf4a1d62a",
"funding_output": 1
}
]
}

from lightning.