gridflare / lndpytools Goto Github PK

Collection of Python scripts for managing LND

License: MIT License

Python 100.00%

lndpytools's Introduction

lndpytools

This repository is a collection of handy Python scripts for lightning node management. More scripts will be added over time.

Available scripts

checkchannels.py     LX Identify your least effective channels. WIP.
checklndconf.py         Check an lnd.conf file for some simple recommendations.
checkmail.py         L  Check for recent keysends with metadata.
improvecentrality.py  X Find nodes that maximize centrality to improve routing.
nodeview.py             Some box&whisker plots of node data.
plotforwards.py      L  Heatmap of recent forwarding activity.
relativefees.py         Summarize a node's fee policy vs neighbours.
setfeepolicy.py      L  A basic script for setting fees and HTLC size.
watchhtlcstream.py   L  Human-readable log and CSV of HTLC events in real time.

L - Requires a connection to LND
X - CPU intensive, do not run on node hardware.

Individual documentation for each script is contained in the top section of each file.

Usage

Setup

Download the repository

$ git clone https://github.com/Gridflare/lndpytools.git

Change into the new directory

$ cd lndpytools

Install requirements

$ pip3 install -r requirements.txt --user

(The --user flag avoids conflicts with python libs installed by your system)

With LND gRPC

Most scripts use gRPC to connect to lnd directly after some setup, create the config file with

$ python3 nodeinterface.py

Double check node.conf and rerun nodeinterface, it will say Connected to node <alias> if everything is correct

With describegraph.json

Instead of connecting to lnd, some scripts can use a fresh copy of describegraph.json in the lndpytools directory. The json file will be preferred over connecting to LND where possible. Create this file from lnd with

$ lncli describegraph > describegraph.json

Running

You are now ready to run the scripts like so

$ python3 checkmail.py

Updates

You can download updates to the repo with

$ git pull

Tor proxy for 1ml.com lookups

The improvecentrality.py script uses 1ml.com for information on other nodes. To use use Tor to access 1ml.com information use the ALL_PROXY environment variable. For example:

$ ALL_PROXY="socks5://localhost:9050" improvecentrality.py.

Advanced

All scripts are based on nodeinterface.py or lnGraph.py.

NodeInterface provides an introspective thin wrapper around the LND gRPC API. It tries to be faithful to the official docs while being much less verbose. Many calls are still unsupported, work in progress.

lnGraph provides an interface for loading LN graph data from a JSON file or LND into NetworkX or iGraph. For computationally intense centrality measures, the fastcentrality module can translate a NetworkX graph into iGraph for improved performance.

lndpytools's People

Contributors

Stargazers

Watchers

Forkers

phjlljp strnglv downrangedevops bliotti velomig imirkth keblek davisv7 nullcount lightningbrie dmp1ce 22388o zx9r mariaa144 newabel

lndpytools's Issues

Consider adding proxy flag for scripts doing 1ml lookups

When querying the 1ml.com site, it would be nice to use Tor as a proxy for better privacy.

Perhaps a --tor or --proxy flag could be allowed for the improvecentrality.py script.

Error running improvecentrality.py - TypeError: '<' not supported between instances of 'DegreeView' and 'int'

Running this under Python 3.8.10 on a Windows Subsystem for Linux (WSL) Ubuntu 20.04 distro. The node is on my local network and other scripts like nodeinterface.py work fine.

Command 'python3 ./improvecentrality.py'.

Cannot connect to remote node

I have a node.conf configured to find my tls.cert and readonly.macaroon with the correct server address, but I still cannot connect. I get this error:

Traceback (most recent call last):
  File "/home/david/Workspace/lndpytools/nodeinterface.py", line 210, in <module>
    print('Connected to node', ni.getAlias())
  File "/home/david/Workspace/lndpytools/nodeinterface.py", line 190, in getAlias
    return self.GetInfo().alias
  File "/home/david/Workspace/lndpytools/nodeinterface.py", line 45, in rpcCommand
    return stubfunc(lnfunc(*args, **kwargs))
  File "/home/david/.local/lib/python3.10/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/david/.local/lib/python3.10/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1649818912.783105054","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3128,"referenced_errors":[{"created":"@1649818912.783104544","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
>

I have rpclisten=0.0.0.0:10009 set in the lnd.conf. Can you give me a suggestion on how to debug this?

Feature request: ignore fresh channels

When I run checkchannels on my node, the recommendations list contains several channels that are just a couple of days old. My own fee adjustment scripts need around 1 month to reach low levels, and I don't expect channels to perform in that time frame. As such, I'd appreciate a grace period in the computation, so that only old and useless channels show up in the checkchannels output. Let's say 2016 blocks aka 2 weeks?

checkchannels: division by zero

Traceback (most recent call last):
  File "/home/bitcoin/scripts/lndpytools/./checkchannels.py", line 129, in <module>
    printcentralitydiffs(mynode, myneighbours)
  File "/home/bitcoin/scripts/lndpytools/./checkchannels.py", line 119, in printcentralitydiffs
    f'{theircentdelta/base_centralities[pub_key]:+6.1%}',
ZeroDivisionError: float division by zero

This happened after the graph analysis, after I've seen all but one of the nodes I was expecting to see.

add describegraph process

Never heard of describegraph before, and its important for this process

Suggesting a small addition to cover that piece:

On your node, create the json file used to power the improvecentrality.py script. The describegraph.json file must reside in the same directory as the script.
$ lncli describegraph > describegraph.json

graph age check should avoid times in the future

last_update field in graph edges can have a date in the future. That is because channel policy info can be whatever a node sends. It is not checked by lnd since it would be tricky because clocks aren't always in sync

So, this dates in the future should be avoided when checking if the graph is too old (currently more than 6 hours)

I'm sending a pull request to solve this

Improvecentrality should be aware of system resources

The script currently consumes about 400 MB per CPU core, on high-core systems this may exceed available memory.

The script should check the available memory and cores and make an intelligent choice about how much to use.

improvecentrality.py : TypeError: unsupported operand type(s) for >>: 'str' and 'int'

user@Standard-PC-Q35-ICH9-2110:~/dev/lndpytools$ python3 improvecentrality.py
Loaded graph. Number of nodes: 14567 Edges: 61097
Latest update in graph: Thu Aug 12 19:22:18 2021
Simplified graph. Number of nodes: 9824 Edges: 33893
Performing analysis for xxxxxxxxx
Traceback (most recent call last):
File "improvecentrality.py", line 510, in
newchannelcandidates = selectinitialcandidates(g, filters)
File "improvecentrality.py", line 219, in selectinitialcandidates
return [n['pub_key'] for n in
File "improvecentrality.py", line 219, in
return [n['pub_key'] for n in
File "improvecentrality.py", line 220, in
filter(lambda n: filtercandidatenodes(n, graph, filters),
File "improvecentrality.py", line 215, in filtercandidatenodes
and calcavgchanage(nkey, graph, currblockheight) >= minavgchanageblks
File "improvecentrality.py", line 166, in calcavgchanage
chanblk = chan['channel_id'] >> 40
TypeError: unsupported operand type(s) for >>: 'str' and 'int'

Refactor filtered_graph

During the creation of the filtered_graph it removes all channels below 500k, EXCEPT if they are neighbors of pub_key.

This makes it impossible to re-use this same graph for other nodes (when running the main code in a loop), since it is now specific to one particular node.

It would be much better if these neighbors were not added tofiltered_graph but instead to the graph.copy() later on in the processing.

Refactor CandidateFilter

If CandidateFilter would take fast_graph as the input instead of filtered_graph, it would make the whole creation of both types of graphs unnecessary. As only fast_graph would have to be generated instead of both, saving a few seconds.

Error While Running

After installing requirements.txt, I ran nodeinterface.py and got the following error. I do not have a node.conf file.

$:~/src/lndpytools$ python3 nodeinterface.py
  File "nodeinterface.py", line 160
    while (fwdbatch := getfwdbatch(starttime, offset)):
                    ^
SyntaxError: invalid syntax
$:~/src/lndpytools$ python3 --version
Python 3.7.9

No centrality matches

I created a fresh describegraph.json file and had the script generate its own improvecentrality.conf file, yet I still don't get any matches. Very odd.

I do see matches when I check a different tool:
https://www.moneni.com/nodematch?node=03cd502dee15bb9859a6d9c56a5608c1f76894736ec4cfec906eb025b7623293dc

Loaded graph. Number of nodes: 19004 Edges: 82124
Latest update in graph: Mon Oct 11 08:21:45 2021
Simplified graph. Number of nodes: 11871 Edges: 4423
Performing analysis for BitcoinRevolution
First filtering pass found 0 candidates for new channels
Running modified farness score calculations
Completed modified farness score calculations in 1.7s
Checking 1ML availability statistics
1ML availability filter selected 0 candidates for new channels in 0.0s
No candidates found, is your graph stale?
If issue persists, delete describegraph.json and improvecentrality.conf
Traceback (most recent call last):
File "scripts/lndpytools/improvecentrality.py", line 248, in
main()
File "scripts/lndpytools/improvecentrality.py", line 239, in main
raise ValueError('No valid candidates')
ValueError: No valid candidates

No longer works since Nov 1

Was this possibly due to the LND bug?

Fetching graph data from lnd Loaded graph. Number of nodes: 17919 Edges: 77505 Latest update in graph: Tue Nov 1 06:13:26 2022 (1667301206) Traceback (most recent call last): File "/home/bacon/lndpytools/improvecentrality.py", line 257, in <module> main() File "/home/bacon/lndpytools/improvecentrality.py", line 217, in main filtered_graph = GraphFilter(graph, pub_key, graph_filters).filtered_g File "/home/bacon/lndpytools/GraphFilter.py", line 24, in __init__ self.filtered_g = self.filter_relevant_nodes() File "/home/bacon/lndpytools/GraphFilter.py", line 42, in filter_relevant_nodes raise RuntimeError('Graph is more than 6 hours old, results will be innaccurate') RuntimeError: Graph is more than 6 hours old, results will be innaccurate