A spike in this metric allows us to detect many old connections being dropped at the same time. Currently, most connections dropped are very short-lived (a second or less) but we can't differentiate them from the long-lived (and probably more important) connections

It should be possible to disable certain tracepoint groups (i.e. connection tracepoints) with a command line argument.

Also, being able to set a custom port with a command line argument would be good.

We could use the clap crate for that.

This should be available and can be added to:

Posting potential cleanups and small refactors that should be made at some point here:

The logger tool is quite spammy currently as it prints all events (p2p message, connections, addrman (if enabled), validation, and mempool). It would be good to have a feature where we can pass a command line argument to enable only the desired event types. E.g. passing --log-p2pmsgs would only log p2p messages. Passing --log-p2pmsgs --log-connections would log p2p messages and p2p connections.

Both these + 1's are outdated since bitcoin/bitcoin#25832 (comment)

All other tools also use clap too.

Currently, most tools and the extractor don't have proper logging. We might want to:

• move the simple_logger dependency into shared to have only one place to upgrade and to be sure we're only using one version across all tools
• replace all println! macros with their respective log:<level>! macros
• insert SimpleLogger::new().init() at startup
• (optionally) make logging levels configurable through clap
• ...

Currently, the shared::wrapper::Wrapper field wrap isn't really expressive of what it is. It's a peer-observer event and should be renamed.

The current Grafana dashboards show a the raw numbers from Prometheus (via the metrics) tool. Anomaly detection and alerting is not yet implemented.

For example:

Here, an anomaly could be a sudden drop in inbound peers connected to one or more peers as in https://b10c.me/observations/05-inbound-connection-flooder-down/. To detect this, a Z-score could be used. If the z-score is above a certain threshold, send an alert.

Here, a spike in outbound and (inbound too) address messages across all nodes could indicate an anomaly. Here a Z-score could be used. Maybe there are other possible ways to explore which can be used to detect anomalies.

This issue can be used for discussion and brainstorming.

bpftime (https://github.com/eunomia-bpf/bpftime) might allow us to trace in userspace without needing to interact with the kernel. bpftime supports USDT. This means, no more starting with privileges required?

Would be good to explore this!

The prometheus metrics endpoint currently lists a counter for each address that ever closed an opened a connection to us. This can grow quite large and scraping this can become resource intensive.

A few options:

• Write a custom text encoder that only includes address metrics if we saw that address more than e.g. 5 times in total. Downside: We potentially don't see long-running connections that only ever connected once.
• Only include the subnet and not the full address. This should reduce the number of lines drastically
• Use compression in the metrics webserver
• Reset metrics every 24h in some way:
  • custom registry for the address metrics?

Complains about "Could not determine the UTC offset on this system."

Tools, like metrics, should retry for a while.

Currently, rust bitcoin v0.29.2 is used. Current latest release is https://github.com/rust-bitcoin/rust-bitcoin/releases/tag/bitcoin-0.30.2

Currently, there is no feature to detect spy peers/nodes, which has been discussed here.

Some important anomalies to consider are:

• When INV is sent to the peer, but the peer doesn't send GETDATA to our node
• Spy nodes never send us INV for transactions we sent

We only handle INV, GETDATA, and TX p2p messages. We need to maintain a shared state – one entry (IP address + Port) for each connection with the number of INV, GETDATA, and TX sent and received. Additionally, spy peers will close the connection, so we also need to account for handling closed connections. Once all this is implemented, we can have some stats on normal/spy peers/nodes.

One approach could be to find the ratio of INV/GETDATA for each peer, but there might be other heuristics to detect peer identity.

A reference implementation has been done at the following URL: https://github.com/i-am-yuvi/peer-observer/tree/spy-detection.

At least some minimal description of what the tool does and a usage example.

By recording and then replaying some of the nanomsg/protobuf communication that the extractor publishes, we can write regression test for the tools in the CI. This would be useful to have!

See also: #25 (comment)

multiple tools currently each define a dependency on the nng (nanomsg) crate. We could move this to the shared module as it is needed for every tool