Without this change, users can run into the mysterious error:

lndmon_1      | read /root/.lnd/tls.cert: is a directory

As prerequisites, I have:

docker-compose 1.8.0-2
docker-ce=5:18.09.8~3-0~debian-stretch
docker-ce-cli=5:18.09.8~3-0~debian-stretch

Yet, when following instructions for Option 2: Local Usage ( https://github.com/lightninglabs/lndmon/blob/master/INSTALL.md#option-2-local-usage )

in first step (docker-compose up from the lndmon repository) I get:

ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a version of "2" (or "2.0") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/

I'm trying to setup a dashboard for our lnd node with lndmon and I am running into an issue that the lnd_peer_count metric type is a counter instead of a gauge.

To me it seems this value can go and and down and thus a gauge is better suited.

Thanks for clarifying!

	ch <- prometheus.MustNewConstMetric(
		p.peerCountDesc, prometheus.CounterValue,
		float64(len(listPeersResp)),
	)

Rather than having Prometheus calculate the avg. base fee and avg. fee rate, calculate these metrics on the lndmon exporter side. Also, add metrics for min, max, and median base fee and fee rates (also on the lndmon side).

Similar to #40 but for synced_to_graph, I made a PR for this #92

BR,

Customers requested this to be switched on by default for container images we publish (according to Ryan).Build lnd with the build tag monitoring.
From the docs:

Utilizing the monitoring build tag requires building lnd from source. To build lnd from source, follow the instructions here except instead of running make && make install, run make && make install tags=monitoring.

After upgrading to LND 0.15.1, lndmon crashes with this error.

2022-09-11 08:33:29.979 [INF] LNDMON: Starting Prometheus exporter...
2022-09-11 08:33:29.980 [INF] HTLC: Starting Htlc Monitor
2022-09-11 08:33:29.980 [INF] LNDMON: Prometheus active!
Lndmon exiting with error: WalletCollector ListUnspent failed with: invalid utxo address type 4
2022/09/11 08:33:34 Stopping Prometheus Exporter
2022-09-11 08:33:34.429 [INF] HTLC: Stopping Htlc Monitor
WalletCollector ListUnspent failed with: invalid utxo address type 4

Seems to be a known grafana issue: https://community.grafana.com/t/after-upgrade-from-grafana-5-to-6-dashboard-keeps-logging-out/14931

If lnd is restarted while lndmon is running, lndmon will fail to connect back to lnd and continue grabbing metrics. The result is that the gap in the metrics ends up much larger than the actual downtime of lnd until lndmon is restarted entirely.

It would be great if you could publish a Docker Image (just the Dockerfile as it is right now) to Docker Hub.

I am running lndmon in a Kubernetes cluster. Therefor I have to build and publish the image to a private registry before using.
Building on provisioning like with docker-compose is not possible in this setup.

A ready-to-go Docker image would make it a lot easier for me!

With the --disablehtlc switch enabled, the htlc stream is still active.

It's visible in the log outgoing, which continues to print log messages related to the htlc stream:

2024-01-21 23:27:12.097 [INF] HTLC: resolved htlc: (Chan ID=825371:1478:1, HTLC ID=553) -> (Chan ID=814836:461:0, HTLC ID=0) original forward not found
2024-01-21 23:27:22.758 [INF] HTLC: resolved htlc: (Chan ID=822334:2670:3, HTLC ID=317070) -> (Chan ID=814781:2014:1, HTLC ID=0) original forward not found
2024-01-21 23:27:40.875 [INF] HTLC: resolved htlc: (Chan ID=814836:461:0, HTLC ID=788227) -> (Chan ID=815499:1205:0, HTLC ID=0) original forward not found
2024-01-21 23:27:57.623 [INF] HTLC: resolved htlc: (Chan ID=822334:2670:3, HTLC ID=317106) -> (Chan ID=814781:2014:1, HTLC ID=0) original forward not found
2024-01-21 23:29:21.002 [INF] HTLC: resolved htlc: (Chan ID=823849:1703:5, HTLC ID=94805) -> (Chan ID=815499:1205:0, HTLC ID=0) original forward not found
2024-01-21 23:29:28.342 [INF] HTLC: resolved htlc: (Chan ID=822334:2670:3, HTLC ID=317190) -> (Chan ID=814781:2014:1, HTLC ID=0) original forward not found
2024-01-21 23:29:29.666 [INF] HTLC: resolved htlc: (Chan ID=822334:2670:3, HTLC ID=317193) -> (Chan ID=814781:2014:1, HTLC ID=0) original forward not found
2024-01-21 23:30:24.975 [INF] HTLC: resolved htlc: (Chan ID=822334:2670:3, HTLC ID=317273) -> (Chan ID=814781:2014:1, HTLC ID=0) original forward not found
2024-01-21 23:30:36.889 [INF] HTLC: resolved htlc: (Chan ID=822334:2670:3, HTLC ID=317294) -> (Chan ID=814781:2014:1, HTLC ID=0) original forward not found
2024-01-21 23:31:13.148 [INF] HTLC: resolved htlc: (Chan ID=823849:1703:5, HTLC ID=94827) -> (Chan ID=814781:2014:1, HTLC ID=0) original forward not found

Relevant code:

Is there any value in connecting to and monitoring the stream if we aren't collecting any metrics from it?

We are running lndmon inside a kubernetes pod and are experiencing strange behavior with the /health endpoint. Our liveness and/or readiness probes start failing randomly with timeout errors. We were having ~3 incidents per day but now it's close to ~30 per day.

We are using a timeout of 20s which should be more than enough for the health check.

on both v0.13.3-beta and v0.14.1-beta if nodes aren't compacted often it seems that latency starts creeping up for the /metrics endpoint. It gets pretty bad, over 10 seconds. My scrape interval is 15s and my timeout is 10s, so when nodes start to fail to return after 10s, I know it's time to restart the node.

Is there something wrong with LND / LNDMON / or my setup?

I would imagine a prometheus metric endpoint would take less than 1s to return data on average. Even after restart/compacting the node it still takes several seconds to retrieve data from /metrics

Current limits are:

    client_body_buffer_size  4K;
    client_header_buffer_size 4k;
    client_max_body_size 4k;
    large_client_header_buffers 2 4k;

which is causing the problem from this issue for large dashboards: grafana/grafana#3176

Currently it's not clear that you can start lndmon locally OR with nginx.

Also helpful: add table of contents.

I am trying to use lnd+lndmon on a rock64 board (similar to rpi, with arm64 and 4GB RAM) but Grafana only shows data points coming directly from lnd (Go Runtime + Performance dashboard). Everything supposed to come from lndmon is not there.

I noticed that when running simple queryes with PromQL I immediately got the error: "the queries returned no data for a table". Then went to Explore section and checked for up, there I can see how the lndmon process is reported to be down, which is not true.

After that I tried to get the metrics directly and I realized I was getting slow response times on the metrics endpoint (between 10s and 12s usually):

$ time curl -s --output /dev/null localhost:9092/metrics

real	0m10.717s
user	0m0.022s
sys	0m0.015s

I haven't investigated this deeply yet but the instance has more than enough Ram, and the CPU usage and load average don't look that bad.

Will try to spend more time in another moment but wanted to report soon just in case it's happening to more people.

I realized that the --help flag does not avoid the daemon from starting.

./lndmon --help
Usage:
  lndmon [OPTIONS]

prometheus:
      --prometheus.listenaddr=                       the interface we should listen on for prometheus (default: localhost:9092)
      --prometheus.logdir=                           Directory to log output (default: /Users/xavi/Library/Application Support/Lndmon/logs)
      --prometheus.maxlogfiles=                      Maximum log files to keep (0 for no rotation) (default: 3)
      --prometheus.maxlogfilesize=                   Maximum log file size in MB (default: 10)

lnd:
      --lnd.host=                                    lnd instance rpc address (default: localhost:10009)
      --lnd.network=[regtest|testnet|mainnet|simnet] network to run on (default: mainnet)
      --lnd.macaroondir=                             Path to lnd macaroons
      --lnd.tlspath=                                 Path to lnd tls certificate

Help Options:
  -h, --help                                         Show this help message

2019-08-06 11:23:38.387 [INF] LNDMON: Starting Prometheus exporter...

I'm getting the following error

error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection closed

My lightning version is newer than the version listed

lnd_1         | 2022-07-21 17:56:19.450 [INF] LTND: Version: 0.15.0-beta commit=v0.15.0-beta-141-gd9f4b36cb, build=production, logging=default, debuglevel=info
lnd_1         | 2022-07-21 17:56:19.450 [INF] LTND: Active chain: Bitcoin (network=mainnet)

I tried rolling back to v0.13.0-beta explicitly and still got the error.

Any ideas on what the problem is?

Rather than polling DescribeGraph every scrape_interval seconds, just call it once at startup, cache the state, then subscribe to graph notifications and update the cached state.

Rationale: DescribeGraph is an expensive call, so this optimization should reduce memory usage.

The routing dash added in #59 surfaces failure/success rates, channel utilization and resolution time for htlcs that lnd processes.

One thing that's missing from this dash is htlc amounts, which also provide insight into node behaviour. Since the htlc stream is ephemeral, we may want to get this information from forwarding history, rather than our stream. Some things we could surface:

• Bar Graph of amounts (as is done for resolution time)
• Average htlc size (segmented by failure/success)

Once thing we're missing right now is the ability to examine the forwarding activity of a node in real-time. We can use the streaming RPC for HTLC events SubscribeHtlcEvents to export this information to allow node operators to watch their forward/settle/fail activity from afar.

Not sure if this even possible with current configuration (although I did see a Draft PR #42 that makes me think it's not yet), but having this capability and having it documented would be very helpful, especially since conceivably lndmon's target audience is more for power uses who are more likely than normal users to have multiple nodes they'd like to manage/monitor.

An issue with lnd results in duplicate entries in the on chain transaction list.

This results in metric collection failure as multiple series with the same labels are created.

Error:

An error has occurred while serving metrics:

collected metric "lnd_tx_num_confs" { label:<name:"tx_hash" value:"xxxx" > counter:<value:11894 > } was collected before with the same name and label values

I have a docker-compose.yaml where I'm spinning up btcd, lnd, lndmon and prometheus containers.

After running docker-compose up, lndmon starts but stops with the following error logs:

lndmon                     | Lndmon exiting with error: WalletCollector ListAccountsfailed with: rpc error: code = Unknown desc = account default has unsupported key scope m/1017'/1'
lnd                        | 2022-08-07 13:58:30.881 [ERR] RPCS: [/walletrpc.WalletKit/ListAccounts]: account default has unsupported key scope m/1017'/1'
lndmon                     | 2022/08/07 13:58:30 Stopping Prometheus Exporter
lndmon                     | 2022-08-07 13:58:30.883 [INF] HTLC: Stopping Htlc Monitor
lndmon                     | WalletCollector ListAccountsfailed with: rpc error: code = Unknown desc = account default has unsupported key scope m/1017'/1'
lndmon exited with code 1

Some more info:

• my lnd Dockerfile is very similar to the default one
• running on latest master, both for lnd and lndmon.
• running on simnet
• starting lnd with --noseedbackup

Any help would be massively appreciated :)

When running lndmon against a testnet node, I'm seeing the following logs:

lndmon_1      | 2019-11-12 13:04:38.461 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:04:58.319 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:05:19.721 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:05:38.079 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:06:01.603 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:06:18.419 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:06:40.344 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:07:04.836 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:07:36.522 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:07:50.300 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)
lndmon_1      | 2019-11-12 13:08:16.318 [ERR] WALT: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (61820513 vs. 52428800)

Is this related to this issue? lightningnetwork/lnd#2374
I'm not familiar with how lndmon connects to lnd, but could it be that the connecting client need to bump the max block size?

The latest release visible in this repo is from 2019 https://github.com/lightninglabs/lndmon/releases/tag/v0.1.0.
Is this compatible with latest lnd? There have been many commits since then - is there a place where the latest official image (that is known to work with current lnd) is published? Is there a way to be notified when there is a newer release?

From "Channel State" dashboard.

Grafana allows the admin user to change its password on first Login. To reach that step the user first needs to log in as user=admin password=admin.

https://github.com/prometheus/prometheus/wiki/Default-port-allocations

In that wiki there is a list off port allactions for exporters.

This is an extract of the prometheus documentation

A user may have many exporters and Prometheus components on the same machine, so to make that easier each has a unique port number.

https://github.com/prometheus/prometheus/wiki/Default-port-allocations is where we track them, this is publicly editable.

Feel free to grab the next free port number when developing your exporter, preferably before publicly announcing it. If you’re not ready to release yet, putting your username and WIP is fine.

This is a registry to make our users’ lives a little easier, not a commitment to develop particular exporters. For exporters for internal applications we recommend using ports outside of the range of default port allocations.

lndmon is using 9092 by default. It might be good to choose something else and publicly announce it in that wiki

https://api.lightning.community/#forwardinghistory

I would be nice to have a label on all metric with the node pubkey and maybe the node alias. This way if you have more than one node running and being scraped by the same prometheus instance, you could identify and filter the metrics by the actual node id.

The sample dashboards are great, and they seem to be used as a starting point by the majority of lndmon users. Unfortunately the fully rendered dashboards are difficult to iterate on, especially in terms of contributing changes back to this repo.

One solution could be to package the dashboards and rules+alerts as a Prometheus Monitoring Mixin, which is designed to be reusable and extendable. Grafana dashboards can be generated by grafonnet for better modularity.

Would be convenient if latest release was offer on dockerhub: https://hub.docker.com/r/lightninglabs/lndmon/tags
looks like last version was 2 months ago.

We've been successfully running lndmon for a long time. Recently, we changed our bitcoind nodes, and since then, all our lndmon pods have been crashing every few hours. Lnd works fine and all the other lnd auxialiary services work fine, only lndmon keeps crashing.

Here's what I see in the logs:

Lndmon exiting with error: ChainCollector GetInfo failed with: rpc error: code = DeadlineExceeded desc = context deadline exceeded

We are using the latest lndmon version, v0.2.7.

Ive tried increasing prometheus scrape interval/timeout but lndmon keeps crashing

Any help would be much appreciated.

e.g. disk space, memory

I am trying to run lndmon in kubernetes in its own pod, separate from my lnd pod. I am using these arguments
"--prometheus.listenaddr=0.0.0.0:9092", "--lnd.host=lnd-service:10009", "--lnd.macaroondir=/root/.lnd", "--lnd.tlspath=/root/.lnd/tls.cert", "--lnd.network=regtest".

lnd-service is the service that is bound to the lnd pod.
I have verified that my tls.cert and readonly.macaroon have been mounted correctly by ssh'ing into the lndmon pod.

Looking at the logs, I see only one entry -

2020-10-16 13:26:07.260 [INF] LNDMON: Starting Prometheus exporter...

It looks like lndmon is unable to communicate with my lnd pod, but there are no errors in the log to indicate as such. When I visit the /metrics endpoint of lndmon, I do not see any lnd related metrics, which suggests that lndmon indeed is unable to talk to lnd.

The logs do not show any info whatsoever that could help in debugging the issue.

Ideally, lndmon should exit if it's unable to talk to lnd and log the errors for easier debugging.

Tried to setup lndmon with a custom signet, but was informed by the error message that signet is not yet supported.

Need help understanding what's going on with my setup or if this is a bug.

Note, currently running lndmon for many nodes using the standard bbolt/boltdb backend.
For some reason it seems like I'm getting errors when using LND with postgres.

logs:

2021-12-22 02:39:55.978 [INF] LNDMON: Starting Prometheus exporter...
2021-12-22 02:39:55.978 [INF] HTLC: Starting Htlc Monitor
2021-12-22 02:39:55.979 [INF] LNDMON: Prometheus active!
Lndmon exiting with error: GraphCollector DescribeGraph failed with: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2021-12-22 02:40:35.757 [INF] HTLC: Stopping Htlc Monitor
2021/12/22 02:40:35 Stopping Prometheus Exporter
GraphCollector DescribeGraph failed with: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Sometimes I'll just get this for the error in the logs:

lnd compatibility check failed: unable to get info for lnd node: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Hello, I use lndmon since many years but when I updated lnd to v0.16.2-beta I had :

lndmon                | 2023-04-29 20:46:34.428 [INF] LNDMON: Starting Prometheus exporter...
lndmon                | 2023-04-29 20:46:34.429 [INF] HTLC: Starting Htlc Monitor
lndmon                | 2023-04-29 20:46:34.430 [INF] LNDMON: Prometheus active!
lndmon                | Lndmon exiting with error: unknown htlc event type: *routerrpc.HtlcEvent
lndmon                | 2023-04-29 20:46:34.431 [INF] HTLC: Stopping Htlc Monitor
lndmon                | 2023/04/29 20:46:34 Stopping Prometheus Exporter
lndmon                | unknown htlc event type: *routerrpc.HtlcEvent

Any ideas ?

Regards,

H.

This isn't configurable via grafana.ini but it is technically configurable through Grafana's API.

Steps:

1. Add a docker container that depends_on Grafana's container to docker-compose
2. In this docker container, first get the desired dashboard ID through this API using the UID of the desired dashboard.
3. Use this API to update the user's home dashboard ID to the ID you just retrieved.

You should receive the response {"message":"Preferences updated"} and the container can exit after that.

More info on why step 2 is necessary: https://grafana.com/docs/http_api/dashboard/#identifier-id-vs-unique-identifier-uid

Getting this error from lndmon repeatedly in our logs:

2021/03/26 09:06:14 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)

Posts like these suggest it would be fixed by updating to a recent version of client_golang

prometheus/statsd_exporter#257
prometheus/client_golang#730
google/cadvisor#2537
thanos-io/thanos#1086