sigp / lighthouse-metrics Goto Github PK

A docker-compose with Grafana + Prometheus for monitoring Lighthouse

Dockerfile 100.00%

lighthouse-metrics's Introduction

Lighthouse Metrics

Provides a docker-compose environment which scrapes metrics from Lighthouse nodes using Prometheus and presents them in a browser-based Grafana GUI.

Usage

Start a lighthouse node with $ lighthouse beacon --metrics
- The --metrics flag is required for metrics.
Bring the environment up with $ docker-compose up --build.
Ensure that Prometheus can access your Lighthouse node by ensuring it is in the UP state at http://localhost:9090/targets.
Browse to http://localhost:3000
- Username: admin
- Password: changeme
Import some dashboards from the dashboards directory in this repo:
- In the Grafana UI, go to Dashboards -> Manage -> Import -> Upload .json file.
- The Summary.json dashboard is a good place to start.

Hosting Publicly

By default Prometheus and Grafana will only bind to localhost (127.0.0.1), in order to protect you from accidentally exposing them to the public internet. If you would like to change this you must edit the http_addr in grafana.ini.

Maintenance and Contributing

The Lighthouse team has a hosted version of this stack where we do the majority of the monitoring. The dashboards in this repo are not guaranteed to be kept updated as we add/modify metrics to Lighthouse. If you're having problems, please reach out and we can update the dashboards here.

Feel free to create your own dashboards, export them and submit them here as PRs.

lighthouse-metrics's People

Contributors

Stargazers

Watchers

lighthouse-metrics's Issues

network_mode error while trying to setup via docker-compose

Command

docker-compose up -d

Error

Creating lighthouse-metrics_prometheus_1 ...
ERROR: for lighthouse-metrics_grafana_1  "host" network_mode is incompatible with port_bindings
ERROR: for lighthouse-metrics_prometheus_1  "host" network_mode is incompatible with port_bindings
ERROR: for grafana  "host" network_mode is incompatible with port_bindings
ERROR: for prometheus  "host" network_mode is incompatible with port_bindings```

docker

Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:55 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:01:25 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker-compose

docker-compose version 1.27.4, build unknown
docker-py version: 4.4.0
CPython version: 3.7.3
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Metrics are out of date

Hi team,
Most of the metrics used by Grafana dashboard files in lighthouse-metrics/dashboards/ are out of dated. Can you update them?

Update dashboard that uses AngularJS plugins

Grafana 10.4 and above deprecates the support for angular plugin (Ref: https://grafana.com/docs/grafana/latest/developers/angular_deprecation/)

Some affected dashboards include Summary.json, ValidatorMonitor.json (as reported by @Phigo on Discord). The affected graphs will show a triangle on the top right corner of the graphs. The dashboards can be updated following this table: https://grafana.com/docs/grafana/latest/developers/angular_deprecation/angular-plugins/#automatic-migration-of-plugins

Alert rules

It would be nice to have some basic alert rules for prometheus.

Summary.json Dashboard - negative number for connected clients

In the connected clients section I regularly get negative numbers. Often everything is negative except prysm.

Current status:

Either I'm not getting this chart or something is very wrong.

Add filter + variables to filter by instance

One thing I noticed that would be good is a filter per instance in each of the panels and the reason is... I run many mainnet and testnet instances of beacon nodes and validators on different machines to spread traffic and have certain redundancies and in the case of the testnet, to validate changes before going live.
Be able to filter only the instance I want, is def a plus... with the new release https://github.com/sigp/lighthouse/releases/tag/v1.1.0 where the validator monitor was added, it makes even more sense to do it

Fallback eth1 node and beacon node

It would be nice to add CLI flags for a fallback eth1 node and for the VC a fallback BN.

The logic would be that if the designated eth1 node doesn't work (i.e a local eth1 node) we attempt to connect to the fallback eth1 node.

Similarly for the VC have a backup BN if the http connection fails.