nhatfield / hnt_monitor Goto Github PK
View Code? Open in Web Editor NEWProduce metrics from miner apis and the Helium blockchain. We can use the metrics to diagnose, alert, and prevent poor mining performance.
Produce metrics from miner apis and the Helium blockchain. We can use the metrics to diagnose, alert, and prevent poor mining performance.
i saw there is a closed issue with the same error:
Step 9/9 : ENTRYPOINT ["./entrypoint"]
---> Using cache
---> e05d02cdda97
Successfully built e05d02cdda97
Successfully tagged hnt_monitor:latest
Recreating 992ae87577fa_prometheus ...
Recreating 6c8e55c4bf64_grafana ... error
Recreating 5b1c29c87d20_prometheus_pushgateway ...
Recreating 992ae87577fa_prometheus ... error
ERROR: for 5b1c29c87d20_prometheus_pushgateway Cannot create container for service prometheus_pushgateway: invalid IPv4 address: 10.30.0.04
ERROR: for 992ae87577fa_prometheus Cannot create container for service prometheus: invalid IPv4 address: 10.30.0.03
ERROR: for grafana Cannot create container for service grafana: invalid IPv4 address: 10.30.0.02
ERROR: for prometheus_pushgateway Cannot create container for service prometheus_pushgateway: invalid IPv4 address: 10.30.0.04
ERROR: for prometheus Cannot create container for service prometheus: invalid IPv4 address: 10.30.0.03
ERROR: Encountered errors while bringing up the project.
root@store:/home/steam/hnt_monitor-7.10.0#
in the closed issue someone say it caused by version of docker but i use the latest:
root@store:/home/steam/hnt_monitor-7.10.0# docker --version
Docker version 20.10.16, build aa7e414
Getting the following error after installing the stack.
2021-12-01T02:52:34 [INFO] [collector.info] [local] - getting hotspot info data for [local (115....k21)]
2021-12-01T02:52:34 [INFO] [collector.rewards] [local] - getting hotspot rewards data for [local (115....k21)]
2021-12-01T02:52:34 [INFO] [collector.bobcat.miner] [local] - getting bobcat miner data for [local (192.168.88.67)]
2021-12-01T02:52:34 [WARN] [collector.bobcat.status] [local] - bad response from the api gateway while retrieving status data for 192.168.88.67. Retrying in 5 seconds...
2021-12-01T02:52:50 [WARN] [collector.rewards] [local] - bad response from the api gateway while retrieving rewards data for 115....k21. Retrying in 5 seconds...
The bobcat miner dashboards have data but anything with the helium network is not being updated because of the api gateway error. I truncated the key but it is the full key of course in the setup. Unable to find any help on this on the web. I'm sure its something simple.
System is ubuntu 18.04. thanks.
Centos does not have official support for the epel repo, so this workaround has to be done in order to build the docker image for ARM platforms.
This is not related to the grafana but the prometheus unless the API is giving back false data.
First of:
hnt_miner_reward_earned_1d is correct
however the 7d and 30d showing the exact same value which would be only true if I would just setup the miner not long ago.
How are these windows working then? Like my current daily reward is 0.06 HNT, according to the explorer my 7d is 0.5 HNT and the 30D is 1.8 HNT whilst with your metrics I get:
hnt_miner_reward_earned_1d 0.06
hnt_miner_reward_earned_7d 0.06
hnt_miner_reward_earned_30d 0.06
Now things gets more odd with the hnt_miner_reward_1d 7d 30d, I don't know what this supposed to display but I get 5 for all, even my overall HNT earnings are not 5.
Need a build pipeline to test updates
Raspberry Pi 4 with newest OS.
Error Message:
Complete!
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
Successfully tagged hnt_monitor:latest
Creating grafana ... error
Creating prometheus ...
Creating prometheus_pushgateway ...
Creating prometheus_pushgateway ... error
Creating prometheus ... error
ERROR: for prometheus_pushgateway Cannot create container for service prometheus_pushgateway: invalid IPv4 address: 10.30.0.04
ERROR: for prometheus Cannot create container for service prometheus: invalid IPv4 address: 10.30.0.03
ERROR: for grafana Cannot create container for service grafana: invalid IPv4 address: 10.30.0.02
ERROR: for prometheus_pushgateway Cannot create container for service prometheus_pushgateway: invalid IPv4 address: 10.30.0.04
ERROR: for prometheus Cannot create container for service prometheus: invalid IPv4 address: 10.30.0.03
Fix: remove the zero from the last ips like 04 -> 4 and docker-compose -f hnt_monitor.yml up -d --build
Pls fix in next version.
How do you get the block height and cpu core temperatures per miner.
The block height only available as a HNT metric coming from their system not the nodes. CPU is not available at all
Mine is nebra I have added this to the hnt_monitor.yml
environment:
DO_NOT_REMOVE: "setup"
HNT_HOTSPOT_MONITOR: "true"
HNT_NEBRA_MONITOR: "true"
Also what is the point of having the internal LAN ips defined for Nebras?
I have 4 hotspot addresses in HNT_HOTSPOT_ADDRESSES. 3 Rak MNTD and 1 Sensecap. I also have the Sensecap API setup. This morning I went to check my dashboard and now see two additional miners in my metrics: shambolic-walnut-chimpanzee and zealous-marigold-mouse. This is day 5 with the stack running and the first time I've seen them.
I checked these out on Helium Explorer just to cross references their addresses to my config and verify their wallet addresses weren't similar.
is this code still running with the Solana Blockchain?
Block height for Sensecap seems to show the blockchain height, not the synced height of a hotspot.
In the API Docs (https://status-api-doc.sensecapmx.cloud/) in the device detail request there's a separate field for Block height of device (called 'height') and another for the height of the blockchain (called 'totalHeight'), looks like the monitoring might be reading the latter field.
Add prometheus, prometheus push gateway, grafana, and the hnt_monitor as a single docker-compose file.
I have the Docker version running on my synology diskstation, but unfortunately I can't get to the panel
Application endpoint
grafana http: // localhost: 3000
prometheus http: // localhost: 9090
prometheus pushgateway http: // localhost: 9091
date | stream | content |
---|---|---|
2021-10-03 09:20:50 | stdout | 2021-10-03T09:20:50 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:20:49 | stdout | 2021-10-03T09:20:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:19:50 | stdout | 2021-10-03T09:19:50 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:19:50 | stdout | 2021-10-03T09:19:50 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:19:49 | stdout | 2021-10-03T09:19:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:19:49 | stdout | 2021-10-03T09:19:49 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:18:50 | stdout | 2021-10-03T09:18:50 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:18:49 | stdout | 2021-10-03T09:18:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:17:50 | stdout | 2021-10-03T09:17:50 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:17:50 | stdout | 2021-10-03T09:17:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:17:49 | stdout | 2021-10-03T09:17:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:17:49 | stdout | 2021-10-03T09:17:49 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:16:49 | stdout | 2021-10-03T09:16:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:16:49 | stdout | 2021-10-03T09:16:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:15:50 | stdout | 2021-10-03T09:15:50 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:15:49 | stdout | 2021-10-03T09:15:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:15:49 | stdout | 2021-10-03T09:15:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:15:49 | stdout | 2021-10-03T09:15:49 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:14:49 | stdout | 2021-10-03T09:14:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:14:49 | stdout | 2021-10-03T09:14:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:13:50 | stdout | 2021-10-03T09:13:50 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:13:49 | stdout | 2021-10-03T09:13:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:13:49 | stdout | 2021-10-03T09:13:49 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:13:49 | stdout | 2021-10-03T09:13:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:12:49 | stdout | 2021-10-03T09:12:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:12:49 | stdout | 2021-10-03T09:12:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:11:50 | stdout | 2021-10-03T09:11:50 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:11:49 | stdout | 2021-10-03T09:11:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:11:49 | stdout | 2021-10-03T09:11:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:11:49 | stdout | 2021-10-03T09:11:49 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:10:49 | stdout | 2021-10-03T09:10:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:10:49 | stdout | 2021-10-03T09:10:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:09:50 | stdout | 2021-10-03T09:09:50 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:09:49 | stdout | 2021-10-03T09:09:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:09:49 | stdout | 2021-10-03T09:09:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:09:49 | stdout | 2021-10-03T09:09:49 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:08:49 | stdout | 2021-10-03T09:08:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:08:49 | stdout | 2021-10-03T09:08:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:07:50 | stdout | 2021-10-03T09:07:50 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:07:49 | stdout | 2021-10-03T09:07:49 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:07:49 | stdout | 2021-10-03T09:07:49 [INFO]: [system.cleanup] - starting data cleanup service |
2021-10-03 09:07:49 | stdout | 2021-10-03T09:07:49 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:07:49 | stdout | 2021-10-03T09:07:49 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:06:49 | stdout | 2021-10-03T09:06:49 [INFO]: [main] - setting initialize to false |
2021-10-03 09:06:40 | stdout | 2021-10-03T09:06:40 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [main] - allowing collections to finish before processesing data |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [main] - removing stale locks |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [main] - initializing hnt monitor |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [trace]: false |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [sensecap_serial_numbers]: |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [sensecap_monitor]: false |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [sensecap_api_key]: |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [prometheus_pg_host]: http://localhost:9091 |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [project]: hnt_miner |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [nebra_monitor]: false |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [nebra_ips]: |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [longap_monitor]: false |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [longap_addresses]: |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [logpath]: /dev/ |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [logfile]: stdout |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [hotspot_url]: https://api.helium.io/v1/hotspots |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [hotspot_monitor]: false |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [hotspot_addresses]: |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [helium_monitor]: true |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [debug]: false |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [bobcat_monitor]: false |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [bobcat_ips]: |
2021-10-03 09:06:39 | stdout | 2021-10-03T09:06:39 [INFO]: [startup.config] - Configuring [blocks_url]: https://api.helium.io/v1/blocks |
2021-10-03 09:04:10 | stdout | 2021-10-03T09:04:10 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:04:10 | stdout | 2021-10-03T09:04:10 [INFO]: [etl.block.height] - Sending data to prometheus pushgateway |
2021-10-03 09:04:10 | stdout | 2021-10-03T09:04:10 [INFO]: [system.cleanup] - starting data cleanup service |
2021-10-03 09:04:10 | stdout | 2021-10-03T09:04:10 [INFO]: [etl.block.height] - processing data |
2021-10-03 09:04:10 | stdout | 2021-10-03T09:04:10 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:03:10 | stdout | 2021-10-03T09:03:10 [INFO]: [main] - setting initialize to false |
2021-10-03 09:03:01 | stdout | 2021-10-03T09:03:01 [INFO]: [collector.height] - Block height data ready to process |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [collector.height] - getting block height data |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [main] - allowing collections to finish before processesing data |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [main] - removing stale locks |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [main] - initializing hnt monitor |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [trace]: false |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [sensecap_serial_numbers]: |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [sensecap_monitor]: false |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [sensecap_api_key]: |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [prometheus_pg_host]: http://localhost:9091 |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [project]: hnt_miner |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [nebra_monitor]: false |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [nebra_ips]: |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [longap_monitor]: false |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [longap_addresses]: |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [logpath]: /dev/ |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [logfile]: stdout |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [hotspot_url]: https://api.helium.io/v1/hotspots |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [hotspot_monitor]: false |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [hotspot_addresses]: |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [helium_monitor]: true |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [debug]: false |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [bobcat_monitor]: false |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [bobcat_ips]: |
2021-10-03 09:03:00 | stdout | 2021-10-03T09:03:00 [INFO]: [startup.config] - Configuring [blocks_url]: https://api.helium.io/v1/blocks |
Right now the data grows indefinitely, until manually purged. The system should handle this with some user adjustable thresholds
Add the reward earnings metrics as a new collector
Only options were Bobcat, LongAP, Nebra, and SenseCap. I did not see a RAK option. Is it there and I'm just not seeing it?
Great project! Already using it for some days. :-)
Would be nice if all the SenseCAP API provided information would be pulled.
E.g. FW version, Fan-Status, ...
And if we could have a helium wallet overview, e.g. total balance of wallet, daily income
When first launching the services they should backfill 14 days worth of data. This should only happen on initial setups that have no existing data
The docker.io needs to be installed as root but if the parts that don't need root could be installed as none root it would make things more secure.
Possibly others might need this same treatment. Currently when the api does not respond, it produces 0's in the UI. We should retry the api query before sending null values
Hi,
I don't have access to my miners because they are in a different location. But I would like to grab some data via the API. I came across your project and it seems awesome. I was able to set it up in docker and I define my two hotspots but the beacon and challenges are reporting like zero. Are those required for local IP or API?
So my witnesses are reporting back zero. Kind of odd.
I basically just want to use your project to scrape API data and show it in grafana.
hope anyone can help me why reward values are wrong for me
hnt_miner_reward_earned_15m to 30 days is 0 or either 1
hnt_miner_reward_earned_1d is 0.0181 same as 7d and 30d... all 3 values are wrong
other stuff is correct like CPU temp, usage, reward scale and so on but rewards are very wrong
We should create a database for this and store the data locally. That would allow us to not tax the API as often, and also return small data sets of only 15 minutes
Thinking about a backend like cassandra with modification to the scripts. Then we can post the same data while also not hitting the API as hard
Create a log function so it is easier to send logged data. Right now each service has to tailor the full log message multiple times.
Add the block height to collected metrics
It looks like while Installing on Rocky Linux docker compose does not get found in the repos.
Nice work around is the following:
Source: https://tastethelinux.com/install-docker-and-docker-compose-on-rocky-linux-8/
Prometheus limitations make monitoring multiple hosts cumbersome. Instead of allowing multiple miners to push the same metric with different tags, you have to create a new metric for each miner. Statsd will allow a single metric from multiple sources as long as the tags are different
The instance name is hard coded to custom-rusty-porcupine
Need test for the build confidence
Would be nice to add pisces also. Did not research if they have a public api.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.