ytx1991 / chiamonitor Goto Github PK

An approach for monitoring multiple Chia harvester in one web app based dashboard

Shell 49.29% Batchfile 50.71%

chiamonitor's Introduction

ChiaMonitor

This is a Chia monitoring solution for people who need to run multiple harvesters on different hosts. It is based on Grafana (Data Visualization), Prometheus (Data Persistence & Query), and Mtail (Data Collector).

All these components support most popular OS such as Linux/Windows/MacOS.

中文用户可以参考视频：https://www.bilibili.com/video/BV1sQ4y1R741/

Features

Check the following screenshot. It provides many useful numbers, and you can set your plot size (K32 by default) and Netspace Unit (It is increasing so fast!)

How to Setup

First, checkout this repo from the Github. In short, you need to install Grafana & Prometheus on your full node farmer host and install/copy mtail on all the hosts running harvesters.

Install Mtail on all the harvesters hosts (include main host if you run harvester on it)

Add Mtail default port 3903 to your inbound firewall rule
Go to https://github.com/google/mtail/releases download and unzip latest version (not earlier than 3.0.0 rc45)
Copy the mtail (.exe) binary file to the data_collect folder in the repo folder you checked out
Copy the the data_collect folder to your Chia install folder. This is because of the script is using relative path. For Linux, it should be /your/path/to/chia-blockchain For Windows, the default path looks like: C:\Users\YOUR_USER_NAME\AppData\Local\chia-blockchain\app-1.1.5
Modify the wallet fingerprint in the run_mtail.sh / farmer_summary.bat !!!
Run ./run_mtail.sh (Linux) / ./run_mtail.bat IN the data_collect folder (Windows needs to keep the two pop-up windows open).
Open http://localhost:3903 you should see a primitive mtail website.

Note:

This requires you enabled Chia INFO level log.
Mtail needs some time to read through your log. If you want to speed up, try to delete the current debug.log.

Install Prometheus on the main host

Add Prometheus default port 9090 to your inbound firewall rule
Go to https://prometheus.io/download/ download right version based on your OS.
Unzip the file and open the folder
Copy and override the prometheus.yml by the one provided by this repo.
Replace the harvester host ip by yours.
Run Prometheus. (You need to run this at the backend in the Linux)
Check the http://localhost:9090. You should see the Prometheus portal.

Install Grafana on the main host

Add Grafana default port 3000 to your inbound firewall rule
Go to https://grafana.com/docs/grafana/latest/installation/ install right version based on your OS.
Follow the instruction on the Grafana website to run the service.
Use web browser open http://localhost:3000, you will see the Grafana portal
Create a new password for the admin (default username/password = admin/admin)
Create a new data source named as Chia, check the screenshot.
Import the dashboard json based on the language you like.

You are all set! Enjoy and give a star if you like this! Donate xch1adq8a7r5e2qkeyxgw0pml735efnsqaqk4uyyj926e4h0zg28faqsufq0k4

chiamonitor's People

Stargazers

Watchers

chiamonitor's Issues

Problem with connection through Prometheus (No data)

I, did everything as described but I receive following message in target connection of Prometheus:

ip's are good and port connection (inbound rules Firewall) are open and allowed. What else can be wrong?

Thanks in advance,
Alex.

DockerCompose or docker image

Would it be possible to create a docker-compose file with everythign that is configured and just works and then all users need to do is set up the Mtail's to send the logs to it? It could greatly simplify setup and make it less error prone

chialog.mtaill fails to compile

$ mtail --progs chialog.mtail --logs ~/.chia/mainnet/log/debug.log -compile_only

E0429 09:50:23.689628 1064409 main.go:151] Compile encountered errors:
compile failed for chialog.mtail:
chialog.mtail:8:11-24: syntax error: unexpected ID, expecting NL

Seperating the HDD from SSDs

Hi,

The dash board works beautifully but I found that the the disk usage displayed includes all harddrives which is not what we need. Ideally we only need total usage on HDD only as a total count on SSD is meaningless. Is there a way we could change that - I'm sure it's just a change of the data in the data source and the respectively dash board grid. Could you please point me in the right direction?

Many thanks!
Cheers,
Chao

无法获取数据

最新版本的 chia 钱包，已经同步区块，能显示已经挖的 xch数量全网算力也在实时更新但是其他数据没有更新

大部分数据无法显示

chia钱包的loglevel已经调整为info。
但是，看板只能展示很少的数据，看3903/metric返回的指标，其他的也确实没有。

这是什么原因。

chia 1.2升级后数据显示不正常

chia 1.2版本升级后，爆块统计出现很大偏差，请帮忙解决。升级之前显示一切正常！

Looks incredible, but, not working at all

I'm not sure if there are major problems here, or issues with the instructions, or if I'm just missing something?

I've got Prometheus and Grafana up, no problem, was already running Grafana, just had to spin up a Prometheus instance.

It looks like it is expecting a debug.log and a farmer_summary.log? But neither of these exist with a default setup. The only logging config I see in the Chia configuration pertains to running a full node, but I'm deploying this on my harvesters... so... not sure how to get them making a debug.log. Also not sure about this farmer_summary.log at all ... I tried firing up the first command from the run script in a separate screen session and redirecting it out to that file as someone mentioned in another issue here, then running mtail by itself, as, the way the start script is written, it just hangs up on the first command and never runs mtail ... that got mtail listening on on 3903 at least, but still no data on my dashboard.

You say Prometheus and Grafana need to be running on the full node? I'm, not sure why, at all? I see nothing in the config or setup instructions for them to pull any information from the full node ...

Totally lost here, and shocked to see comments in other issues saying "Got it working" as that almost looks impossible from the pieces here.

Any advice? Or plans to add a little more to the instructions and clean up big obvious issues like the fact that the startup script is fundamentally broken?

Thanks!

null

linux系统获取farmer_summary Log报错

Connection error. Check if wallet is running at 9256
Connection error. Check if harvester is running at 8560
Connection error. Check if full node is running at 8555
Connection error. Check if farmer is running at 8559
Farming status: Not available
Total chia farmed: Unknown
User transaction fees: Unknown
Block rewards: Unknown
Last height farmed: Unknown
Plot count: Unknown
Total size of plots: Unknown
Estimated network space: Unknown
Expected time to win: Unknown
Note: log into your key using 'chia wallet show' to see rewards for each key

Regex match for mtail on Windows needs fixing

Figured it out after coming across this (wants \r at end of regex): google/mtail#518

...Total (?P<total_plots>\d+) plots\r$/ {

Missing some data

Hi,

I've finally get it working partial, i am missing expect xch per day and earned and expected time to win and netspace.

Anyone know how to fix that been trying for 16 hours straight (going a bit nut now)

Thanks

grafana_dashboard_en.json missing datasource reference

I did a global find and replace on grafana_dashboard_en.json in vscode and it's working properly now.

Find: "datasource": null
Replace: "datasource": "Chia"

Make sure to create the datasource in Grafana before you import the dashboard.

第一步就出错，无法抓取数据，奇亚是最新版，1.27

Usage: chia.exe wallet show [OPTIONS]
Try 'chia.exe wallet show -h' for help.

Error: Invalid value for '-f' / '--fingerprint': Your_Wallet_Fingerprint is not a valid integer
Usage: chia.exe plotnft show [OPTIONS]
Try 'chia.exe plotnft show -h' for help.

Error: Invalid value for '-f' / '--fingerprint': Your_Wallet_Fingerprint is not a valid integer
"Fetching farmer summary ... Don't close this window"

Duration 95th Percentile 大于0.5s正常吗

好几天都不爆块了，这个值有异常吗

Average check/95th percentile check breakdown by individual hosts/harvesters?

Hi, just thinking this would be useful for some of us who have a harvester reading from NAS and other slow storage vs other harvesters just reading from internal or USB HDDs

第一步就无法拿到数据

一直无法拿到数据，是否能给个比较详细的配置步骤？

Please include download instructions for mtail

People might feel safer downloading the mtail binary from the original source.

https://github.com/google/mtail

No data in dashboard

Followed instructions for installation.

Am running harvester and full node / dashboard on same host to test.

I used SCREEN to keep run_mtail.sh in the background.

Have imported JSON but all the graphs show "No data".

Apologies - first time with Grafana / Prometheus.

Suspect mtail is not running. Have reviewed run_mtail.sh and I note that when I try to execute at the prompt ./mtail I receive a "command not found" response.

PassFilter and Challenge counts

Not sure if ti is a bug or just a confusing stat but i have over 800 plots so i have multiple plots match a challenge. The issue is that the total challengers is less than the passed filter eg 199 total and 320 passed because multiple plots can meet a challenge giving a passed filter rate of over 160%. I find this misleading and would prefer to have 2 seperate stats.

Passed Filter which is when 1 or more plots pass the filter for a challenge. this would never go over 100%
Average plots that pass the filter per challenge.

Also it would be great to be able to get the check duration etc per harvester or to have min/avg/max in a bar form per harvester so i can find my slow ones

Possible to see how many new plots are made per day on each harvester

Hi,

I dont know if its possible, but it would be nice to see the new plots on each harvester that you make every day.

Farm 高度为0

Farm 高度为0，其他数据正常……不知道修改哪里。。。。ORZ...

When i open my dash the average challenges go down

Hi,

Anyone noticed when you open the dashboard that the average challenges go down?
https://www.mobilrent.be/dash.png

Do you guys run the dash on a farmer or on a diffrent pc?

只有全网算力，获取不到其他数据是什么问题？

farmer_summary.log 已获取内容：
Farming status: Farming
Total chia farmed: 22.00000008
User transaction fees: 8e-08
Block rewards: 22.0
Last height farmed: 444177
Plot count: 3554
Total size of plots: 351.794 TiB
Estimated network space: 25344.576 PiB
Expected time to win: 2 weeks and 2 days
Note: log into your key using 'chia wallet show' to see rewards for each key

只有全网算力，获取不到其他数据是什么问题？哪里没设置好？

提交一个翻译错误

  "pluginVersion": "7.5.4",
  "targets": [
    {
      "exemplar": true,
      "expr": "max(last_farm_height)",
      "interval": "",
      "legendFormat": "",
      "refId": "A"
    }
  ],
  "title": "上次收割高度",
  "type": "stat"
},

无法显示farm summary数据

安装在ubuntu20.04
run_mtail.sh 中farmsummary.log 数据获取命令目录不正确，修改路径后,命令正常执行，farmsummary.log 中数据正常。