etclabscore / classic-geth-supervisor.sh Goto Github PK

I just turned this service off, since it depended on classic geth's mlog feature.

Rel http://etcstatus.live

Just leaving this here for posterity.

[Unit]
Description=IsaacsClassicSupervisor

[Service]
Type=simple
User=root
Restart=always
RestartSec=3
TimeoutStopSec=10min
ExecStart=/bin/sh -c 'tail -F /mnt/chaindata_vol/.ethereum-classic/mainnet/mlogs/geth.log | /root/classic-geth-supervisor.sh/collect/mlog-event-pipe.sh /root/classic-geth-supervisor.sh/analyze/supervise.sh /root/classic-geth-supervisor.sh/alert/isaacecho.sh'

[Install]
WantedBy=default.target

[Unit]
Description=IsaacsClassicSupervisor_Emailer

[Service]
Type=simple
User=root
Restart=always
RestartSec=3
TimeoutStopSec=10min
ExecStart=/bin/sh -c '/root/.env && /root/classic-geth-supervisor.sh/analyze/supervise.sh /root/classic-geth-supervisor.sh/alert/isaacemail.sh; echo "sleeping 60..." && sleep 60'

[Install]
WantedBy=default.target

currently alert status's are built with $((50-M_margin_delta)) where M_margin_delta is also used for watching for etherbase share variation. this doesn't make sense.

further, variation alert-limits should have floors, to not alert when an etherbase normally at, say, 3%, reaches 10% in the last 30minutes. we should care more about larger miners than smaller ones, and schedule variation alert parameters as a function of relative share.

currently the logic just tails latest blocks, assuming that the client is fully synced and importing blocks, and the content of it's output assumes the output is updated in realtime (or only relevant to a one-off run).

including block identifiers like number and hash can help to inform the accuracy of those assumptions.

when you start from scratch, the analyzer will overreact b/c it's comparing latest 1 or 2 blocks against a total aggregate of only 1 or 2 blocks, so relative comparisons are invalid.

to solve, should cause analyzer to incorporate existing amount of "aggregate" log data (eventually ~5000 lines) better into it's calculations

currently supervisor.sh only reports known "bad" cases, when it's analyzers returns an undesirable result for any of it's analyses. this is problem because you can't see what's actually OK, and can't understand a context of what's actually being checked without reading the code.

to resolve this, should refactor analysis functions to return a standard-formatted output, reporting their exit code 0 if ok, >0 if not, and describe in text the kind of analysis they did and the dataset they ran against.

that way the generated "analysis report" includes a full context of tests that were run and BOTH the positive and negative outcomes of each.

should just use printf

currently there are quite a few arbitrary hardcoded analysis params. they should be exported and named in ./env.sh for sake of configurability and clarity.