metwork-framework / mfsysmon Goto Github PK

View Code? Open in Web Editor NEW

0.0 3.0 2.0 432 KB

metwork/mfsysmon module

Home Page: http://metwork-framework.org/mfsysmon

License: BSD 3-Clause "New" or "Revised" License

Makefile 7.12% Shell 46.75% Python 46.13%

metwork integration-level-5 autoreadme github-actions

mfsysmon's Introduction

mfsysmon

Status (master branch)

What is MetWork FrameWork?

MetWork Framework is an opensource system for building and managing production grade applications or micro-services (from development to operations).

What is MFSYSMON?

MetWork Framework is an open source system for building and managing production grade applications or micro-services.

MFSYSMON is the Metwork Framework "SYStem MONitoring" module. It is a little agent which collects some system metrics (RAM, CPU, Disk, Network...) and send them to a MFADMIN module (which stores them).

Cheatsheet

A cheatsheet for this module is available here

Reference documentation

(for master (development) version), see this dedicated site for reference documentation.
(for latest released stable version), see this dedicated site for reference documentation.

For very specific use cases, you might be interested in reference documentation for integration branch.

And if you are looking for an old released version, you can search here.

Installation guide

See this document.

Configuration guide

See this document.

Contributing guide

See CONTRIBUTING.md file.

Code of Conduct

See CODE_OF_CONDUCT.md file.

mfsysmon's People

Contributors

Watchers

Forkers

metworkbot dearith

mfsysmon's Issues

Error while starting mfsysmon - circus watcher for telegraf

In MetWork v2.2, RockyLinux 8.7, I face an error while starting mfsysmon, an ERROR is displayed while "waiting for start of telegraf" :

- Waiting for start of telegraf...                         [ ERROR ]

It appears that below command returns an error:

_circusctl --endpoint ipc:///home/mfsysmon/var/circus.socket status "telegraf"
error

While in version 2.1, it returns the expected value active.

See the wait_watcher_started method in file circus.py https://github.com/metwork-framework/mfext/blob/master/adm/circus.py

The call stack is:
mfsysmon.start ->
_circus_wait_watcher_started.sh telegraf ->
layer_wrapper --layers=python3_circus@mfext -- _circus_wait_watcher_started.py telegraf ->
client.wait_watcher_started(name='telegraf', timeout='20') (MetWorkCircusClient from circus.py)
_circusctl --endpoint ipc:///home/mfsysmon/var/circus.socket status "telegraf" ->
layer_wrapper --layers=python3_circus@mfext -- python -m circus.circusctl --endpoint ipc:///home/mfsysmon/var/circus.socket status "telegraf" ...

add a custom netio telegraf collector like custom_diskio

The python function to call:

https://psutil.readthedocs.io/en/latest/#psutil.net_io_counters

We are interested in:

bytes_sent_per_second
bytes_recv_per_second
packets_sent_per_second
packet_recv_per_second
errin_per_second
errout_per_second
dropin_per_second
dropout_per_second
dropin_percent = 100.0 * dropin_per_second / packet_recv_per_second (borné entre 0 et 100)
dropout_percent = 100.0 * dropout_per_second / packet_sent_per_second (borné entre 0 et 100)

interesting dashboard for system metrics

https://grafana.com/grafana/dashboards/10506

cpu load issues

Each 7 hours, we have CPU load spikes (higher than 6), in our two servers, as you can see in our grafana (mfadmin) screenshot shown below:

The threshold is 0.5 so we received alerts raised by our shinken probe:

Problem started at 07:21:35 on 2023.03.23
Problem name: Load average is too high (per CPU load over 1.5 for 5m)
Host: ***************
Severity: High
Operational data: Load averages(1m 5m 15m): (6.57 4.09 1.9), # of CPUs: 2
Original problem ID: 309747841

Analysis in progress... seems related to the way telegraf collects system metrics data on mfsymon module.

Issue open for follow-up, please add your comments below.