cdown / psi-notify Goto Github PK

View Code? Open in Web Editor NEW

217.0 9.0 12.0 265 KB

Alert when your machine is becoming oversaturated, *before* it slows to a crawl

License: MIT License

C 86.90% Makefile 5.97% Shell 3.12% Roff 4.01%

psi-notify's People

Contributors

Stargazers

Watchers

Forkers

nhorus odex-tos michel-slm keszybz xaionaro sjanulonoks foxboron mpolanski jorik041 cgzones lgtm-migrator sasg

psi-notify's Issues

Avoid entering link_path_walk so much

Currently we spend a good amount of time doing path walking as part of our total CPU time (well, the total isn't much, so it doesn't matter much, but it would be nice to reduce):

This is especially pronounced if we want to use a logind seat. It should be trivial to maintain an fopenat()-style function which can present back a FILE* from a directory stored in the Resource.

Reduce alert volatility even further

Before 1.0.0, we added active management of notifications: that is, they aren't fire-and-forget any more, we will close them when the alert clears. If the user closes them manually, we won't reopen them again while this alert is active. This works well to avoid noise for the user.

However, there is room to improve further. Here's my suggestion:

Don't notify again if the alert for this resource last cleared less than N seconds ago. N could be called renotify_min_sec, with the default set to (say) 30 seconds. I guess we'd still report to stdout.
Add a threshold penalty which PSI must go under before an alert clears, eg if the alert is 40 and tripped, the pressure must go below 40 - N to trip. Let's say a good default is maybe threshold - 5 or pressure 0.10, whichever is higher. This could be configurable as threshold_clear_pct.

#2 will allow us to avoid clearing the notification too early in the first case, which helps with #1 because we might enter alert state again without wanting to send a notification.

Perhaps at the beginning we don't make these configurable, and see if there's user demand for customisation.

Create debian package

This is to request Debian package for psi-notify.

If this is not planned in Roadmap, I am willing to contribute this feature.

Though installing on Ubuntu works with rpm package via alien as shown below:

disdi@disdi-Precision-5540:~/Downloads$ sudo alien -i psi-notify-1.2.1-5.fc36.x86_64.rpm
Preparing to unpack psi-notify_1.2.1-6_amd64.deb ... Unpacking psi-notify (1.2.1-6) ... Setting up psi-notify (1.2.1-6) ...

Use nanosleep + timespec to wait for remaining time after reload

Add manual page

Allow users to specify an alternative cgroup path

Alert remains active after config reload?

Seems rare, but during testing I saw this:

● psi-notify.service
     Loaded: loaded (/usr/lib/systemd/user/psi-notify.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-05-28 21:00:17 BST; 9min ago
    Process: 27645 ExecReload=/usr/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
   Main PID: 8263 (psi-notify)
     Status: "Waiting for next interval."
      Tasks: 3 (limit: 23698)
     Memory: 1.2M
        CPU: 24ms
     CGroup: /user.slice/user-1000.slice/[email protected]/psi-notify.service
             └─8263 /usr/bin/psi-notify

May 28 21:00:17 roujiamo systemd[820]: Starting psi-notify.service...
May 28 21:00:17 roujiamo psi-notify[8263]: INFO: Using pressures from the current user's systemd-logind seat.
May 28 21:00:17 roujiamo psi-notify[8263]: INFO: Config:
May 28 21:00:17 roujiamo psi-notify[8263]:       Log pressures: false
May 28 21:00:17 roujiamo psi-notify[8263]:       Update interval: 5s
May 28 21:00:17 roujiamo psi-notify[8263]:       Thresholds:
May 28 21:00:17 roujiamo psi-notify[8263]:         - CPU avg10 some: 50.00
May 28 21:00:17 roujiamo psi-notify[8263]:         - Memory avg10 some: 10.00
May 28 21:00:17 roujiamo psi-notify[8263]:         - I/O avg10 some: 10.00
May 28 21:00:17 roujiamo psi-notify[8263]: INFO: Pressure monitoring started.
May 28 21:00:17 roujiamo systemd[820]: Started psi-notify.service.
May 28 21:01:22 roujiamo psi-notify[8263]: INFO: CPU alert: active
May 28 21:02:27 roujiamo psi-notify[8263]: INFO: CPU alert: stabilising
May 28 21:02:32 roujiamo psi-notify[8263]: INFO: CPU alert: inactive
May 28 21:03:27 roujiamo psi-notify[8263]: INFO: CPU alert: active
May 28 21:03:37 roujiamo psi-notify[8263]: INFO: CPU alert: stabilising
May 28 21:03:42 roujiamo psi-notify[8263]: INFO: CPU alert: inactive
May 28 21:03:57 roujiamo psi-notify[8263]: INFO: CPU alert: active
May 28 21:04:17 roujiamo psi-notify[8263]: INFO: CPU alert: stabilising
May 28 21:04:22 roujiamo psi-notify[8263]: INFO: CPU alert: inactive
May 28 21:04:32 roujiamo psi-notify[8263]: INFO: CPU alert: active
May 28 21:04:37 roujiamo psi-notify[8263]: INFO: CPU alert: stabilising
May 28 21:04:42 roujiamo psi-notify[8263]: INFO: CPU alert: inactive
May 28 21:04:50 roujiamo systemd[820]: Reloading psi-notify.service.
May 28 21:04:50 roujiamo psi-notify[8263]: INFO: Config reloaded. New config after reload:
May 28 21:04:50 roujiamo psi-notify[8263]:       Log pressures: false
May 28 21:04:50 roujiamo psi-notify[8263]:       Update interval: 5s
May 28 21:04:50 roujiamo psi-notify[8263]:       Thresholds:
May 28 21:04:50 roujiamo psi-notify[8263]:         - CPU avg10 some: 5.00
May 28 21:04:50 roujiamo psi-notify[8263]:         - Memory avg10 some: 10.00
May 28 21:04:50 roujiamo psi-notify[8263]:         - I/O avg10 some: 10.00
May 28 21:04:50 roujiamo psi-notify[8263]: INFO: CPU alert: active
May 28 21:04:50 roujiamo systemd[820]: Reloaded psi-notify.service.
May 28 21:07:23 roujiamo systemd[820]: Reloading psi-notify.service.
May 28 21:07:23 roujiamo psi-notify[8263]: INFO: Config reloaded. New config after reload:
May 28 21:07:23 roujiamo psi-notify[8263]:       Log pressures: true
May 28 21:07:23 roujiamo psi-notify[8263]:       Update interval: 5s
May 28 21:07:23 roujiamo psi-notify[8263]:       Thresholds:
May 28 21:07:23 roujiamo psi-notify[8263]:         - CPU avg10 some: 5.00
May 28 21:07:23 roujiamo psi-notify[8263]:         - Memory avg10 some: 10.00
May 28 21:07:23 roujiamo psi-notify[8263]:         - I/O avg10 some: 10.00
May 28 21:07:23 roujiamo psi-notify[8263]: INFO: Current CPU pressures: some avg10=0.01 avg60=8.10 avg300=16.11
May 28 21:07:23 roujiamo psi-notify[8263]: INFO: Current memory pressures: some avg10=0.00 avg60=0.00 avg300=0.00
May 28 21:07:23 roujiamo psi-notify[8263]: INFO: Current memory pressures: full avg10=0.00 avg60=0.00 avg300=0.00
May 28 21:07:23 roujiamo psi-notify[8263]: INFO: Current I/O pressures: some avg10=0.00 avg60=0.02 avg300=0.14
May 28 21:07:23 roujiamo psi-notify[8263]: INFO: Current I/O pressures: full avg10=0.00 avg60=0.02 avg300=0.13
May 28 21:07:23 roujiamo systemd[820]: Reloaded psi-notify.service.
May 28 21:07:28 roujiamo psi-notify[8263]: INFO: Current CPU pressures: some avg10=0.00 avg60=7.33 avg300=15.78
May 28 21:07:28 roujiamo psi-notify[8263]: INFO: CPU alert: stabilising
May 28 21:07:28 roujiamo psi-notify[8263]: INFO: Current memory pressures: some avg10=0.00 avg60=0.00 avg300=0.00
May 28 21:07:28 roujiamo psi-notify[8263]: INFO: Current memory pressures: full avg10=0.00 avg60=0.00 avg300=0.00
May 28 21:07:28 roujiamo psi-notify[8263]: INFO: Current I/O pressures: some avg10=0.00 avg60=0.02 avg300=0.14
May 28 21:07:28 roujiamo psi-notify[8263]: INFO: Current I/O pressures: full avg10=0.00 avg60=0.02 avg300=0.13
May 28 21:07:33 roujiamo psi-notify[8263]: INFO: Current CPU pressures: some avg10=0.00 avg60=6.86 avg300=15.57
May 28 21:07:33 roujiamo psi-notify[8263]: INFO: CPU alert: inactive
May 28 21:07:33 roujiamo psi-notify[8263]: INFO: Current memory pressures: some avg10=0.00 avg60=0.00 avg300=0.00
May 28 21:07:33 roujiamo psi-notify[8263]: INFO: Current memory pressures: full avg10=0.00 avg60=0.00 avg300=0.00
May 28 21:07:33 roujiamo psi-notify[8263]: INFO: Current I/O pressures: some avg10=0.00 avg60=0.02 avg300=0.14
May 28 21:07:33 roujiamo psi-notify[8263]: INFO: Current I/O pressures: full avg10=0.00 avg60=0.02 avg300=0.13

Somehow after the first reload, we got stuck in active and never went down to stabilising again until another reload was issued.

It seems it was still sending watchdog events since systemd didn't restart it, although I should have checked the watchdog timestamp.

Add some unit/integ tests in addition to fuzzing/e2e

The fuzzers do a pretty good job checking that there aren't crashes in the config or parser code, but some other parts of the code could probably just do with some unit or integration tests.

We should evaluate how best to do this in a fairly unobtrusive way.

Support --help

If cgroup read fails, try to reopen fd

Right now if the cgroup is created and destroyed for the /sys/fs/cgroup use case, we don't reopen the dir. Strangely the dir_fd isn't invalid after that, but doing openat() will fail.

As such, we can try to reopen the outer dir on file open failure, instead of giving off the warning the first time.

Make fuzzing functionality generic

For fuzzing, we currently have the following custom code:

Current/max number of iterations of the event loop
Custom pressure path.

Both of these could be consolidated to non-fuzzer specific code. For example, the custom pressure path can just be searched for either resource.pressure or resource instead of /proc and the logind seat cgroup.

Both can then be attached to a getopt_long loop. A help should be added as well.

Service start times out

On Arch, using the AUR package and the included service file, it seems psi-notify is not successfully notifying systemd that it is ready, and as a result it keeps getting restarted every ~90 seconds.

$ journald --user-unit psi-notify --since '3m ago'
Nov 20 02:31:07 peanut.codl.fr systemd[1188]: Starting psi-notify.service...
Nov 20 02:31:07 peanut.codl.fr psi-notify[5624]: INFO: Using system-global resource pressures.
Nov 20 02:31:07 peanut.codl.fr psi-notify[5624]: INFO: Config:
Nov 20 02:31:07 peanut.codl.fr psi-notify[5624]:       Log pressures: false
Nov 20 02:31:07 peanut.codl.fr psi-notify[5624]:       Update interval: 5s
Nov 20 02:31:07 peanut.codl.fr psi-notify[5624]:       Thresholds:
Nov 20 02:31:07 peanut.codl.fr psi-notify[5624]:         - CPU avg10 some: 50.00
Nov 20 02:31:07 peanut.codl.fr psi-notify[5624]:         - Memory avg10 some: 10.00
Nov 20 02:31:07 peanut.codl.fr psi-notify[5624]:         - I/O avg10 full: 15.00
Nov 20 02:31:07 peanut.codl.fr psi-notify[5624]: INFO: Pressure monitoring started.
Nov 20 02:32:38 peanut.codl.fr systemd[1188]: psi-notify.service: start operation timed out. Terminating.
Nov 20 02:32:38 peanut.codl.fr psi-notify[5624]: INFO: Terminating after 19 intervals elapsed.
Nov 20 02:32:38 peanut.codl.fr systemd[1188]: psi-notify.service: Failed with result 'timeout'.
Nov 20 02:32:38 peanut.codl.fr systemd[1188]: Failed to start psi-notify.service.
Nov 20 02:32:38 peanut.codl.fr systemd[1188]: psi-notify.service: Scheduled restart job, restart counter is at 33.
Nov 20 02:32:38 peanut.codl.fr systemd[1188]: Stopped psi-notify.service.
Nov 20 02:32:38 peanut.codl.fr systemd[1188]: Starting psi-notify.service...
Nov 20 02:32:38 peanut.codl.fr psi-notify[5641]: INFO: Using system-global resource pressures.
etc...

Between restarts it seems to work fine, I can cause pressure and get a notification.

Is there something I am missing?

Running but not logging or reporting

On Arch after installing via the AUR package, running psi-notify gives the following:

~/ > psi-notify
INFO: Using system-global resource pressures.
INFO: Config:

      Log pressures: true
      Update interval: 5s

      Thresholds:
        - CPU avg10 some: 50.00
        - Memory avg10 some: 10.00
        - I/O avg10 full: 15.00

INFO: Pressure monitoring started.
WARN: PSI dir (logind seat) seems to have gone away, reopening

And nothing is ever logged, no notifications are sent. The systemd service does the same (as expected).

sd_notify support

Allow silencing output without having to redirect to /dev/null

The info outputs take a small amount of CPU to calculate, and we can skip them if passed a flag. Maybe it's time to start using getopt_long.

Simplify option parsing

Right now everything is done based on a bunch of if statements, but as we add more config options this will become unwieldy. Add a small data structure with parser rules and their parser.

Possibility for alternate notification method

Hi,

Feature request - could an alternate notification method me made possible. libnotify is not very suitable for SSH sessions in particular.

This could just be `notification_method=script or even another unit name.

Steve.

Only notify once per incident

Use inotify to watch for changes to config

Show violated value and threshold

alert_stabilising/alert_stop should not use magic numbers for state

We should just have an struct of 2 enums:

existing/new
inactive/stabilising

Evaluate sscanf vs. fscanf

It may be possible for us to use fscanf, which tends to have increased performance and lower CPU usage. For example, see here.

However, this probably requires some code restructuring and fseek()ing.

Question about process using 100% CPU of one core

Really appreciate psi-notify, awesome project :)

I got one small problem/question, but it may be out of psi-notifys scope.

I sometime get one process or another stuck at 100% CPU. Right now it seems to be some problem with pipewire that uses 100% CPU of one core. My problem is that I got 4 cores / 8 threads so this does not really get picked up by psi-notify. Even with my current config:

threshold cpu some avg10 1.00

I guess the total load of my CPU/machine is not really affected enough even if this process is stuck at using 100% CPU of one core.

I'm not running with log_pressures enabled at the moment so I can't really tell what the pressure is reported as at when this happens.

I could try to lower the CPU pressure even lower than 1.00 but that'll may result in a lot of false positives.

Any advice for picking this up or do I just have to try with lower settings and find the right threshold for reporting this?

[Request] Include process name in notification

If it were possible, including the process name in the notification could be useful for quickly identifying what tripped the alert.

test_pressure_check fails on CONFIG_PSI=n

test_pressure_check:
WARN: No pressure dir found. Are you using kernel >=4.20 with CONFIG_PSI=y?
  [FAIL:88] pressure_check(&cfg.memory, psi_f) == A_STABILISING

It, instead, returns A_ACTIVE?

crash (SIGABRT) when gnome-shell killed by OOM killer

I had psi-notify crash (SIGABRT) after gnome-shell got killed by the OOM killer.

I am using psi-notify 1.2.1-3 from Debian package maintained by @michel-slm with gnome-shell 42.0-4 and Linux 5.17.6-1 on Debian bookworm. In case it makes a difference, gnome-shell is in Wayland mode.

I don't have any easy way to debug this, but I guess you could simulate the situation by sending a SIGKILL to gnome-shell or whichever desktop you use.

If the info below and attachment is not useful, please close this bug report.

Here is the short gdb backtrace (long one attached):

#0  0x00007fbe62a8f87f in __GI___poll (fds=0x55c7dda7cc50, nfds=1, timeout=24784) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fbe62cc01ce in g_main_context_poll (priority=<optimized out>, n_fds=1, fds=0x55c7dda7cc50, timeout=<optimized out>, context=0x55c7dda7e630) at ../../../glib/gmain.c:4516
#2  g_main_context_iterate (context=0x55c7dda7e630, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../../../glib/gmain.c:4206
#3  0x00007fbe62cc0523 in g_main_loop_run (loop=0x55c7dda759e0) at ../../../glib/gmain.c:4411
#4  0x00007fbe628978ad in g_dbus_connection_send_message_with_reply_sync (connection=connection@entry=0x55c7dda6c050 [GDBusConnection], message=message@entry=0x55c7dda77c50 [GDBusMessage], flags=flags@entry=G_DBUS_SEND_MESSAGE_FLAGS_NONE, timeout_msec=timeout_msec@entry=-1, out_serial=out_serial@entry=0x0, cancellable=cancellable@entry=0x0, error=0x7ffcf59c1820) at ../../../gio/gdbusconnection.c:2163
#5  0x00007fbe62897cdc in g_dbus_connection_call_sync_internal (connection=0x55c7dda6c050 [GDBusConnection], bus_name=bus_name@entry=0x55c7dda784c0 ":1.44", object_path=0x55c7dda6a790 "/org/freedesktop/Notifications", interface_name=interface_name@entry=0x55c7dda6a7c0 "org.freedesktop.Notifications", method_name=method_name@entry=0x7fbe62e0a3f5 "Notify", parameters=parameters@entry=0x55c7dda7e290, reply_type=0x7fbe628fbe46, flags=G_DBUS_CALL_FLAGS_NONE, timeout_msec=-1, fd_list=0x0, out_fd_list=0x0, cancellable=0x0, error=0x7ffcf59c1ba0) at ../../../gio/gdbusconnection.c:6105
#6  0x00007fbe6289a265 in g_dbus_connection_call_with_unix_fd_list_sync (connection=<optimized out>, bus_name=bus_name@entry=0x55c7dda784c0 ":1.44", object_path=<optimized out>, interface_name=interface_name@entry=0x55c7dda6a7c0 "org.freedesktop.Notifications", method_name=method_name@entry=0x7fbe62e0a3f5 "Notify", parameters=parameters@entry=0x55c7dda7e290, reply_type=0x0, flags=G_DBUS_CALL_FLAGS_NONE, timeout_msec=-1, fd_list=0x0, out_fd_list=0x0, cancellable=0x0, error=0x7ffcf59c1ba0) at ../../../gio/gdbusconnection.c:6479
#7  0x00007fbe628a52db in g_dbus_proxy_call_sync_internal (proxy=proxy@entry=0x55c7dda5e6d0 [GDBusProxy], method_name=method_name@entry=0x7fbe62e0a3f5 "Notify", parameters=0x55c7dda7e290, flags=flags@entry=G_DBUS_CALL_FLAGS_NONE, timeout_msec=timeout_msec@entry=-1, fd_list=fd_list@entry=0x0, out_fd_list=0x0, cancellable=0x0, error=0x7ffcf59c1ba0) at ../../../gio/gdbusproxy.c:2853
#8  0x00007fbe628a6774 in g_dbus_proxy_call_sync (proxy=proxy@entry=0x55c7dda5e6d0 [GDBusProxy], method_name=method_name@entry=0x7fbe62e0a3f5 "Notify", parameters=<optimized out>, flags=flags@entry=G_DBUS_CALL_FLAGS_NONE, timeout_msec=timeout_msec@entry=-1, cancellable=cancellable@entry=0x0, error=0x7ffcf59c1ba0) at ../../../gio/gdbusproxy.c:3045
#9  0x00007fbe62e082de in notify_notification_show (notification=notification@entry=0x55c7dda5f140 [NotifyNotification], error=error@entry=0x7ffcf59c1ba0) at ../libnotify/notification.c:867
#10 0x000055c7dc1499b8 in alert_user (resource=<optimized out>) at ./psi-notify.c:88
#11 alert_user_if_new (r=<optimized out>) at ./psi-notify.c:569
#12 pressure_check_notify_if_new (r=<optimized out>) at ./psi-notify.c:619
#13 main (argc=<optimized out>, argv=<optimized out>) at ./psi-notify.c:785

Here is a summary of the systemd journal log during this time:

May 16 12:25:48 psi-notify[390290]: INFO: Memory alert: active
May 16 12:25:59 systemd[986]: psi-notify.service: Watchdog timeout (limit 10s)!
May 16 12:26:48 kernel: gnome-shell invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=200
...
May 16 12:26:49 kernel: Out of memory: Killed process 19091 (gnome-shell) total-vm:5940980kB, anon-rss:802560kB, file-rss:1292kB, shmem-rss:73064kB, UID:1000 pgtables:3644kB oom_score_adj:200
May 16 12:26:00 systemd[986]: psi-notify.service: Killing process 390290 (psi-notify) with signal SIGABRT.
...
May 16 12:26:52 systemd-coredump[2012428]: Process 390290 (psi-notify) of user 1000 dumped core.
...
May 16 12:26:52 systemd[986]: psi-notify.service: Main process exited, code=dumped, status=6/ABRT
May 16 12:26:52 systemd[986]: psi-notify.service: Failed with result 'watchdog'.
May 16 12:26:52 systemd[986]: psi-notify.service: Consumed 11.490s CPU time.
May 16 12:26:52 systemd[1]: [email protected]: Deactivated successfully.
May 16 12:26:52 systemd[986]: psi-notify.service: Scheduled restart job, restart counter is at 1.
May 16 12:26:52 systemd[986]: Stopped psi-notify.service.
May 16 12:26:52 systemd[986]: psi-notify.service: Consumed 11.490s CPU time.
May 16 12:26:52 psi-notify[2012552]: INFO: Using pressures from the current user's systemd-logind seat.
May 16 12:26:52 psi-notify[2012552]: INFO: Config:
May 16 12:26:52 psi-notify[2012552]:       Log pressures: false
May 16 12:26:52 psi-notify[2012552]:       Update interval: 5s
May 16 12:26:52 psi-notify[2012552]:       Thresholds:
May 16 12:26:52 psi-notify[2012552]:         - CPU avg10 some: 50.00
May 16 12:26:52 psi-notify[2012552]:         - Memory avg10 some: 10.00
May 16 12:26:52 psi-notify[2012552]:         - I/O avg10 full: 15.00
May 16 12:26:52 psi-notify[2012552]: INFO: Pressure monitoring started.
May 16 12:26:52 psi-notify[2012552]: INFO: Memory alert: active
May 16 12:26:52 systemd[986]: Starting psi-notify.service...
May 16 12:26:52 systemd[986]: Started psi-notify.service.
May 16 12:26:52 psi-notify[2012552]: WARN: Cannot display notification: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.Notifications was not provided by any .service files
May 16 12:26:52 psi-notify[2012552]: INFO: I/O alert: active
May 16 12:26:52 systemd[1]: session-13.scope: Deactivated successfully.
May 16 12:26:52 systemd[1]: session-13.scope: Consumed 3.697s CPU time.
May 16 12:26:52 systemd-logind[741]: Session 13 logged out. Waiting for processes to exit.
May 16 12:26:52 systemd-logind[741]: Removed session 13.
May 16 12:26:52 psi-notify[2012552]: WARN: Cannot display notification: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.Notifications was not provided by any .service files
May 16 12:26:52 gdm-launch-environment][2012555]: pam_unix(gdm-launch-environment:session): session opened for user Debian-gdm(uid=107) by (uid=0)
May 16 12:26:52 systemd[1]: Created slice User Slice of UID 107.
May 16 12:26:52 systemd[1]: Starting User Runtime Directory /run/user/107...
May 16 12:26:52 systemd-logind[741]: New session c3 of user Debian-gdm.
May 16 12:26:52 systemd[1]: Finished User Runtime Directory /run/user/107.
May 16 12:26:52 systemd[1]: Starting User Manager for UID 107...
May 16 12:26:52 systemd[2012561]: pam_unix(systemd-user:session): session opened for user Debian-gdm(uid=107) by (uid=0)