how to monitor worker processes about delayed HOT 2 CLOSED

Hi @daichi5! We currently don't run our worker pods with a readinessProbe or livenessProbe config, largely because the processes don't accept any outside HTTP traffic, so we don't need the health checks for load balancing purposes. Instead we rely on the default behavior, which is that if the main process (at PID 1) exits, the container restarts.

We also use our cron/scheduler process to enqueue a background job once per minute, and that job emits a metric that we can monitor to alert ourselves if there are no workers running. But this exists outside of our k8s infrastructure, and we haven't shipped a generic version of this behavior, since it depends on the specifics of our internal monitoring/alerting infrastructure.

from delayed.

Hi @smudge! Thanks for the response.
I understand how you operate worker pods.

Instead we rely on the default behavior, which is that if the main process (at PID 1) exits, the container restarts.

As you said, health check may not be necessary because the container will be restarted if the main process exits.
However, we have enabled shareProcessNamespace, so our situation may be a little different.

We also use our cron/scheduler process to enqueue a background job once per minute, and that job emits a metric that we can monitor to alert ourselves if there are no workers running

The idea of this monitoring jobs is very helpful.
I think we'll try to use a similar approach to this one to manage worker processes.
Thank you!

from delayed.