Celery plugin provides ability of graceful worker stopping.
Production deployment of long running tasks require worker to be stopped gracefully. Unfortunately celery got different behaviour:
- Receiving
SIGTERM
signal bycelery
results starting Warm shutdown procedure. Due this procedure,inspect
andcontrol
commands become unavailable. - More than that, all tasks are terminated forcely by the second
SIGTERM
with the Cold shutdown procedure.
This module provides more consistent approach to this problem, it
- overrides
SIGTERM
receiver to prevent default Warm shutdown and Cold shutdown worker behaviour, - forces
inspect
andcontrol
commands to be working even afterSIGTERM
signal received.
pip install git+https://github.com/MnogoByte/celery-graceful-stop.git
Append your proj/celery.py
file containg app
instance with the following lines.
import celery_graceful_stop
celery_graceful_stop.register(app)
CELERY_GRACEFUL_STOP
(boolean). Controls graceful stop function. (True
by default)
- Define 1 service per each worker you got (systemd require only one master
pid
for restart). - Provide
/etc/conf.d/celery_<service_name>
configuration file for each worker. - Add [email protected] file into your system.
- Register your service with
systemctl enable celery@<service_name>
.
- This module disables
pool_shrink
,pool_grow
,autoscale
,pool_reload
,add_consumer
,cancel_consumer
control commands after receivingSIGTERM
signal. Actually, you'll does not need to call them when the worker is shutting down, because worker will not starts new tasks. - This module has been tested only with celery 3.1 with pool=prefork.
BSD - 3