[feature] Option to stop/continue current Job with parallel tasks about tork HOT 6 CLOSED

You should be able to terminate workers mid-processing. RabbitMQ will automatically re-queue the tasks and redistribute them to whatever workers are left online.

from tork.

I know, but let's image that in one task I have some x200 sub tasks - like downloading\uploading something and I dont want to terminate this. I just want to let those tasks finish and the rest become onhold

from tork.

They will finish. It's not going to restart the parent task. Each of the subtasks will get auto requeued. Those that finish will not be impacted.

from tork.

Ok, let me rephrase:
I want to download 1m urls > I split this job to 1000 parallel tasks > and inside each task I have 1k urls
I have 100 workers. During the run of each task I have finished, lets image, 500 tasks in this Job. If I stop this Job all tasks will be marked as Canceled and if I Restart this Job - Tork will run all tasks that already Completed - 500 tasks already completed and I don't want to re-run them.
So thats why I dont want to re-run completed tasks inside Job with parallel tasks.

from tork.

Simply don't stop/cancel the job. Let it run. Terminate whatever machines you like, any tasks on these machines will be requeued to be handled by other workers.

from tork.

So long as RabbitMQ stays online you should be fine with doing whatever you want with the workers and even the coordinator

from tork.