Persist Scheduled Tasks to the Backing Store about gofer.net HOT 2 CLOSED

This issue is more important than I realized.

Currently, a common pattern is to have a web client offloading work to a set of workers.
The case where a web client schedules tasks, but does not also listen for tasks causes the scheduled tasks never to be run.

from gofer.net.

In solving this issue, an important design consideration came up regarding recurring tasks.

I'm writing about it here to record the tradeoffs in case the consideration should be revisited.

In the event of all task schedulers failing or being turned off, and then being turned on after some non-negligible amount of time, the consideration is whether or not the recurring tasks should run for any missed intervals during that time off.

What was implemented does not run missed intervals for recurring tasks but will start running the next intervals as soon as at least one scheduler comes back up.

The motivating scenario was if the schedulers are manually turned off for an evening (or so) then when they get switched back on for the next day, a flood of recurring tasks should not then be poured into the task queue.

An important tradeoff here is that a task scheduler that is backed up may skip recurring task intervals.

from gofer.net.