Implement a Task Queue about auviewer HOT 2 CLOSED

We've decided to initially implement the task queue using Python's multiprocessing library. Later, we could implement an upgrade path to use celery+redis/rabbitmq which would mainly bring the advantage of being able to distribute workloads across machines.

We will architect this as follows:

• In the main thread of serve.py, we'll establish two process Pool instances for low & high priority tasks. Downsampling jobs will go to the low priority queue, and featurization & weak labeling jobs will go to the high priority queue.
• For simplicity for now, let's acquire the number of cores available and then use floor(0.5*numCores) as the number of processes in both low & high priority queues. E.g. a machine with 8 cores will end up with 4 low priority workers & 4 high priority workers.

from auviewer.

Complete.

from auviewer.