uneven workflow distribution about workflow HOT 4 CLOSED

Interesting - distribution is essentially random. Each worker will take tasks as it can. I'll have a look if there's something other than randomness causing this. It if is indeed random, though, should we introduce some kind of distribution mechanism that tries to evenly distribute tasks?

from workflow.

If it is really random then I don't think we need another mechanism: small
variations are fine. The problem here is that it doesn't look random. Here
is how it looks on our production system with about 1 million tasks
executed. In this case, we have about 15 different task types. The level of
concurrency configured for each type of task is different. The pattern
looks more or less the same for each task type. After running multiple
experiments I think the pattern is more visible when you execute a large
number of tasks.

[image: Inline image 2]

On Wed, Dec 23, 2015 at 9:35 AM, Jordan Zimmerman [email protected]
wrote:

Interesting - distribution is essentially random. Each worker will take
tasks as it can. I'll have a look if there's something other than
randomness causing this. It if is indeed random, though, should we
introduce some kind of distribution mechanism that tries to evenly
distribute tasks?

—
Reply to this email directly or view it on GitHub
#19 (comment).

from workflow.

I believe I know what's happening. Internally, Workflow uses Curator's DistributedQueue to get tasks that need processing. Unfortunately, ZooKeeper only provides getChildren() to get children under a node - i.e. it returns ALL children. Each Task worker then tries to process all current tasks. There's code to prevent duplicates but it could mean that the first process to call getChildren() will likely process more than other processes.

Fortunately, there is a workaround. Curator's PriorityQueue processes children a small amount at a time to allow for higher priority new items getting inserted. I did some testing with this and seems to help. So, simply change your TaskType's mode from the default to TaskMode.PRIORITY. I ran your tests with the default and I get this:

workflow tester workflowManager-1 has executed 376 tasks
workflow tester workflowManager-3 has executed 336 tasks
workflow tester workflowManager-2 has executed 288 tasks

Using TaskMode.PRIORITY (the only change) I now get this:

workflow tester workflowManager-2 has executed 353 tasks
workflow tester workflowManager-3 has executed 313 tasks
workflow tester workflowManager-1 has executed 334 tasks

So, if possible, please test using TaskMode.PRIORITY. It shouldn't affect performance as all tasks will have the same priority by default.

from workflow.

I've re-written the queue code to more evenly distribute. Please test with: https://github.com/NirmataOSS/workflow/tree/simple-queue (i.e. branch "simple-queue")

from workflow.