MultiThreadedExecutor runs executor even when there is one builder in the respective rank. about databuilderframework HOT 3 CLOSED

Unless there is a demonstrated gain in performance due to this, it would introduce a lot of unnecessary complexity.

from databuilderframework.

agreed will try out a test case and post some numbers here to take this forward.

from databuilderframework.

Below is the output from running a new test case in "issue_9" branch of my fork.
se - simpleExecutor
me- multiThreadedExecutor
ome - omptimizedMultiThreadedExecutor - avoid using thread pool if size among the same rank is 1.

From below stats we can se that when concurrency is more that builder thread pool size "ome" gives considerable performance gains compared "me" - Again this is assuming the dataflow execution graph has many single nodes in between parallel nodes in topology.

Running com.flipkart.databuilderframework.cmplxscenariotest.ContextSwitchOverHeadTest
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
running unbounded queue test with builder size 150 and concurrecy 100
se 3590
me 2401
me run: context switches over threshold count 0, max latent switch: 0, rejections 0
ome 2091
ome run: context switches over threshold count 0, max latent switch: 0, rejections 0
testWhenConccurencyIsMoreThanBuilderSize
running unbounded queue test with builder size 50 and concurrecy 150
se 2518
me 6874
me run: context switches over threshold count 15112, max latent switch: 263, rejections 0
ome 3993
ome run: context switches over threshold count 10120, max latent switch: 288, rejections 0
testWhenConccurencyIsSameAsBuilderSize
running unbounded queue test with builder size 150 and concurrecy 150
se 2556
me 2362
me run: context switches over threshold count 342, max latent switch: 73, rejections 0
ome 1581
ome run: context switches over threshold count 196, max latent switch: 80, rejections 0
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.18 sec

from databuilderframework.