Parallel is really slow for many small jobs about parallel HOT 11 CLOSED

yep, also ran into this problem quiet often :)
i think it would be good to have this double-piping-magic, maybe a fallback could be added to get the old behavior, in case someone really needs 'one process per item'.

from parallel.

can you try to merge/add it, or should i give it a try ?

from parallel.

I can definitely merge it in. It may be a while before I get a chance to do it.

-Nick

from parallel.

Great, made myself an reminder to do it in 1 week, hope you can beat me :)

from parallel.

any progress ?

from parallel.

nope sorry. :-(

from parallel.

then ill give it a try tomorrow, already had a close look at the code :>

from parallel.

im finished so far (its on the 'worker' branch)
only thing i do not understand:

• why rescue from Interrupt, when does it happen / why is it bad ?
• why wait for the processes after the threads are joined, the processes should have finished by then / can be killed

from parallel.

rescue from interrupt so that if you have 10 parallel processes, and one blows up, you don't get multiple stack traces. It's easier to debug forking issues this way. Not necessary, just helpful.

You have to either wait or detach a process or it could go zombie on you. This is very important if someone is using parallel in a long running process, like rails. The thread could have joined because the pipe broke, but the process could keep going. Believe me, you want it to hang on wait when something goes wrong, not just keep going. You could replace wait with kill if you want, but I think wait is better. Definitely don't detach. That could get crazy.

from parallel.

fixed in 0.5.0

from parallel.

Very cool. I'd like to try this out with parallel_tests, to see if it improves testing performance.

EDIT: never mind. I see you shell out in groups so it probably won't do too much in that case.

from parallel.