High memory usage - how to reduce? about deco HOT 3 CLOSED

Perhaps there's a way to call wait part way through and when the first batch finishes, continue to the next?

from deco.

That idea seems to be working actually. I'll go with that unless you see any problems with the approach.

@concurrent
def slow(index):
    ... do something

def run():
    for chunk in chunk(iterator_w_200K_items):
        for item in chunk:
            slow(index)
        slow.wait()
    slow.wait() # clear up stragglers?

from deco.

That seems to be the best way to approach the problem, it doesn't seem like multiprocessing.Pool has any mechanisms to deal with this sort of problem either. It would be possible to include this functionality in deco, like allowing you to define a batch size or something. However, I think since this solution you've provided exists, and seems rather straight forward, I'll leave it out in favor of simplicity.

Thanks for mentioning this by the way, I hope this can help others if they find themselves in a similar situation.

from deco.