ibm / torc_py Goto Github PK

Hi, this looks like a cool project - I'm looking forward to trying it more extensively.

In a few other distributed libraries (e.g. dask and ray) you can pass objects around indirectly just by re-submitting the future. E.g. something like:

def foo(x):
    return x + 1

def bar(x):
    return x**2

# get the first future
f1 = pool.submit(foo, 2)

# submit its answer by reference
f2 = pool.submit(bar, f1)

# get the final answer, without ever gathering f1.result()
f2.result()
# 9

I.e. the workers communicate the result of f1 directly without it passing through the main process, which can be a big efficiency saving for large return objects.

I guess my questions are:

• I'm not totally sure how torcpy is designed, so would it still be beneficial here? and if so, how easy?

I was wondering if/how it is possible, to wait for only the completion of some sub-tasks but not all. If I am reading the code correct, the torc.wait() function does not use arguments, is that right?

I was hoping to do something along the lines of

task1 = torc.submit(...).
task2 = torc.submit(...).
torc.wait(task1)

so that the code only has to wait for task1 to complete but can continue while task2 is still running. Is that possible?