MultiProcessScheduler breaks rdd.fold() about dpark HOT 11 CLOSED

I think this bug is in serialize.py. Closures are acting funny. Also this:

mike@EDEN:~/Development/dpark/dpark$ python serialize.py
(21, 'glob:foo: some global last')
Traceback (most recent call last):
  File "serialize.py", line 140, in <module>
    print ff(2)
  File "serialize.py", line 131, in the_closure
    return (a * x + int(b), glob_func(foo(some_global)+last))
TypeError: unsupported operand type(s) for *: 'int' and 'function'

from dpark.

I have run this test case without problem:

davies@DoB ~/work/dpark $cat t.py 
import dpark
rdd = dpark.makeRDD([1, 2, 3, 4])
print rdd.fold(0, max)
davies@DoB ~/work/dpark $python t.py -m process
2013-04-19 11:02:55,061 [INFO] [scheduler] Got a job with 2 tasks
2013-04-19 11:02:55,070 [INFO] [scheduler] Job finished in 0.0 seconds                    
4
davies@DoB ~/work/dpark $python dpark/serialize.py
(21, 'glob:foo: some global last')
(21, 'glob:foo: some global last')

What is version of you Python ? Python3 ? We have not test DPark under Python3.

from dpark.

I'm on python 2.7.4 on Mac OS X. Strange.

from dpark.

Any more information about how to re-produce this problem ? Thanks

from dpark.

No, I tried your test case and it breaks for me. I'm going to continue to investigate. Please don't close this issue. I'll let you know either when I give up or when I figure it out. Thanks.

from dpark.

I have access to some linux machines and while getting the prereqs in order for t.py is kind of hard, I can easily run python dpark/serialize.py where I think the problem lies and it indeed works fine. This sounds like it could be a Mac problem. Now I've got access to a working copy and a broken copy and I'll let you know what I find.

from dpark.

It has to do with func_closure returning cells in a different order.

Linux:

$ cat t.py 
def a(b):
    c = 10.0
    def d():
        return b + c
    return d

z = a(5)
print [c.cell_contents for c in z.func_closure]

$ python t.py 
[10.0, 5]

Mac:

$ cat t.py 
def a(b):
    c = 10.0
    def d():
        return b + c
    return d

z = a(5)
print [c.cell_contents for c in z.func_closure]

$ python t.py 
[5, 10.0]

So when you try to reconstruct the closure, you're getting them backwards on mac. Strangely, there's a values.reverse() call before you return the func_closure. If I put that behind if sys.platform != 'darwin': it seems to work. I'm not sure if that's the right solution though.

from dpark.

FWIW, Linux and Windows agree on the order. Mac is the one that differs.

from dpark.

Here's a good test case for this problem, which has enough closure cells to verify that simply not doing .reverse() on Mac is the correct answer:

import serialize

def a(b, e, f, g):
    c = 10.0
    def d():
        return b + c + e + f + g
    return d

x = a(5, 6, 7, 8)
z = serialize.load_func(serialize.dump_func(x))
print 'x=', [c.cell_contents for c in x.func_closure]
print 'z=', [c.cell_contents for c in z.func_closure]

assert [c.cell_contents for c in x.func_closure] == [c.cell_contents for c in z.func_closure]
assert x() == z()

Linux:

x= [10.0, 5, 6, 8, 7]
z= [10.0, 5, 6, 8, 7]

Mac:

x= [5, 10.0, 6, 7, 8]
z= [5, 10.0, 6, 7, 8]

Feel free to make the change in the best way you see fit if mac compatibility is important to you. Thanks for your help!

from dpark.

I can verify that your change works on my Mac. Thanks Davies.

from dpark.

You are welcome. Thank you for reporting.

from dpark.