dynamically generated class methods fail to pickle about dill HOT 8 OPEN

Yup, its the circular dependency I think I've highlighted:

I don't know why it differs so much from a "normal" class:

from dill.

In the test above if I change line 9 to setattr(cls, 'methodname', proto_method) instead of explicitly using MethodType then the test passes.

But when I try that approach with the real program, dill still fails on circular references (but with a cell involved)

INFO:dill:Co: <code object method at 0x7f9be4e08230, file "/path/to.py", line 625>
INFO:dill:F2: <function _unmarshal at 0x7f9bed9d2140>
INFO:dill:D4: <dict object at 0x7f9be5bbfb40>
INFO:dill:Ce: <cell at 0x7f9be58a6be8: str object at 0x7f9bf418c7b0>
INFO:dill:F2: <function _create_cell at 0x7f9bed9d2578>
INFO:dill:Ce: <cell at 0x7f9be58a6bb0: type object at 0x251b6d0>
INFO:dill:T2: <class '__main__.NewCls'>
INFO:dill:F2: <function _create_type at 0x7f9bed9d2230>
INFO:dill:T1: <type 'type'>
INFO:dill:D2: <dict object at 0x7f9bdf2266e0>
INFO:dill:F1: <function f at 0x7f9be1c106e0>
INFO:dill:D4: <dict object at 0x7f9be5bbfb40>
INFO:dill:Ce: <cell at 0x7f9be58a6bb0: type object at 0x251b6d0>
INFO:dill:T2: <class '__main__.NewCls'>
INFO:dill:D2: <dict object at 0x7f9bdf370050>
INFO:dill:F1: <function f at 0x7f9be1c106e0>
INFO:dill:D4: <dict object at 0x7f9be5bbfb40>
INFO:dill:Ce: <cell at 0x7f9be58a6bb0: type object at 0x251b6d0>
INFO:dill:T2: <class '__main__.NewCls'>
INFO:dill:D2: <dict object at 0x7f9bdf222a28>
INFO:dill:F1: <function f at 0x7f9be1c106e0>
INFO:dill:D4: <dict object at 0x7f9be5bbfb40>
INFO:dill:Ce: <cell at 0x7f9be58a6bb0: type object at 0x251b6d0>
INFO:dill:T2: <class '__main__.NewCls'>

I will try to find a minimal test case to reproduce this too.

from dill.

In the test above if I change line 9 to setattr(cls, 'methodname', proto_method) instead of explicitly using MethodType then the test passes.

Interesting. Maybe this is one of the things they simplified in python3, for reasons like this.

from dill.

I think in essence, this is a problem with pickle, not dill, as when saving an object, pickle resolved the dependencies before marking the object as memorised, which means that circular references are not well handled.

from dill.

This may mean digging into exactly how the object is created by python, and seeing if there's a way to rebuild it from any component parts that are pickleable. In this case, looking at what exactly setattr is doing, seeing if there's a simple way to detect that setattr was used, and then building a function to replicate the object construction.

from dill.

I've found the difference with the first issue:

>>> import types
>>> def f(): pass
... 
>>> class A:
...     def f():pass
... 
>>> class A2: pass
... 
>>> f2=types.MethodType(f, None, A2)
>>> A2.f=f2
>>> A.__dict__["f"]
<function f at 0x7f3d83ec7b90>
>>> A.f
<unbound method A.f>
>>> A2.__dict__["f"]
<unbound method A2.f>
>>> A2.f
<unbound method A2.f>
>>> 

from dill.

Ach.. that's bordering on being a python bug!

A loose plan to serialize A2 might be:

1. Identify unbound methods in the __dict__
2. serialize the methods as: A2.f.im_func (or possibly A2.f) in A2, or "with A2" as a tuple of (A2, dict) where dict has string names (i.e. "f") as keys and the objects as values
3. in unpickling, deserialize A2, and then "reapply" the "special" methods exactly how they were built.

from dill.

Turns out this has indeed been fixed in python3. The instancemethod type has been removed and all unbound methods are now type function.
setattr(cls, 'methodname', f) is the only idom that works in both versions so it seems people should avoid types.MethodType for portable code in any case. I'll focus on trying to isolate the second issue.

from dill.