Comments (13)
A very similar situation at sympy/sympy#2538. I can't pickle Q.positive
with that branch with dill.
from dill.
With the fix in Issue #12, this issue should also be working in general -- however, one might run into issues if there things defined out of scope that have changed... but, in general it works as below for instances as well as class definitions themselves.
>>> class Foo(object):
... def baz(self):
... return 1
...
>>> import dill
>>>
>>> f = Foo()
>>> _f = dill.dumps(f)
>>>
>>> class Foo(object):
... def bar(self):
... return 0
...
>>> o = dill.loads(_f)
>>> o.baz()
1
from dill.
My one still doesn't work. The issue there is not that the class has changed, but that it is created dynamically in __new__
. See https://github.com/sympy/sympy/pull/2538/files#diff-b23e62aa73719754451dd6b06ab6d83bR142
from dill.
@asmeurer: can you send me a minimal example? The reason my above example works is that I now serialize the class definitions. Your class definitions haven't changed, but something is used to dynamically generate the instances with __new__
, right? So, I put some sort of rng to generate a string that would be a method name, or some crazy thing like that... and that would match your case, correct? If that's so, you'd need to serialize the class definition and the info that seeded the dynamic part of __new__
... or you'd need a different code path, possibly generating a static class def, then using that to generate an instance -- I think.
from dill.
I tried copying over the relevant logic, but now I get what looks like a bug in dill
class _sentinel:
pass
class Test:
def __new__(cls, name):
if name == _sentinel:
obj = object.__new__(cls)
else:
obj = type(name.capitalize() + cls.__name__, (cls,),
{})(_sentinel)
obj.name = name
return obj
import dill
t = Test("Aaron")
dill.dumps(t)
Traceback (most recent call last):
File "test.py", line 17, in <module>
dill.dumps(t)
File "/Users/aaronmeurer/anaconda3/envs/dill-dev/lib/python3.3/site-packages/dill/dill.py", line 121, in dumps
dump(obj, file, protocol)
File "/Users/aaronmeurer/anaconda3/envs/dill-dev/lib/python3.3/site-packages/dill/dill.py", line 115, in dump
pik.dump(obj)
File "/Users/aaronmeurer/anaconda3/envs/dill-dev/lib/python3.3/pickle.py", line 235, in dump
self.save(obj)
File "/Users/aaronmeurer/anaconda3/envs/dill-dev/lib/python3.3/pickle.py", line 342, in save
self.save_reduce(obj=obj, *rv)
File "/Users/aaronmeurer/anaconda3/envs/dill-dev/lib/python3.3/pickle.py", line 407, in save_reduce
save(cls)
File "/Users/aaronmeurer/anaconda3/envs/dill-dev/lib/python3.3/pickle.py", line 297, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/aaronmeurer/anaconda3/envs/dill-dev/lib/python3.3/site-packages/dill/dill.py", line 702, in save_type
_dict = _dict_from_dictproxy(obj.__dict__)
File "/Users/aaronmeurer/anaconda3/envs/dill-dev/lib/python3.3/site-packages/dill/dill.py", line 379, in _dict_from_dictproxy
_dict.pop('__dict__')
KeyError: '__dict__'
from dill.
Would it make a difference if I removed the part where I set __doc__
on the generated class?
from dill.
Hmmm... the pop('__dict__')
I put into the code, originally, to avoid infinite recursion in pickling classes's proxy objects. I think that some python objects in 3x. don't expose __dict__
, I believe. Apparently, this issue hadn't percolated in dill, until the most recent changeset changed the course that classes run in their serialization. I need to play with it a little bit... but if there's no infinite recursion in the relevant 3.x object, the fix will just be to not try to execute the pop when in 3.x.
This should probably be a new ticket. I've got to catch a plane, but will dig into this further... as this is a bug I just added for 3.x.
from dill.
I guess, but the SymPy code raises PickleError.
from dill.
@asmeurer: I just updated to get past the KeyError above. Now your test case pickles, but doesn't unpickle. I believe I should be able to get around this with a little more thought.
>>> t = Test("Aaron")
>>> dill.detect.errors(t)
RuntimeError('uninitialized staticmethod object',)
>>> _t = dill.dumps(t)
>>> _t
b'\x80\x03cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01X\x04\x00\x00\x00typeq\x02\x85q\x03Rq\x04X\t\x00\x00\x00AaronTestq\x05h\x00(h\x04X\x04\x00\x00\x00Testq\x06h\x01X\x06\x00\x00\x00objectq\x07\x85q\x08Rq\t\x85q\n}q\x0b(X\x07\x00\x00\x00__new__q\x0ch\x01X\x0c\x00\x00\x00staticmethodq\r\x85q\x0eRq\x0f)\x81q\x10X\n\x00\x00\x00__module__q\x11X\x08\x00\x00\x00__main__q\x12X\x07\x00\x00\x00__doc__q\x13Nutq\x14Rq\x15\x85q\x16}q\x17(h\x13NX\r\x00\x00\x00__slotnames__q\x18]q\x19h\x11h\x12utq\x1aRq\x1b)\x81q\x1c}q\x1dX\x04\x00\x00\x00nameq\x1eX\x05\x00\x00\x00Aaronq\x1fsb.'
>>> dill.loads(_t)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mmckerns/lib/python3.3/site-packages/dill/dill.py", line 135, in loads
return load(file)
File "/Users/mmckerns/lib/python3.3/site-packages/dill/dill.py", line 128, in load
obj = pik.load()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 840, in load
dispatch[key[0]](self)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 1085, in load_newobj
obj = cls.__new__(cls, *args)
RuntimeError: uninitialized staticmethod object
from dill.
Let's move this to it's own issue -- #13.
from dill.
It seems it still can't pickle the actual object I am interested in in my SymPy branch, though. I get PicklingError: Can't pickle <class 'sympy.assumptions.assume.PositivePredicate'>: it's not found as sympy.assumptions.assume.PositivePredicate
. Does the module structure matter here? I don't create the object in the same file as the class.
from dill.
Right, so one of the things that python does that's a bit of a 'mistake' is that it associates a function or instance created in a factory pattern (like a curry) with the top level namespace where it was created and not the associated place in the namespace where the code actually lives. It's a common problem for pickling and some other similar things... and the root of a number of objects that fail to serialize with dill. So, I am looking into how to solve that problem... and it sounds like it could be related to the issue you are having.
Creating the instance in another file shouldn't matter. Module structure shouldn't matter too much, and an import can often make it easier as opposed to a reference from inner to outer scope within the same file. Encapsulation is the key to making pickling easier. The less you rely on python's namespace management magic, the easier you'll have a time in pickling something.
For me, the sympy code would be a back-burner item... but I'm happy to look into it when I get some free space (in ~2-3 weeks) if it's not cleared up by then.
from dill.
No problem. It's not high priority for me at the moment. For now, that code just lives in a pull request, and it probably will for some time actually.
from dill.
Related Issues (20)
- dill.source.findsource fails when in asyncio REPL
- `save_function()` can't save function in a submodule that has the same name as an attribute of the parent module HOT 2
- bytes length not a multiple of item size
- dill.source.getsource returns decorated function instead of function HOT 2
- 0.3.7 incorrectly pickles the class definition for module/class with the same name HOT 7
- Exception: --- Logging error --- HOT 3
- Use dump_module() in jupyter notebook with remote kernel HOT 1
- UnpicklingError using dill but not stdlib pickle HOT 2
- Compatibility with Pyodide HOT 6
- tfp.math.psd_kernels.ExponentiatedQuadratic no longer serialiases with latest versions
- `dill.load_session()` raises `NameError: name 'UnpicklingError' is not defined` HOT 5
- pydantic>=2.5 classes can't be serialized HOT 4
- Failure to load dumped partial function (in older python) properly (in newer python)
- threading.Thread fails to pickle in python > 3.13.0a5 HOT 4
- Error with typing.AbstractContextManager in Python 3.8 and newer dill versions (>=0.3.6) HOT 1
- File gets truncated upon unpickling
- PyTorch C++-generated module "not found as" itself? HOT 1
- Importing dill changes the python pickler behavior HOT 8
- Recursive self-references that trigger a RecursionError
- Attempting to `dill` a function defined in a `doctest` run with `pytest` causes a `TypeError: cannot pickle 'EncodedFile' object`.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dill.