Comments (21)
Faking cls.__module__ = '__main__'
seems to be a good temporary workaround for this issue.
It is reliably forcing the classes to be pickled/unpickled in full.
The further errors which resulted seem to be a separate issue so I've filed the details in #58.
from dill.
@mmckerns Is it easier to fix this for Python 3 only? And if so, would it make sense at least for now?
Maybe due to the wider Python 3 adoption and the fact that Python 2 will be discontinued "soon" (in less than 3 years), concentrating efforts in Python 3 is not a bad idea specially for the cases where Python 2 makes things more complicated (?).
from dill.
@Peque: yes, I believe the fix for 3.x is much easier. There have been a lot of changes since I dug into this last, and several older versions (<= 2.6, and <= 3.4) are basically unsupported by the python community... so it may be much easier of a task than previously.
from dill.
Yeah, it one of the problems with the pickle approach. When it finds a class that's in a module, it assumes that is can be reloaded "on the other side". Dill adjusts this to completely pickle all "things" in __main__
, but leaves pickle to do it's stuff everywhere else. As a work around you could do:
def f():
global NewCls
cls = type('NewCls', (object,), dict())
NewCls = cls
print(dill.pickles(cls))
or change cls.__module__
to __main__
.
@mmckerns We could adjust dill.dill.save_type
to first try to find the type, then if it can be found, pass it on to pickle, else pickle it as if it were in __main__
?
from dill.
In a way, this issue is similar to #52 and therefore #1, in that the obvious fix is to treat modules like __main__
.
from dill.
These should be no different than named tuples, which "everyone knows" the instance needs to be named appropriately.
The other part of this issue is having the instance creation done inside a class... and python misidentifies the namespace as being inside the file object -- this is one of the outstanding problems remaining for serialization, and solving it should solve a number of objects, I believe.
I think it's safe to ignore the first part (naming) for now, as it is convention.
from dill.
Thanks. Trying out these workarounds, inserting the class in globals()
allowed dill to find
the class. But as it is pickled by ref, the unpickling fails on the remote host.
The second workaround, setting cls.__module__ = '__main__'
solved the problem for the simplified test case.
When I apply this workaround in my real program, I get RuntimeError: Maximum recursion depth exceeded
thrown during pickling (everything?) in __main__
. I'll take a closer look tomorrow to see what's going on.
from dill.
Feel free to post a minimal version of your program than reproduces the traceback you are seeing. For certain objects, it is currently possible to instruct dill
to not pickle by reference. dill
currently does this for user-defined classes (built in the traditional sense using a class
def), and this could probably be extended to turning off pickling by reference for classes built as above.
from dill.
dill.detect.trace
may be useful for that.
from dill.
Good point. @mattja: there are a few functions in dill.detect
that can be used to investigate where and how things fail to pickle.
from dill.
@mmckerns, in this case, qualname
appears to fail on Python 2. However, the failure is a little surprising as it seems to expect __qualname__
is defined. Apparently, there is some code path where it doesn't actually set the value for this attribute.
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-7-03af44776c0f> in <module>()
----> 1 qualname.qualname(c)
/opt/conda/lib/python2.7/site-packages/qualname.pyc in qualname(obj)
53 _, lineno = inspect.getsourcelines(obj)
54 except (OSError, IOError):
---> 55 return obj.__qualname__ # raises a sensible error
56 elif inspect.isfunction(obj) or inspect.ismethod(obj):
57 if hasattr(obj, 'im_func'):
AttributeError: type object 'NewCls' has no attribute '__qualname__'
I have raised an issue containing more detail about this problem upstream. ( wbolster/qualname#2 )
from dill.
@jakirkham: No, as I mentioned in the other thread, qualname
can't handle a lot of cases… what you are seeing is the author's choice that when inspect
fails to fabricate a line of code that will throw an AttributeErrror
with reference to __qualname__
.
from dill.
I think the behavior of qualname
can be significantly enhanced by the code already in dill.source
, and as I mentioned in the other thread, at quick glance it looks like dill.source
handles more cases including the capacity to work in __main__
. I think the easier route is to leverage the visitor pattern object used in qualname
, and then replace most of the rest of the code with what's already in dill.source
.
from dill.
That's fine. I just wanted to see if qualname
would already work for this case. The answer is no. Whether one considers this a bugfix or feature request for qualname
is an orthogonal issue. If you have the framework to get this working @mmckerns, I am still in favor of your original proposal.
from dill.
FYI, the above is the same as in the discussion from the other thread -- and, yes, I think the easier fix is to do so within dill.source
.
from dill.
qualname
maintainer here. fixes and improvements to https://github.com/wbolster/qualname are welcome as long as its generic enough, since qualname
is basically just a backport of python 3 functionality.
from dill.
Hi @wbolster, as you can see from this thread, this is an issue that has been around for a few years. Actually, it's been known (to at least me) for almost the entire life of dill
(10 years).
I've known the "backport" solution (basically, what is done in qualname
and thus what is done in python 3) since __qualname__
appeared in python 3 -- however, I wasn't able to resolve the issues that qualname
avoids… so I don't have code in dill
to handle it at all yet.
dill.source
enables source introspection (like the inspect
module), with support for objects defined in __main__
as well as extracting source (or imports) for dependencies so an object can be rebuilt from the source. A blend of some of the functions available in dill.source
that can dig into an object's source code with the root code from qualname
, I believe, would cover many of the cases that qualname
currently fails on. At this point, I'm not suggesting augmenting qualname
, but instead extending dill
using dill.source
and some of the code inside qualname
… however, if I find that it'd be better to go the inverse route (extending qualname
not dill.source
) I will submit a patch when I get to this issue.
from dill.
FWIW, it appears that cloudpickle
solves these problems. So, it may be worth looking at.
from dill.
@jakirkham: Thanks. I am very familiar with cloudpickle
, and am aware of it's abilities and limitations in this vein.
from dill.
See interesting mapping done in: python/typeshed#24
from dill.
This should be revisited, especially since #413 happened. Note similarity of #507, and community workarounds such as this one.
from dill.
Related Issues (20)
- Python 3.12: ModuleNotFoundError: No module named 'setuptools' HOT 5
- How to ensure the same functions serialize to the same bytes?
- Cannot use callable that was pickled within pytest HOT 14
- dill.source.getsource applied on lambda function returns many cached lines
- Class attributes changes at load if the object is in a list or tuple. HOT 1
- nan type drift for np.nan HOT 3
- dill.source.findsource fails when in asyncio REPL
- `save_function()` can't save function in a submodule that has the same name as an attribute of the parent module HOT 2
- bytes length not a multiple of item size
- dill.source.getsource returns decorated function instead of function HOT 2
- 0.3.7 incorrectly pickles the class definition for module/class with the same name HOT 7
- Exception: --- Logging error --- HOT 3
- Use dump_module() in jupyter notebook with remote kernel HOT 1
- UnpicklingError using dill but not stdlib pickle HOT 2
- Compatibility with Pyodide HOT 6
- tfp.math.psd_kernels.ExponentiatedQuadratic no longer serialiases with latest versions
- `dill.load_session()` raises `NameError: name 'UnpicklingError' is not defined` HOT 5
- pydantic>=2.5 classes can't be serialized HOT 4
- Failure to load dumped partial function (in older python) properly (in newer python)
- threading.Thread fails to pickle in python > 3.13.0a5 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dill.