Comments (9)
while from command line… toggling byref
does have some impact. However, we always start with B2
, which uses the StockPickler
. T5
is also the StockPickler
, but T2
is not (it's converting a dictproxy to a dict).
dude@hilbert>$ python numpy_new.py
{'color': 'green'}
B2: <built-in function _reconstruct>
T2: <class '__main__.TestArray'>
F2: <function _create_type at 0x110c24140>
T1: <type 'type'>
F2: <function _load_type at 0x110c240c8>
T4: <type 'numpy.ndarray'>
D2: <dict object at 0x7fc0490132d0>
T4: <type 'staticmethod'>
F1: <function __getnewargs__ at 0x110c65cf8>
F2: <function _create_function at 0x110c241b8>
Co: <code object __getnewargs__ at 0x1102deb30, file "numpy_new.py", line 16>
F2: <function _unmarshal at 0x110c24050>
D1: <dict object at 0x7fc04841b850>
D2: <dict object at 0x7fc049009b60>
F1: <function __array_finalize__ at 0x110c65c80>
Co: <code object __array_finalize__ at 0x1102de8b0, file "numpy_new.py", line 10>
D1: <dict object at 0x7fc04841b850>
D2: <dict object at 0x7fc049008490>
T4: <type 'numpy.dtype'>
{}
B2: <built-in function _reconstruct>
T5: <class '__main__.TestArray'>
T4: <type 'numpy.dtype'>
{}
{'color': 'green'}
B2: <built-in function _reconstruct>
T2: <class '__main__.TestArray'>
F2: <function _create_type at 0x110c24140>
T1: <type 'type'>
F2: <function _load_type at 0x110c240c8>
T4: <type 'numpy.ndarray'>
D2: <dict object at 0x7fc049007d90>
T4: <type 'staticmethod'>
F1: <function __getnewargs__ at 0x110c65cf8>
F2: <function _create_function at 0x110c241b8>
Co: <code object __getnewargs__ at 0x1102deb30, file "numpy_new.py", line 16>
F2: <function _unmarshal at 0x110c24050>
D1: <dict object at 0x7fc04841b850>
D2: <dict object at 0x7fc049009b60>
F1: <function __array_finalize__ at 0x110c65c80>
Co: <code object __array_finalize__ at 0x1102de8b0, file "numpy_new.py", line 10>
D1: <dict object at 0x7fc04841b850>
D2: <dict object at 0x7fc049008490>
T4: <type 'numpy.dtype'>
{}
B2: <built-in function _reconstruct>
T5: <class '__main__.TestArray'>
T4: <type 'numpy.dtype'>
{}
from dill.
Even simpler test case than in #13…
import numpy as np
import dill
class TestArray2(np.ndarray):
color = 'blue'
pass
a3 = TestArray2([1,2,3,4,5])
a3.color = 'green'
print(a3.__dict__)
print(dill.copy(a3).__dict__)
print(dill.copy(a3, byref=True).__dict__)
dill.detect.trace(False)
Result is still failure…
{'color': 'green'}
B2: <built-in function _reconstruct>
T2: <class '__main__.TestArray2'>
F2: <function _create_type at 0x110803140>
T1: <type 'type'>
F2: <function _load_type at 0x1108030c8>
T4: <type 'numpy.ndarray'>
D2: <dict object at 0x7f99bc011870>
T4: <type 'numpy.dtype'>
{}
B2: <built-in function _reconstruct>
T5: <class '__main__.TestArray2'>
T4: <type 'numpy.dtype'>
{}
from dill.
where the instance's dict doesn't get saved…
>>> b = TestArray2([1,2,3])
>>> b.color = 'red'
>>>
>>> _b = dill.copy(b)
B2: <built-in function _reconstruct>
T4: <class 'numpy_new.TestArray2'>
T4: <type 'numpy.dtype'>
>>> _b.color
'green'
>>> _b.__dict__
{}
>>> b.__dict__
{'color': 'red'}
from dill.
looks like the mechanism to override is:
>>> n = np.array([1,2,3])
>>> recon, initargs, state = n.__reduce__()
>>> m = np.core.multiarray._reconstruct(*initargs)
>>> m.__setstate__(state)
>>> m
array([1, 2, 3])
and would just need to additionally save the __dict__
in a new reduce method.
Note that the first argument of initargs
is the subclass… (in this case, ndarray
)
>>> initargs
(<type 'numpy.ndarray'>, (0,), 'b')
from dill.
>>> t = TestArray([1,2,3], color='green')
>>> recon, initargs, state = t.__reduce__()
>>> initargs
(<class 'numpy_new.TestArray'>, (0,), 'b')
>>> d = t.__dict__.copy()
>>> m = np.core.multiarray._reconstruct(*initargs)
>>> m.__setstate__(state)
>>> m.__dict__.update(d)
>>>
>>> m
TestArray([1, 2, 3])
>>> m.color
'green'
from dill.
@mattja: nearly fixed in 5f1774d
currently deactivated, since it fails one test for 3.2 - 3.4. Looks ok, but still needs work.
from dill.
Thanks for the quick response!
As you pointed out, the reduce, setstate interface is used by ndarray so
I can see now that overriding these is perhaps what I should have been doing to start with. Now works well.
from dill.
@mattja: True, if you override __reduce__
and __setstate__
, it should work. I added an edit to dill
that should work if you don't… it's just not "switched on" as there's something unresolved with pickling weakref
when it's on. Overrloading the methods numpy
provides is probably a very good approach, and what is expected for subclassing numpy
arrays.
from dill.
done in: eb122e6
from dill.
Related Issues (20)
- UnpicklingError using dill but not stdlib pickle HOT 2
- Compatibility with Pyodide HOT 6
- tfp.math.psd_kernels.ExponentiatedQuadratic no longer serialiases with latest versions
- `dill.load_session()` raises `NameError: name 'UnpicklingError' is not defined` HOT 5
- pydantic>=2.5 classes can't be serialized HOT 4
- Failure to load dumped partial function (in older python) properly (in newer python)
- threading.Thread fails to pickle in python > 3.13.0a5 HOT 9
- Error with typing.AbstractContextManager in Python 3.8 and newer dill versions (>=0.3.6) HOT 1
- File gets truncated upon unpickling HOT 3
- PyTorch C++-generated module "not found as" itself? HOT 1
- Importing dill changes the python pickler behavior HOT 8
- Recursive self-references that trigger a RecursionError
- Attempting to `dill` a function defined in a `doctest` run with `pytest` causes a `TypeError: cannot pickle 'EncodedFile' object`.
- NumPy 2.0.0 test regression: `test_diff.py`: `RecursionError: maximum recursion depth exceeded` HOT 3
- Python 3.13.0b3: `test_detect.py`: `assert len(s) is len(a)` fails HOT 2
- TypeError: 'bytes' object cannot be interpreted as an integer HOT 4
- How to save class definitions from imported modules?
- Does dill have the `register_pickle_by_value` cloudpickle equivalent ?
- Inconsistent serialization behavior between method of imported class and class created in jupyter notebook? HOT 2
- getsource yields python scalars for numpy scalars
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dill.