Giter Club home page Giter Club logo

Comments (9)

mmckerns avatar mmckerns commented on September 19, 2024

while from command line… toggling byrefdoes have some impact. However, we always start with B2, which uses the StockPickler. T5 is also the StockPickler, but T2 is not (it's converting a dictproxy to a dict).

dude@hilbert>$ python numpy_new.py 
{'color': 'green'}
B2: <built-in function _reconstruct>
T2: <class '__main__.TestArray'>
F2: <function _create_type at 0x110c24140>
T1: <type 'type'>
F2: <function _load_type at 0x110c240c8>
T4: <type 'numpy.ndarray'>
D2: <dict object at 0x7fc0490132d0>
T4: <type 'staticmethod'>
F1: <function __getnewargs__ at 0x110c65cf8>
F2: <function _create_function at 0x110c241b8>
Co: <code object __getnewargs__ at 0x1102deb30, file "numpy_new.py", line 16>
F2: <function _unmarshal at 0x110c24050>
D1: <dict object at 0x7fc04841b850>
D2: <dict object at 0x7fc049009b60>
F1: <function __array_finalize__ at 0x110c65c80>
Co: <code object __array_finalize__ at 0x1102de8b0, file "numpy_new.py", line 10>
D1: <dict object at 0x7fc04841b850>
D2: <dict object at 0x7fc049008490>
T4: <type 'numpy.dtype'>
{}
B2: <built-in function _reconstruct>
T5: <class '__main__.TestArray'>
T4: <type 'numpy.dtype'>
{}
{'color': 'green'}
B2: <built-in function _reconstruct>
T2: <class '__main__.TestArray'>
F2: <function _create_type at 0x110c24140>
T1: <type 'type'>
F2: <function _load_type at 0x110c240c8>
T4: <type 'numpy.ndarray'>
D2: <dict object at 0x7fc049007d90>
T4: <type 'staticmethod'>
F1: <function __getnewargs__ at 0x110c65cf8>
F2: <function _create_function at 0x110c241b8>
Co: <code object __getnewargs__ at 0x1102deb30, file "numpy_new.py", line 16>
F2: <function _unmarshal at 0x110c24050>
D1: <dict object at 0x7fc04841b850>
D2: <dict object at 0x7fc049009b60>
F1: <function __array_finalize__ at 0x110c65c80>
Co: <code object __array_finalize__ at 0x1102de8b0, file "numpy_new.py", line 10>
D1: <dict object at 0x7fc04841b850>
D2: <dict object at 0x7fc049008490>
T4: <type 'numpy.dtype'>
{}
B2: <built-in function _reconstruct>
T5: <class '__main__.TestArray'>
T4: <type 'numpy.dtype'>
{}

from dill.

mmckerns avatar mmckerns commented on September 19, 2024

Even simpler test case than in #13

import numpy as np
import dill

class TestArray2(np.ndarray):
    color = 'blue'
    pass

a3 = TestArray2([1,2,3,4,5])
a3.color = 'green'
print(a3.__dict__)
print(dill.copy(a3).__dict__)
print(dill.copy(a3, byref=True).__dict__)
dill.detect.trace(False)

Result is still failure…

{'color': 'green'}
B2: <built-in function _reconstruct>
T2: <class '__main__.TestArray2'>
F2: <function _create_type at 0x110803140>
T1: <type 'type'>
F2: <function _load_type at 0x1108030c8>
T4: <type 'numpy.ndarray'>
D2: <dict object at 0x7f99bc011870>
T4: <type 'numpy.dtype'>
{}
B2: <built-in function _reconstruct>
T5: <class '__main__.TestArray2'>
T4: <type 'numpy.dtype'>
{}

from dill.

mmckerns avatar mmckerns commented on September 19, 2024

where the instance's dict doesn't get saved…

>>> b = TestArray2([1,2,3])
>>> b.color = 'red'
>>> 
>>> _b = dill.copy(b)
B2: <built-in function _reconstruct>
T4: <class 'numpy_new.TestArray2'>
T4: <type 'numpy.dtype'>
>>> _b.color
'green'
>>> _b.__dict__
{}
>>> b.__dict__
{'color': 'red'}

from dill.

mmckerns avatar mmckerns commented on September 19, 2024

looks like the mechanism to override is:

>>> n = np.array([1,2,3])
>>> recon, initargs, state = n.__reduce__()
>>> m = np.core.multiarray._reconstruct(*initargs)
>>> m.__setstate__(state)
>>> m
array([1, 2, 3])

and would just need to additionally save the __dict__ in a new reduce method.

Note that the first argument of initargs is the subclass… (in this case, ndarray)

>>> initargs  
(<type 'numpy.ndarray'>, (0,), 'b')

from dill.

mmckerns avatar mmckerns commented on September 19, 2024
>>> t = TestArray([1,2,3], color='green')
>>> recon, initargs, state = t.__reduce__()
>>> initargs
(<class 'numpy_new.TestArray'>, (0,), 'b')
>>> d = t.__dict__.copy()
>>> m = np.core.multiarray._reconstruct(*initargs)
>>> m.__setstate__(state)
>>> m.__dict__.update(d)
>>> 
>>> m
TestArray([1, 2, 3])
>>> m.color
'green'

from dill.

mmckerns avatar mmckerns commented on September 19, 2024

@mattja: nearly fixed in 5f1774d

currently deactivated, since it fails one test for 3.2 - 3.4. Looks ok, but still needs work.

from dill.

mattja avatar mattja commented on September 19, 2024

Thanks for the quick response!
As you pointed out, the reduce, setstate interface is used by ndarray so
I can see now that overriding these is perhaps what I should have been doing to start with. Now works well.

from dill.

mmckerns avatar mmckerns commented on September 19, 2024

@mattja: True, if you override __reduce__ and __setstate__, it should work. I added an edit to dill that should work if you don't… it's just not "switched on" as there's something unresolved with pickling weakref when it's on. Overrloading the methods numpy provides is probably a very good approach, and what is expected for subclassing numpy arrays.

from dill.

mmckerns avatar mmckerns commented on September 19, 2024

done in: eb122e6

from dill.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.