Feature: replace/patch an imported class at runtime about guppy3 HOT 15 CLOSED

https://gist.github.com/zhuyifei1999/329a2028295b0c28e27b2dd5f79f83c1 implements the above.

class A:
    attr = 1
class B:
    attr = 2
[...]
print(a, a.attr)
do_replace(A, B)
print(a, a.attr)

<__main__.A object at 0x7fdbb1fa7cd0> 1
<__main__.B object at 0x7fdbb1d52c50> 2

from guppy3.

I don't really understand your example. Do you mean something like this?

a.py
main.py

# a.py
class A():
    foo = 1
    pass
# main.py
import a
obj = a.A()

And then you make a change to a.py

class A():
    foo = 2
    pass

You want to, in main.py

# importlib.reload(a)
guppy.reload(a)
assert obj.foo == 2

from guppy3.

Yes, that's what i want to do. Can guppy make it?

from guppy3.

Or more specifically, not reload but replace class A with another class B:

# a.py
from xxx import A # A is a class
class obj:
    self.some_attrs = A()

# main.py
from xxx import B  # B is also a class like A
from a import obj

---> here replace A with B using guppy
instance = obj() # some_attrs of instance is B
assert isinstance(instance.some_attrs, B)

I can not use monkey-patching to replace xxx.A with xxx.B, so I am looking to see if I could replace A with B directly through memory operation:)

from guppy3.

In theory, guppy can find all the all the referrers of the class, and do its best at attempting to replace the reference to the old object with to the new object. However, you can't change ob_type safely without risking memory corruptions (eg: different slots), so you have to reconstruct the instance. And reconstructing the instance means to replace references to the old instance with to the new instance, so some recursion needed.

To reconstruct an instance... this does not seem to be a minor task. Say you have:

class A:
    __slots__ = ('foo',)

a = A()

and you want to replace A with B where:

class B:
    __slots__ = ('bar',)

You can't fit a.foo into B. Class replacement will not succeed in the general case, but in most sane cases it will probably work.

This is gonna be a difficult one. I will think about it.

from guppy3.

Thank you very much:) In my case, there is no __slots__ in the classes and we can assume that the replacement is happened before A and B are instantiated

from guppy3.

we can assume that the replacement is happened before A and B are instantiated

It makes no difference. I can't allow arbitrary changes to references because it risk breaking Python. For example:

tup = (A,)
dct = {tup: 0}
assert tup in dct
guppy.replace(A, B)
assert tup not in dct
assert tup in set(dct.keys())

The direct update to the element will cause the tuple's hash to change, and as a result it would not be found in dict searching unless the whole dict is rehashed.

There are probably other even more insane ideas if direct 'memory operation' is done, like guppy.replace(str, int) -- I don't want to imagine what would happen.

Anyways, I'm gonna think how to do it, sort of 'smart-monkey-patch' way, with almost pure Python. References can be updated if owner is mutable, else owner is reconstructed. Though, if you have suggestions feel free to explain them.

from guppy3.

To demonstrate hash-changing breaking dicts in a reproducable code:

>>> o = type('', (), {'__hash__': lambda self: 1})()
>>> dct = {o: 0}
>>> o in dct
True
>>> hash(o)
1
>>> o.__class__.__hash__ = lambda self: 2
>>> hash(o)
2
>>> o in dct
False
>>> o in set(dct.keys())
True

from guppy3.

Got it:)

from guppy3.

I'm thinking of a slightly-formal definition of what we are trying to do.

Given an object A in in a reference graph, we want to update all references to A into B.
A reference consists of source object Src, relation Rel, and reference destination Dst.
If given a reference (Src, Rel, Dst) you can change Dst into Dst' where id(Dst') != id(Dst), then the reference is mutable, else the reference is immutable.
For any immutable reference, if the destination is to be replaced with new object, the the source is to be replaced with a new object such that any any outgoing references from the new object will point to new replaced objects.

To keep my sanity I'm thinking of doing it in three passes:

1. Reverse-reference traverse from A, attempt to find any reference that is definitely immutable by writing, but not changing, the destination. If the update fails, the reference is immutable, continue traverse into the source of reference. Else we assume the reference is mutable. Collect sources of at least one immutable references into a set Reconstucts, and sources where all traversed references are mutable into another set Mutates
2. Iteratively, reconstruct every object in set Reconstucts and where all outgoing references point to existing objects (either objects that are not going to be replaced or already reconstructed), until the all objects in Reconstucts has been reconstructed. If, during an iteration, all objects in Reconstucts are either already reconstructed or has a direct reference to a non-existing object, then there must be an immutable reference loop and the update cannot be done.
3. Update all mutable reference destinations to point to newly constructed objects. If update fails on any we-assumed-to-be-mutable reference, then we rollback and bail.

from guppy3.

a = 'abc'
b = 'def'
print(a)
do_replace(a, b)
print(b)

This cannot be done no matter what. The frame contains an immutable reference to the code, which contains an immutable reference to all the code constants (such as literals). It would be fine if no currently-executing frame references the code.

from guppy3.

Thank you so much!! that's so magic. Your description about the problem is exactly what I want to do. I will dive into your codes to learn the details. But i am new of guppy, it may take some time for me to understand your codes:)

Close the issue for now, may trouble you again if there are any problems:)

from guppy3.

a = 'abc'
b = 'def'
print(a)
do_replace(a, b)
print(b)

This cannot be done no matter what. The frame contains an immutable reference to the code, which contains an immutable reference to all the code constants (such as literals). It would be fine if no currently-executing frame references the code.

I just want to change the reference of customize classes, it's ok that things like literals can not be changed:)

from guppy3.

I'm thinking of either publishing the file as a separate package or integrate it into guppy. There seem to already be some similar works before:

• https://pythonhosted.org/globalsub/#how-it-works
• https://github.com/cart0113/pyjack/blob/master/pyjack.py#L146
• https://gist.github.com/anonymous/74ae8e03a55a6a3ea533

But they rely on gc, which has gc limitations (limited by what gc sees).

Then there is this which uses guppy's traversal:

• https://benkurtovic.com/2015/01/28/python-object-replacement.html

But it isn't as robust and won't perform object reconstruction.

from guppy3.

Integrate it into guppy as a showcase to demonstrate its power may be a good choice:)

from guppy3.