Feature: monitor external python process, possibly by injecting a stub? about guppy3 HOT 3 OPEN

No, not at the moment.

Guppy uses Python's internal traversal, tp_traverse, to find what each object refers to (see: stdtypes.c#L359 hv.c#L298). This mechanism is also used by the python garbage collector. So at least, guppy needs a stub in the target process to figure out the reference tree. Maybe in the future guppy can use a debugger-like API like ptrace to perform the operation in any target PID, but given that there are so many types Python has, writing the "referees by object" function for each function type could be a huge amount of work. It can't just 'use' the tp_traverse without a way to simulate its execution....

However, if you have Python < 3.9, you can use remote monitor. It will spawn a subinterpreter in a new thread and run guppy from the new subinterpreter, with its stdio attached to a TCP connection. There you can use guppy without interrupting the normal process, though be careful that a lot of libraries don't work well with subinterpreters. Unfortunately, Python 3.9 completely breaks multithreaded subinterpreters (see commit 15fa02d), I'm considering under Python 3.9+ just spawn a new thread and have guppy overate from within, but without subinterpreters this loses the stdio redirection and I don't know of an easy way to workaround :(

from guppy3.

Thanks for your detailed response. I recently came across https://github.com/facebookincubator/memory-analyzer which does exactly that. I hope that this gives you some ideas as well

from guppy3.

Interesting. It seems to inject a piece of python code into the the target process via gdb. Not sure I agree with the runtime safety of doing so but it does sound like an interesting concept.

from guppy3.