zhuyifei1999 / guppy3 Goto Github PK

View Code? Open in Web Editor NEW

373.0 5.0 23.0 1.58 MB

guppy / heapy ported to Python3. It works for real!

Home Page: https://zhuyifei1999.github.io/guppy3/

License: MIT License

Emacs Lisp 0.81% Python 71.27% C 27.92%

guppy3's Introduction

Guppy 3

A Python Programming Environment & Heap analysis toolset.

This package contains the following subpackages:

etc - Support modules. Contains especially the Glue protocol module.
gsl - The Guppy Specification Language implementation. This can be used to create documents and tests from a common source.
heapy - The heap analysis toolset. It can be used to find information about the objects in the heap and display the information in various ways.
sets - Bitsets and 'nodesets' implemented in C.

Guppy 3 is a fork of Guppy-PE, created by Sverker Nilsson for Python 2.

Requirements

You should have Python 3.8, 3.9, 3.10, 3.11, or 3.12. This package is CPython only; PyPy and other Python implementations are not supported. Python 2 support can be obtained from guppy-pe by Sverker Nilsson, from which this package is forked.

To use the graphical browser, Tkinter is needed. To use the remote monitor, threading must be available.

Installation

Install with pip by:

pip install guppy3

Install with conda by:

conda install -c conda-forge guppy3

Usage

The following example shows

How to create the session context: h=hpy()
How to show the reachable objects in the heap: h.heap()
How to show the shortest paths from the root to the single largest object: h.heap().byid[0].sp
How to create and show a set of objects: h.iso(1,[],{})

>>> from guppy import hpy; h=hpy()
>>> h.heap()
Partition of a set of 30976 objects. Total size = 3544220 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0   8292  27   739022  21    739022  21 str
     1   7834  25   625624  18   1364646  39 tuple
     2   2079   7   300624   8   1665270  47 types.CodeType
     3    400   1   297088   8   1962358  55 type
     4   4168  13   279278   8   2241636  63 bytes
     5   1869   6   269136   8   2510772  71 function
     6    400   1   228464   6   2739236  77 dict of type
     7     79   0   139704   4   2878940  81 dict of module
     8   1061   3    93368   3   2972308  84 types.WrapperDescriptorType
     9    172   1    81712   2   3054020  86 dict (no owner)
<89 more rows. Type e.g. '_.more' to view.>
>>> h.heap().byid[0].sp
 0: h.Root.i0_modules['os'].__dict__
>>> h.iso(1,[],{})
Partition of a set of 3 objects. Total size = 348 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  33      248  71       248  71 dict (no owner)
     1      1  33       72  21       320  92 list
     2      1  33       28   8       348 100 int
>>>

People have written awesome posts on how to use this toolset, including:

Formal and API documentation are also available.

Contributing

Issues and pull requests are welcome. You may also ask for help on using this toolset; however, in such cases, we will only provide guidance, and not profile your code for you.

Please make sure to update tests as appropriate.

Testing

To test if the heapy build and installation was ok, you can do:

>>> from guppy import hpy
>>> hpy().test()
Testing sets
Test #0
Test #1
Test #2
...

There will be several more tests. Some tests may take a while.

License

The right is granted to copy, use, modify and redistribute this code according to the rules in what is commonly referred to as an MIT license.

This is not an official Google product.

*** USE AT YOUR OWN RISK AND BE AWARE THAT THIS IS AN EARLY RELEASE ***

guppy3's People

Contributors

Stargazers

Watchers

guppy3's Issues

test_horizon Segfault in horizon_get_org_dealloc on guppy.heapy.View.Horizon object after horizon_remove called

Tracing some relevant functions:

test_horizon (guppy.heapy.test.test_View.FirstCase) ... [pid 2010953] horizon_new(0x7ffff6990940 <NyHorizon_Type>, (<guppy.heapy.heapyc.HeapView at remote 0x7ffff606c898>,), 0x0) = <unfinished ...>
[pid 2010953] horizon_patch_dealloc(0x7ffff6990c80 <NyRootState_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d4eb40 <PyDict_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d48860 <PyModule_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d4ff00 <PyList_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d4f840 <PyCFunction_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d4ad00 <PyFrame_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d48520 <_PyNone_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d4cf00 <PyCode_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d4f360 <PyBaseObject_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d4f500 <PyType_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d500a0 <PyTuple_Type>) = 0x0
[...]
[pid 2010953] horizon_patch_dealloc(0x7ffff7d5d640 <PyBufferedReader_Type>) = 0x0
[pid 2010953] horizon_patch_dealloc(0x7ffff7d5b920 <ZipImporter_Type>) = 0x0
[pid 2010953] horizon_patched_dealloc([]) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.MutNodeSet at remote 0x7ffff6452fc0>) = <unfinished ...>
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x5555559cbe10>) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x555555b109f0>) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x555555af9ad0>) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x5555559bb520>) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x555555cc95d0>) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x555555afff30>) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x555555b02760>) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x5555559664c0>) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x555555b1b7c0>) = void
[pid 2010953] <... horizon_patched_dealloc resumed> = void
[pid 2010953] horizon_patch_dealloc(0x7ffff6990940 <NyHorizon_Type>) = 0x0
[pid 2010953] <... horizon_new resumed> = <guppy.heapy.heapyc.Horizon at remote 0x7ffff63430b0>
[pid 2010953] horizon_patched_dealloc((<guppy.heapy.heapyc.HeapView at remote 0x7ffff606c898>,)) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x555555ae6898, for file /home/zhuyifei1999/guppy3/guppy/heapy/View.py, line 15, in __init__ (self=<Horizon(mod=<Interface(_share=<Share(module=<module at remote 0x7ffff69c89a8>, parent=<Share(module=<module at remote 0x7ffff6a578b8>, parent=<Share(module=<module at remote 0x7ffff6a577c8>, parent=<Share(module=None, parent=<...>, name='', setable=(), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': None, 'guppy': <...>, 'types': <Share(module=<module at remote 0x7ffff6ac24a8>, parent=<...>, name='types', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7ffff6ac24a8>}, owners={}, ispackage=None, recursion=0, preload=(...)) at remote 0x7ffff5cddcf8>, 'sys': <Share(module=<module at remote 0x7ffff6c1eef8>, parent=<...>, name='sys', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={...(truncated)) = void
[...]
[pid 2010953] horizon_patched_dealloc(Frame 0x7ffff6246930, for file /home/zhuyifei1999/guppy3/guppy/heapy/View.py, line 19, in news (self=<Horizon(mod=<Interface(_share=<Share(module=<module at remote 0x7ffff69c89a8>, parent=<Share(module=<module at remote 0x7ffff6a578b8>, parent=<Share(module=<module at remote 0x7ffff6a577c8>, parent=<Share(module=None, parent=<...>, name='', setable=(), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': None, 'guppy': <...>, 'types': <Share(module=<module at remote 0x7ffff6ac24a8>, parent=<...>, name='types', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7ffff6ac24a8>}, owners={}, ispackage=None, recursion=0, preload=(...)) at remote 0x7ffff5cddcf8>, 'sys': <Share(module=<module at remote 0x7ffff6c1eef8>, parent=<...>, name='sys', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_pa...(truncated)) = void
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.MutNodeSet at remote 0x7ffff5f37630>) = <unfinished ...>
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmBitSet at remote 0x7ffff696bb30>) = void
[pid 2010953] <... horizon_patched_dealloc resumed> = void
[pid 2010953] horizon_patched_dealloc(((), [])) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x7ffff6122ac8, for file /home/zhuyifei1999/guppy3/guppy/heapy/ImpSet.py, line 20, in immnodeset (self=<Interface(_share=<Share(module=<module at remote 0x7ffff61371d8>, parent=<Share(module=<module at remote 0x7ffff6a578b8>, parent=<Share(module=<module at remote 0x7ffff6a577c8>, parent=<Share(module=None, parent=<...>, name='', setable=(), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': None, 'guppy': <...>, 'types': <Share(module=<module at remote 0x7ffff6ac24a8>, parent=<...>, name='types', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7ffff6ac24a8>}, owners={}, ispackage=None, recursion=0, preload=(...)) at remote 0x7ffff5cddcf8>, 'sys': <Share(module=<module at remote 0x7ffff6c1eef8>, parent=<...>, name='sys', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent'...(truncated)) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x7ffff61ab788, for file /home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py, line 1516, in _cons (self=<IdentitySetFamily(mod=<Interface(_share=<Share(module=<module at remote 0x7ffff6137638>, parent=<Share(module=<module at remote 0x7ffff6a578b8>, parent=<Share(module=<module at remote 0x7ffff6a577c8>, parent=<Share(module=None, parent=<...>, name='', setable=(), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': None, 'guppy': <...>, 'types': <Share(module=<module at remote 0x7ffff6ac24a8>, parent=<...>, name='types', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7ffff6ac24a8>}, owners={}, ispackage=None, recursion=0, preload=(...)) at remote 0x7ffff5cddcf8>, 'sys': <Share(module=<module at remote 0x7ffff6c1eef8>, parent=<...>, name='sys', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrappin...(truncated)) = <unfinished ...>
[pid 2010953] horizon_patched_dealloc(<guppy.sets.setsc.ImmNodeSet at remote 0x7ffff5f37f30>) = void
[pid 2010953] <... horizon_patched_dealloc resumed> = void
[pid 2010953] horizon_patched_dealloc(<method at remote 0x7ffff5c9f488>) = void
[pid 2010953] horizon_patched_dealloc(((),)) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x7ffff611fcc0, for file /home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py, line 1510, in __call__ (self=<IdentitySetFamily(mod=<Interface(_share=<Share(module=<module at remote 0x7ffff6137638>, parent=<Share(module=<module at remote 0x7ffff6a578b8>, parent=<Share(module=<module at remote 0x7ffff6a577c8>, parent=<Share(module=None, parent=<...>, name='', setable=(), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': None, 'guppy': <...>, 'types': <Share(module=<module at remote 0x7ffff6ac24a8>, parent=<...>, name='types', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7ffff6ac24a8>}, owners={}, ispackage=None, recursion=0, preload=(...)) at remote 0x7ffff5cddcf8>, 'sys': <Share(module=<module at remote 0x7ffff6c1eef8>, parent=<...>, name='sys', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrap...(truncated)) = <unfinished ...>
[pid 2010953] horizon_patched_dealloc({}) = void
[pid 2010953] <... horizon_patched_dealloc resumed> = void
[pid 2010953] horizon_patched_dealloc(<method at remote 0x7ffff5c9fe48>) = void
[pid 2010953] horizon_patched_dealloc(<method at remote 0x7ffff5cfef08>) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x555555ae1898, for file /home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py, line 82, in __le__ (self=<IdentitySetMulti at remote 0x7ffff5ee7888>, other=<...>)) = void
[pid 2010953] horizon_patched_dealloc((<IdentitySetMulti at remote 0x7ffff5ee7888>,)) = void
[pid 2010953] horizon_patched_dealloc(<method at remote 0x7ffff5c9f488>) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x7ffff5ff6d68, for file /home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py, line 58, in __ge__ (self=<IdentitySetMulti at remote 0x7ffff5ee7888>, other=<...>)) = void
[pid 2010953] horizon_patched_dealloc((<IdentitySetMulti at remote 0x7ffff5ee7888>,)) = void
[pid 2010953] horizon_patched_dealloc(<method at remote 0x7ffff5c9f488>) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x7ffff617d408, for file /home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py, line 37, in __eq__ (self=<IdentitySetMulti at remote 0x7ffff5ee7888>, other=<...>)) = void
[pid 2010953] horizon_patched_dealloc((<IdentitySetMulti at remote 0x7ffff5ee7888>,)) = void
[pid 2010953] horizon_patched_dealloc(<method at remote 0x7ffff5c9fe48>) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x555555899f78, for file /home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py, line 149, in __ne__ (self=<IdentitySetMulti at remote 0x7ffff5ee7888>, other=<...>)) = void
[pid 2010953] horizon_patched_dealloc((<IdentitySetMulti at remote 0x7ffff5ee7888>,)) = void
[pid 2010953] horizon_patched_dealloc(<method at remote 0x7ffff5cfef08>) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x555555885948, for file /home/zhuyifei1999/guppy3/guppy/heapy/test/support.py, line 103, in aseq (self=<FirstCase(_testMethodName='test_horizon', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=4, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7ffff6c40630>, _original_stderr=<_io.TextIOWrapper at remote 0x7ffff6c40708>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7ffff6c40630>) at remote 0x7ffff6909470>, showAll=True, dots=False, descriptions=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x5555559831f8>) at remote 0x7ffff5f93be0>, result_supports_subtests=True, success=True, skipped=[], expectedFailure=None, errors=[(<...>, None)]) at remote 0x7ffff5f50eb8>, _testMethodDoc=None, _cleanups=[], _subtest=None,...(truncated)) = void
[pid 2010953] horizon_patched_dealloc(Frame 0x55555590f5b8, for file /home/zhuyifei1999/guppy3/guppy/heapy/test/test_View.py, line 73, in test_horizon (self=<FirstCase(_testMethodName='test_horizon', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=4, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7ffff6c40630>, _original_stderr=<_io.TextIOWrapper at remote 0x7ffff6c40708>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7ffff6c40630>) at remote 0x7ffff6909470>, showAll=True, dots=False, descriptions=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x5555559831f8>) at remote 0x7ffff5f93be0>, result_supports_subtests=True, success=True, skipped=[], expectedFailure=None, errors=[(<...>, None)]) at remote 0x7ffff5f50eb8>, _testMethodDoc=None, _cleanups=[], _subt...(truncated)) = <unfinished ...>
[pid 2010953] horizon_patched_dealloc({'mod': <Interface(_share=<Share(module=<module at remote 0x7ffff69c89a8>, parent=<Share(module=<module at remote 0x7ffff6a578b8>, parent=<Share(module=<module at remote 0x7ffff6a577c8>, parent=<Share(module=None, parent=<...>, name='', setable=(), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': None, 'guppy': <...>, 'types': <Share(module=<module at remote 0x7ffff6ac24a8>, parent=<...>, name='types', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7ffff6ac24a8>}, owners={}, ispackage=None, recursion=0, preload=(...)) at remote 0x7ffff5cddcf8>, 'sys': <Share(module=<module at remote 0x7ffff6c1eef8>, parent=<...>, name='sys', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7ffff6c1eef8>, 'exc_info': <built-in method e...(truncated)) = <unfinished ...>
[pid 2010953] horizon_patched_dealloc(<guppy.heapy.heapyc.Horizon at remote 0x7ffff63430b0>) = <unfinished ...>
[pid 2010953] horizon_remove(0x7ffff63430b0) = void
[pid 2010953] <... horizon_patched_dealloc resumed> = void
[pid 2010953] <... horizon_patched_dealloc resumed> = void
[pid 2010953] horizon_patched_dealloc(<Horizon at remote 0x7ffff5d08278>) = 
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
PyDict_GetItem (op=0x0, key=<type at remote 0x7ffff7d4f360>) at /usr/src/debug/dev-lang/python-3.6.9/Python-3.6.9/Objects/dictobject.c:1411
1411	    if (!PyDict_Check(op))

gef➤  bt
#0  PyDict_GetItem (op=0x0, key=<type at remote 0x7ffff7d4f360>) at /usr/src/debug/dev-lang/python-3.6.9/Python-3.6.9/Objects/dictobject.c:1411
#1  0x00007ffff697c72f in horizon_get_org_dealloc (t=0x7ffff7d4f360 <PyBaseObject_Type>) at /home/zhuyifei1999/guppy3/src/heapy/horizon.c:38
#2  0x00007ffff697ca43 in horizon_patched_dealloc (v=<Horizon at remote 0x7ffff5d08278>) at /home/zhuyifei1999/guppy3/src/heapy/horizon.c:98
#3  0x00007ffff7c031a3 in subtype_dealloc (self=<Horizon at remote 0x7ffff5d08278>) at /usr/src/debug/dev-lang/python-3.6.9/Python-3.6.9/Objects/typeobject.c:1222
#4  0x00007ffff7c5041b in frame_dealloc (f=Frame 0x55555590f5b8, for file /home/zhuyifei1999/guppy3/guppy/heapy/test/test_View.py, line 73, in test_horizon (hn=<IdentitySetMulti at remote 0x7ffff5ee7888>)) at /usr/src/debug/dev-lang/python-3.6.9/Python-3.6.9/Objects/frameobject.c:462
#5  0x00007ffff697ca4c in horizon_patched_dealloc (v=Frame 0x55555590f5b8, for file /home/zhuyifei1999/guppy3/guppy/heapy/test/test_View.py, line 73, in test_horizon (hn=<IdentitySetMulti at remote 0x7ffff5ee7888>)) at /home/zhuyifei1999/guppy3/src/heapy/horizon.c:98
#6  0x00007ffff7b8115f in _PyFunction_FastCall (co=<optimized out>, args=<optimized out>, nargs=0x1, globals=<optimized out>) at /usr/src/debug/dev-lang/python-3.6.9/Python-3.6.9/Python/ceval.c:4936
#7  0x00007ffff7b82de2 in fast_function (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/dev-lang/python-3.6.9/Python-3.6.9/Python/ceval.c:4975
#8  0x00007ffff7b82ee5 in call_function (pp_stack=0x7fffffffb008, oparg=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/dev-lang/python-3.6.9/Python-3.6.9/Python/ceval.c:4872
#9  0x00007ffff7b87ef4 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/dev-lang/python-3.6.9/Python-3.6.9/Python/ceval.c:3335
#10 0x00007ffff7b82a41 in PyEval_EvalFrameEx (throwflag=0x0, f=Frame 0x5555558bab38, for file /usr/lib64/python3.6/unittest/case.py, line 605, in run (self=<FirstCase(_testMethodName='test_horizon', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=4, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7ffff6c40630>, _original_stderr=<_io.TextIOWrapper at remote 0x7ffff6c40708>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7ffff6c40630>) at remote 0x7ffff6909470>, showAll=True, dots=False, descriptions=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x5555559831f8>) at remote 0x7ffff5f93be0>, result_supports_subtests=True, success=True, skipped=[], expectedFailure=None, errors=[(<...>, None)]) at remote 0x7ffff5f50eb8>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_f...(truncated)) at /usr/src/debug/dev-lang/python-3.6.9/Python-3.6.9/Python/ceval.c:754
[...]

gef➤  py-bt
Traceback (most recent call first):
  File "/usr/lib64/python3.6/unittest/case.py", line 605, in run
    testMethod()
  File "/usr/lib64/python3.6/unittest/case.py", line 653, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib64/python3.6/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib64/python3.6/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib64/python3.6/unittest/runner.py", line 176, in run
    test(result)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/support.py", line 51, in run_suite
    result = runner.run(suite)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/support.py", line 73, in run_unittest
    run_suite(suite, testclass)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/test_View.py", line 275, in test_main
    support.run_unittest(FirstCase, debug)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/test_all.py", line 36, in test_main
    f(debug=debug)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/Use.py", line 363, in test
    self._parent.test.test_all.test_main(debug)
  File "test.py", line 8, in <module>
    hpy().test()

How could this work?

AttributeError with guppy.heapy.UniSet.IdentitySetMulti.partition

When trying to print heap, I got an AttributeError. I tweaked the code a bit in UniSet.py to report the types of self and a as well as b:

File "/home/r_pang_apple_com/.local/lib/python3.8/site-packages/guppy/heapy/UniSet.py", line 336, in getitem
def getitem(self, idx): return self.fam.c_getitem(self, idx)
File "/home/r_pang_apple_com/.local/lib/python3.8/site-packages/guppy/heapy/UniSet.py", line 1613, in c_getitem
return a.partition.get_set(idx)
File "/home/r_pang_apple_com/.local/lib/python3.8/site-packages/guppy/heapy/UniSet.py", line 74, in getattr
return self.fam.mod.View.enter(lambda: self.fam.c_getattr(self, other))
File "/home/r_pang_apple_com/.local/lib/python3.8/site-packages/guppy/heapy/View.py", line 256, in enter
retval = func()
File "/home/r_pang_apple_com/.local/lib/python3.8/site-packages/guppy/heapy/UniSet.py", line 74, in
return self.fam.mod.View.enter(lambda: self.fam.c_getattr(self, other))
File "/home/r_pang_apple_com/.local/lib/python3.8/site-packages/guppy/heapy/UniSet.py", line 800, in c_getattr
return self.c_getattr2(a, b)
File "/home/r_pang_apple_com/.local/lib/python3.8/site-packages/guppy/heapy/UniSet.py", line 803, in c_getattr2
raise AttributeError(f"{type(self)}.{type(a)}.{b}")
AttributeError: <class 'guppy.heapy.UniSet.IdentitySetFamily'>.<class 'guppy.heapy.UniSet.IdentitySetMulti'>.partition

Is this expected?

Thanks.

Heisenbug: test_RefPat.RefPatCase.test_presentation fails sometimes on Python 3.9 on Windows

Failure runs:

Seems to be the refpat sometimes terminates prematurely with [R],

Attempts to reproduce by a Windows development machine fails.
Attempts to reproduce by ssh-ing to GitHub Actions fails, and the test passes post-ssh: https://github.com/zhuyifei1999/guppy3/actions/runs/1663126825

PathsIter missing next method

When doing the following we get an exception. It seems Python3 needs the next method for iterator not the "_ next _" method. When I define the next method in PathsIter in guppy/heapy/Path.py it works. I have tested in the Guppy3-3.0.3 release as well as latest trunk as of today 10 sep 2019.

Python 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.

from guppy import hpy
h=hpy()
hp=h.heap()
hp
Partition of a set of 34117 objects. Total size = 2248255 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 9003 26 595755 26 595755 26 str
1 8437 25 303296 13 899051 40 tuple
2 4369 13 229084 10 1128135 50 bytes
3 2187 6 183708 8 1311843 58 types.CodeType
4 402 1 177108 8 1488951 66 type
5 2141 6 145588 6 1634539 73 function
6 402 1 120156 5 1754695 78 dict of type
7 93 0 82324 4 1837019 82 dict of module
8 189 1 70004 3 1907023 85 dict (no owner)
9 396 1 44352 2 1951375 87 set
<93 more rows. Type e.g. '_.more' to view.>
hp.sp
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.6/dist-packages/guppy3-3.0.3-py3.6-linux-i686.egg/guppy/heapy/Path.py", line 330, in repr
self.pp(output=f)
File "/usr/local/lib/python3.6/dist-packages/guppy3-3.0.3-py3.6-linux-i686.egg/guppy/heapy/Path.py", line 412, in pp
self.more(start, output=output)
File "/usr/local/lib/python3.6/dist-packages/guppy3-3.0.3-py3.6-linux-i686.egg/guppy/heapy/Path.py", line 258, in call
self.printiter(it, output)
File "/usr/local/lib/python3.6/dist-packages/guppy3-3.0.3-py3.6-linux-i686.egg/guppy/heapy/Path.py", line 433, in printiter
it.next().pp(output=output)
AttributeError: 'PathsIter' object has no attribute 'next'

Use commas for big numbers

This is my dump from Heapy:

Partition of a set of 1235525 objects. Total size = 153363236 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 260111  21 34847420  23  34847420  23 str
     1  82607   7 16193248  11  51040668  33 dict (no owner)
     2 102994   8 14936792  10  65977460  43 bytes
     3 189724  15 14334888   9  80312348  52 tuple
     4  73463   6  9990968   7  90303316  59 function
     5  48013   4  8660794   6  98964110  65 types.CodeType
     6  64567   5  5171896   3 104136006  68 list
     7   4888   0  4896088   3 109032094  71 type
     8   2817   0  4368312   3 113400406  74 dict of module
     9   4888   0  2434624   2 115835030  76 dict of type
<1421 more rows. Type e.g. '_.more' to view.>

Numbers like 153363236 bytes are difficult to read. 153,363,236 bytes would be easier.

Add support to release aarch64 wheels

Problem

On aarch64, pip install guppy3 builds the wheels from source code and then install it. It requires user to have development environment installed on his system. also, it take some time to build the wheels than downloading and extracting the wheels from pypi.

Resolution

On aarch64, pip install guppy3 should download the wheels from pypi

@zhuyifei1999 , Please let me know your interest on releasing aarch64 wheels. I can help in this.

Performance hit when classifying many dicts (non-GC-tracked and with no owner)

This occured in commit 6373998 which was apparently after V3.0.6 was released.

In hv_cli_dictof_update_new_method in src/hv_cli_dictof.c, the call to hv_get_objects was replaced with a call to gc_get_objects. This was to be faster because of an easier way to find objects and they were also fewer. However, we didn't get non-gc-collected objects such as simple dicts with non-gc-collected members. This made the hv_cli_dictof_update method be called each time we tried to classify such dicts, from hv_cli_dictof_classify, because we didn't store any information about those dicts in the nodegraph self->owners. The classifier thought it needed more information each time.

Code classifying an idset() of a list of 5000 empty dicts went from about 0.3 seconds (with the change reverted) to 9 seconds in Python 3 with the modification mentioned.

# dictofno.py
from guppy import hpy
h=hpy()
d=[{} for i in range(5000)]
print (h.idset(d))

Checking in the debugger revealed that the update function was called many times.

It has been suggested (by the author of Guppy 3, YiFei Zhu) that the hv_cli_dictof_classify function should call the update method only once.

I presume this means only once during the lifetime of the classifier. I think this would work if the classifier is allocated together with the set it classifies, because those sets are immutable so the classification would not change. I think this is the current situation.

However, if we would turn to have memoized classifiers in the Heapy guppy.hpy() structure, (to speed things up) then there might be the case that new dicts are created that should require to update the dict ownership graph.

Then it would require many calls to update during the lifetime of the classifier. I don't know if this will be a reality but it may be something to consider.

str() doesn't work on .more though repr() does (in the .byid case at least)

Example:

>>> from guppy import hpy
>>> h=hpy()
>>> x=h.idset(range(100)).byid
>>> print(repr(x.more))
 Index     Size   %   Cumulative  %   Value
    10       14   1.0       154  11.0 11
    11       14   1.0       168  12.0 12
    12       14   1.0       182  13.0 13
    13       14   1.0       196  14.0 14
    14       14   1.0       210  15.0 15
    15       14   1.0       224  16.0 16
    16       14   1.0       238  17.0 17
    17       14   1.0       252  18.0 18
    18       14   1.0       266  19.0 19
    19       14   1.0       280  20.0 20
<80 more rows. Type e.g. '_.more' to view.>
>>> print(str(x.more))
Set of 100 <int> objects. Total size = 1398 bytes.
 Index     Size   %   Cumulative  %   Value
     0       14   1.0        14   1.0 1
     1       14   1.0        28   2.0 2
     2       14   1.0        42   3.0 3
     3       14   1.0        56   4.0 4
     4       14   1.0        70   5.0 5
     5       14   1.0        84   6.0 6
     6       14   1.0        98   7.0 7
     7       14   1.0       112   8.0 8
     8       14   1.0       126   9.0 9
     9       14   1.0       140  10.0 10
<90 more rows. Type e.g. '_.more' to view.>
>>>

Exception in creating hpy instance

Hi Experts,

Meet an exception in creating hpy instance like below:
2024-04-01T07:21:41Z Wa(180)[+] sha: h=guppy.hpy()
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/init.py", line 38, in hpy
2024-04-01T07:21:41Z Wa(180)[+] sha: return r.guppy.heapy.Use
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 50, in getattr
2024-04-01T07:21:41Z Wa(180)[+] sha: return self._share.getattr(self, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 219, in getattr
2024-04-01T07:21:41Z Wa(180)[+] sha: d = self.getattr2(inter, cache, owner, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 236, in getattr2
2024-04-01T07:21:41Z Wa(180)[+] sha: x = self.getattr_package(inter, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 284, in getattr_package
2024-04-01T07:21:41Z Wa(180)[+] sha: x = self.makeModule(x, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 350, in makeModule
2024-04-01T07:21:41Z Wa(180)[+] sha: return Share(module, self, module.name, Clamp)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 208, in init
2024-04-01T07:21:41Z Wa(180)[+] sha: getattr(inter, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 50, in getattr
2024-04-01T07:21:41Z Wa(180)[+] sha: return self._share.getattr(self, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 219, in getattr
2024-04-01T07:21:41Z Wa(180)[+] sha: d = self.getattr2(inter, cache, owner, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 238, in getattr2
2024-04-01T07:21:41Z Wa(180)[+] sha: x = self.getattr3(inter, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 323, in getattr3
2024-04-01T07:21:41Z Wa(180)[+] sha: pa = getattr(pa, at)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 50, in getattr
2024-04-01T07:21:41Z Wa(180)[+] sha: return self._share.getattr(self, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 219, in getattr
2024-04-01T07:21:41Z Wa(180)[+] sha: d = self.getattr2(inter, cache, owner, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 236, in getattr2
2024-04-01T07:21:41Z Wa(180)[+] sha: x = self.getattr_package(inter, name)
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/etc/Glue.py", line 273, in getattr_package
2024-04-01T07:21:41Z Wa(180)[+] sha: x = import(self.makeName(name))
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/heapy/View.py", line 605, in
2024-04-01T07:21:41Z Wa(180)[+] sha: prime_builtin_types()
2024-04-01T07:21:41Z Wa(180)[+] sha: File "/usr/lib/x/y/lib/python/guppy/heapy/View.py", line 551, in prime_builtin_types
2024-04-01T07:21:41Z Wa(180)[+] sha: if isinstance(t, types):
2024-04-01T07:21:41Z Wa(180)[+] sha: TypeError: isinstance() arg 2 must be a type or tuple of types

To accommodate our platform, I compiled the source to build a manylinux version wheel, then unzip and copy the stuff in the package to the platform for reference.

BR/Amber

TypeError: '<' not supported between instances of 'weakref' and 'weakref'

Hi. I'm trying to execute ">>> x.shpaths" for some x, which results in the following stack trace.

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/OutputHandling.py", line 386, in reprfunc
    return self.printer.get_str_of_top()
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/OutputHandling.py", line 272, in get_str_of_top
    return self.get_str(self, True)
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/OutputHandling.py", line 339, in get_str
    return self.mod.View.enter(lambda: f())
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/View.py", line 256, in enter
    retval = func()
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/OutputHandling.py", line 339, in <lambda>
    return self.mod.View.enter(lambda: f())
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/OutputHandling.py", line 307, in f
    nxt = next(it)
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/OutputHandling.py", line 232, in lines_from
    yield self.line_at(idx)
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/OutputHandling.py", line 212, in line_at
    li = next(self.line_iter)
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/Path.py", line 368, in _oh_get_line_iter
    for el in self:
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/Path.py", line 289, in __iter__
    return self.iter()
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/Path.py", line 292, in iter
    return PathsIter(self, start, stop)
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/Path.py", line 176, in __init__
    self.reset(start)
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/Path.py", line 209, in reset
    sr = self.mod.sortedrels(self.paths.IG, src)
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/Path.py", line 462, in sortedrels
    t.sort(key=lambda x: x[0])
  File "/home/pino/projects/krrez_9_venv/lib/python3.9/site-packages/guppy/heapy/Path.py", line 94, in __lt__
    return self.r < other.r
TypeError: '<' not supported between instances of 'weakref' and 'weakref'

python.exe crashed on hpy.heap() after Import Official Dropbox SDK for Python

Hi @zhuyifei1999,

Happened in Windows OS.

Installed Official Dropbox SDK from https://pypi.org/project/dropbox/.
pip install dropbox

Ran Python Interpreter
python

Python 3.8.10 (default, May 19 2021, 13:12:57) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.

import dropbox
import guppy
hp = guppy.hpy()
before = hp.heap()

Windows thrown out exception:

Python has stopped working.
Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	python.exe
  Application Version:	3.8.10150.1013
  Application Timestamp:	60a5556d
  Fault Module Name:	python38.dll
  Fault Module Version:	3.8.10150.1013
  Fault Module Timestamp:	60a55542
  Exception Code:	c0000005
  Exception Offset:	000000000006dc5b
  Locale ID:	17417
  Additional Information 1:	b59d
  Additional Information 2:	b59df1129b1415b7f11d657440546bce
  Additional Information 3:	53eb
  Additional Information 4:	53eb508f924550baeb8a6a8bb1af95dd

If skip importing Dropbox SDK or hpy().heap(), there is no issue.

It is a showstopper issue to me as I'm looking forward for both dropbox and guppy module to work together.

Idea: by 'producer' / by 'line of allocation' classifier?

Quote the thesis:

A producer profile classifies cells by the program components that created them.

Of these, the producer profile would require special instrumentation of the Python interpreter to record the call stack when allocating objects, and is therefore outside the current scope of Heapy.

But we no longer need special instrumentation now that Python 3 supports it natively:

$ python -X tracemalloc=10
Python 3.7.4 (default, Aug 26 2019, 23:27:06) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tracemalloc
>>> for f in tracemalloc.get_object_traceback(tracemalloc.get_object_traceback).format(): print(f)
... 
  File "<stdin>", line 1
  File "<frozen importlib._bootstrap>", line 983
  File "<frozen importlib._bootstrap>", line 967
  File "<frozen importlib._bootstrap>", line 677
  File "<frozen importlib._bootstrap_external>", line 728
  File "<frozen importlib._bootstrap>", line 219
  File "/usr/lib/python3.7/tracemalloc.py", line 235
    def get_object_traceback(obj):

We could probably somehow use tracemalloc instead of hook into python memory allocator to avoid reinventing the wheel. However, this has a non-negligible memory overhead so we probably don't want to enable it as default and show a message if no object would be classified this way.

Similarly, it would be nice if we could get the trace of a single object with some convenient attribute access on an identityset. Though, admittedly, reference graphs are far more useful.

How should this classifier be called? byprod? bytrace? byalloc?

Deal with upcoming per-interpreter GIL

Per-interpreter GIL might be coming to Python 3.9 [1]. We have zero thread-safety and rely on GIL to make sure other threads don't modify stuffs when we are accessing them. With per-interpreter GIL, traversing across interpreter boundary is unsafe... ideas?

[1] https://bugs.python.org/issue40512

Exception on hpy.heap()

Getting exception output after this:

from guppy import hpy
h=hpy().heap()
print(h)

This is the exception:

Traceback (most recent call last):
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\etc\Glue.py", line 216, in getattr2
    x = self.data[name]
KeyError: 'hv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\etc\Glue.py", line 281, in getattr3
    x = getattr(Clamp, name)
AttributeError: type object '_GLUECLAMP_' has no attribute 'hv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\juan0\Documents\GitHub\VMIwithDRL\VMIwithDRL\src\implementation\VMImodel.py", line 2, in <module>
    from agent_model.training_agent import TrainingAgent
  File "C:\Users\juan0\Documents\GitHub\VMIwithDRL\VMIwithDRL\src\agent_model\training_agent.py", line 8, in <module>
    h=hpy().heap()
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\heapy\Use.py", line 181, in heap
    h = self.View.heap()
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\heapy\View.py", line 353, in heap
    self.root = objs
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\etc\Glue.py", line 53, in __setattr__
    return self._share.setattr(self, name, value)
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\etc\Glue.py", line 371, in setattr
    im(inter, value)
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\heapy\View.py", line 191, in _set_root
    self.hv.root = root
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\etc\Glue.py", line 50, in __getattr__
    return self._share.getattr(self, name)
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\etc\Glue.py", line 209, in getattr
    d = self.getattr2(inter, dct, owner, name)
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\etc\Glue.py", line 227, in getattr2
    x = self.getattr3(inter, name)
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\etc\Glue.py", line 321, in getattr3
    x = f()
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\heapy\View.py", line 162, in _get_hv
    is_hiding_calling_interpreter=self.is_hiding_calling_interpreter)
  File "C:\Users\juan0\Anaconda3\envs\tensor-cpu\lib\site-packages\guppy\heapy\View.py", line 394, in new_hv
    hv = self.heapyc.HeapView(root, heapdefs)
TypeError: heapdefs must be a capsule object

sdist not including a lot of C source code

I'm about to do a release, and this happened:

$ python setup.py sdist
running sdist
running egg_info
writing guppy3.egg-info/PKG-INFO
writing dependency_links to guppy3.egg-info/dependency_links.txt
writing top-level names to guppy3.egg-info/top_level.txt
reading manifest file 'guppy3.egg-info/SOURCES.txt'
writing manifest file 'guppy3.egg-info/SOURCES.txt'
running check
creating guppy3-3.0.6
creating guppy3-3.0.6/guppy
creating guppy3-3.0.6/guppy/etc
creating guppy3-3.0.6/guppy/gsl
creating guppy3-3.0.6/guppy/heapy
creating guppy3-3.0.6/guppy/heapy/test
creating guppy3-3.0.6/guppy/sets
creating guppy3-3.0.6/guppy3.egg-info
creating guppy3-3.0.6/src
creating guppy3-3.0.6/src/heapy
creating guppy3-3.0.6/src/sets
copying files to guppy3-3.0.6...
copying README.md -> guppy3-3.0.6
copying setup.py -> guppy3-3.0.6
copying guppy/__init__.py -> guppy3-3.0.6/guppy
copying guppy/etc/Cat.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/Code.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/Descriptor.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/FSA.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/Glue.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/Help.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/IterPermute.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/KanExtension.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/KnuthBendix.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/RE.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/RE_Rect.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/__init__.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/cmd.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/etc.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/textView.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/tkcursors.py -> guppy3-3.0.6/guppy/etc
copying guppy/etc/xterm.py -> guppy3-3.0.6/guppy/etc
copying guppy/gsl/Document.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/DottedTree.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/Exceptions.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/FileIO.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/Filer.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/Gsml.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/Help.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/Html.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/Latex.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/Main.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/SpecNodes.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/Tester.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/Text.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/XHTML.py -> guppy3-3.0.6/guppy/gsl
copying guppy/gsl/__init__.py -> guppy3-3.0.6/guppy/gsl
copying guppy/heapy/AbstractAlgebra.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Classifiers.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Console.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Doc.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/ImpSet.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Monitor.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/OutputHandling.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Part.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Path.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Prof.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/RM.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/RefPat.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Remote.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/RemoteConstants.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Spec.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Target.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/UniSet.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/Use.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/View.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/__init__.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/pbhelp.py -> guppy3-3.0.6/guppy/heapy
copying guppy/heapy/test/__init__.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/support.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_Classifiers.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_ER.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_OutputHandling.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_Part.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_Path.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_RefPat.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_RetaGraph.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_Spec.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_UniSet.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_View.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_all.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_dependencies.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_gsl.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_heapyc.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_menuleak.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/heapy/test/test_sf.py -> guppy3-3.0.6/guppy/heapy/test
copying guppy/sets/__init__.py -> guppy3-3.0.6/guppy/sets
copying guppy/sets/test.py -> guppy3-3.0.6/guppy/sets
copying guppy3.egg-info/PKG-INFO -> guppy3-3.0.6/guppy3.egg-info
copying guppy3.egg-info/SOURCES.txt -> guppy3-3.0.6/guppy3.egg-info
copying guppy3.egg-info/dependency_links.txt -> guppy3-3.0.6/guppy3.egg-info
copying guppy3.egg-info/top_level.txt -> guppy3-3.0.6/guppy3.egg-info
copying src/heapy/heapyc.c -> guppy3-3.0.6/src/heapy
copying src/heapy/stdtypes.c -> guppy3-3.0.6/src/heapy
copying src/sets/bitset.c -> guppy3-3.0.6/src/sets
copying src/sets/nodeset.c -> guppy3-3.0.6/src/sets
copying src/sets/sets.c -> guppy3-3.0.6/src/sets
Writing guppy3-3.0.6/setup.cfg
Creating tar archive
removing 'guppy3-3.0.6' (and everything under it)

A lot of files under src/ is missing.

Question: How to analyze guppy heap files

Thank you for porting guppy to the latest and greatest Py3. I am wondering how to analyze guppy heap files that has been dump to files with some UI/console tools?

rlcompleter on a heap is super expensive

Not sure if this is fixable on our side, but I should document this:

$ python -i -c 'hp = __import__("guppy").hpy()'
>>> h = hp.heap()
>>> h.[TAB][TAB][HANGS]

GDB:

gef➤  py-bt
Traceback (most recent call first):
  File "/home/zhuyifei1999/guppy3/guppy/heapy/Classifiers.py", line 620, in owners
    self.mod.hv.update_dictowners(drg)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/Classifiers.py", line 717, in owners
    return self.d.owners(X)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py", line 1692, in get_owners
    return self.mod.Use.Clodo.classifier.owners(a)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py", line 521, in <lambda>
    owners = property(lambda self: self.fam.get_owners(self), doc="""\
  <built-in method getattr of module object at remote 0x7f05681d4d10>
  File "/usr/lib/python3.7/rlcompleter.py", line 173, in attr_matches
    val = getattr(thisobject, word)
  File "/usr/lib/python3.7/rlcompleter.py", line 89, in complete
    self.matches = self.attr_matches(text)
gef➤  py-up
#11 Frame 0x559685b8fd90, for file /home/zhuyifei1999/guppy3/guppy/heapy/Classifiers.py, line 717, in owners (self=<ByClassOrDictOwner(mod=<Interface(_share=<Share(module=<module at remote 0x7f0567dead70>, parent=<Share(module=<module at remote 0x7f0567eda8f0>, parent=<Share(module=<module at remote 0x7f0568079f50>, parent=<Share(module=None, parent=<...>, name='', setable=(), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': None, 'guppy': <...>, 'types': <Share(module=<module at remote 0x7f0568092110>, parent=<...>, name='types', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7f0568092110>, 'ModuleType': <type at remote 0x7f05692bc620>, 'CodeType': <type at remote 0x7f05692c1180>, 'FrameType': <type at remote 0x7f05692bee00>}, owners={}, ispackage=None, recursion=0, preload=(...)) at remote 0x7f0567e45ed0>, 'io': <Share(module=<...(truncated)
    return self.d.owners(X)
gef➤  
#15 Frame 0x559685a99280, for file /home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py, line 1692, in get_owners (self=<IdentitySetFamily(mod=<Interface(_share=<Share(module=<module at remote 0x7f0567e700b0>, parent=<Share(module=<module at remote 0x7f0567eda8f0>, parent=<Share(module=<module at remote 0x7f0568079f50>, parent=<Share(module=None, parent=<...>, name='', setable=(), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': None, 'guppy': <...>, 'types': <Share(module=<module at remote 0x7f0568092110>, parent=<...>, name='types', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7f0568092110>, 'ModuleType': <type at remote 0x7f05692bc620>, 'CodeType': <type at remote 0x7f05692c1180>, 'FrameType': <type at remote 0x7f05692bee00>}, owners={}, ispackage=None, recursion=0, preload=(...)) at remote 0x7f0567e45ed0>, 'io': <Share(module=<m...(truncated)
    return self.mod.Use.Clodo.classifier.owners(a)
gef➤  
#19 Frame 0x559685b90bb0, for file /home/zhuyifei1999/guppy3/guppy/heapy/UniSet.py, line 521, in <lambda> (self=<IdentitySetMulti at remote 0x7f0567ee4230>)
    owners = property(lambda self: self.fam.get_owners(self), doc="""\
gef➤  
#29 <built-in method getattr of module object at remote 0x7f05681d4d10>
gef➤  
#31 Frame 0x559685d6e970, for file /usr/lib/python3.7/rlcompleter.py, line 173, in attr_matches (self=<Completer(use_main_ns=1, namespace={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x5596859dbe50>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f05681d4d10>, 'hp': <_GLUECLAMP_(_share=<Share(module=<module at remote 0x7f0567edaad0>, parent=<Share(module=<module at remote 0x7f0567eda8f0>, parent=<Share(module=<module at remote 0x7f0568079f50>, parent=<Share(module=None, parent=<...>, name='', setable=(), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': None, 'guppy': <...>, 'types': <Share(module=<module at remote 0x7f0568092110>, parent=<...>, name='types', setable=(...), chgable=(...), importedfrom={}, nowrap=(...), wrapattr=None, wrapping=0, data={'_parent': <...>, '_root': <...>, '_module': <module at remote 0x7f0568092110>, 'ModuleType': <type at remote 0x7...(truncated)
    val = getattr(thisobject, word)
gef➤  py-print thisobject
local 'thisobject' = <IdentitySetMulti at remote 0x7f0567ee4230>
gef➤  py-print word
local 'word' = 'owners'
gef➤  py-print words
local 'words' = {'__doc__', '__new__', '_doc_nodes', 'owners', '__reduce__', '__getattr__', '__or__', '__subclasshook__', '__dir__', '__bool__', '__xor__', '__str__', '__delattr__', '__mul__', '__getattribute__', '__len__', 'pathsout', '__ge__', '__eq__', '__rand__', '__rxor__', '_partition', 'size', '__call__', 'fam', '__le__', '__and__', '_help_url_', 'test_contains', 'referrers', 'er', '__lshift__', '__hash__', '__sub__', '__class__', '__iter__', '_hiding_tag_', '__module__', 'more', '_instahelp_', 'bysize', '__contains__', '__format__', '_er', '__sizeof__', 'get_examples', 'biper', 'get_rp', 'byid', '__lt__', 'bytype', 'by', 'brief', 'byunity', 'parts', 'stat', '_origin_', 'nodes', 'partition', 'disjoint', 'doc', '_get_help', 'byclodo', '__gt__', '__repr__', '_derive_origin_', 'dump', 'kind', 'rp', 'dictof', '__rsub__', '__getitem__', 'get_render', 'shpaths', 'count', 'sp', 'indisize', '__setattr__', '__slots__', 'maprox', 'get_shpaths', 'referents', 'byrcs', 'byidset', 'get_ckc', '__init_subclass__', '__ne__', 'pathsin'...(truncated)
gef➤  py-list
 168                for word in words:
 169                    if (word[:n] == attr and
 170                        not (noprefix and word[:n+1] == noprefix)):
 171                        match = "%s.%s" % (expr, word)
 172                        try:
>173                            val = getattr(thisobject, word)
 174                        except Exception:
 175                            pass  # Include even if attribute not set
 176                        else:
 177                            match = self._callable_postfix(val, match)
 178                        matches.append(match)

This is https://github.com/python/cpython/blob/77abf23c67c1a465a8899666c69f6bcd6930e003/Lib/rlcompleter.py#L173, it wants to know whether each of out attributes are callable.

Stack overflow on long linked list during traversal

This bug was first reported at:
https://sourceforge.net/p/guppy-pe/bugs/20/

Can we do something about it? It is probably a stack overflow. It would be better if we could check for stack size and report an exception instead of trigging a segmentation fault.

#!/usr/bin/env python

from guppy import hpy

class node(object):
    def __init__(self, value, _next=None):
        self.value = value
        self.next = _next

llist = node(0)
curnode = llist
for i in range(int(1e6)):
    curnode.next = node(0)
    curnode = curnode.next

heap = hpy().heap()

Segmentation fault on import

I'm trying to use Guppy but as soon as I import it, python crashes with a segfault:

root@4738adfde8d2:/usr/src/app# ipython
Python 3.7.1 (default, Nov 16 2018, 22:26:09)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from guppy import hpy
Segmentation fault

Linux 4738adfde8d2 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 GNU/Linux

Not sure what information I can provide to help?

Provide `guppy.version`

Fails to build on Python 3.11 RC2: fatal error: longintrepr.h: No such file or directory

Cannot get this package to build on Python 3.11:

Upstream: home-assistant/core#74650

Build output:

  Building wheel for guppy3 (setup.py): started
  Building wheel for guppy3 (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [96 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/guppy
      copying guppy/_version.py -> build/lib.linux-x86_64-cpython-311/guppy
      copying guppy/__init__.py -> build/lib.linux-x86_64-cpython-311/guppy
      creating build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/etc.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/xterm.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/IterPermute.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/Cat.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
  Running setup.py clean for guppy3
      copying guppy/etc/RE.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/KanExtension.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/textView.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/Glue.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/FSA.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/RE_Rect.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/Help.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/KnuthBendix.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/Code.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/Descriptor.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/cmd.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/__init__.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      copying guppy/etc/tkcursors.py -> build/lib.linux-x86_64-cpython-311/guppy/etc
      creating build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Exceptions.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Document.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Main.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Latex.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Html.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/SpecNodes.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/FileIO.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Filer.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Tester.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Help.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/DottedTree.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Text.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/XHTML.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/Gsml.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      copying guppy/gsl/__init__.py -> build/lib.linux-x86_64-cpython-311/guppy/gsl
      creating build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Part.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/pbhelp.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Classifiers.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/RefPat.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/UniSet.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Spec.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Doc.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Use.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Console.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Remote.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Monitor.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Prof.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/OutputHandling.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/RM.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/View.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Path.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/RemoteConstants.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/ImpSet.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/Target.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      copying guppy/heapy/__init__.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy
      creating build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_menuleak.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_Classifiers.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_all.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_View.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_OutputHandling.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_gsl.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_RetaGraph.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_heapyc.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_ER.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_Path.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_RefPat.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_Spec.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_dependencies.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_UniSet.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_sf.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/test_Part.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/support.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      copying guppy/heapy/test/__init__.py -> build/lib.linux-x86_64-cpython-311/guppy/heapy/test
      creating build/lib.linux-x86_64-cpython-311/guppy/sets
      copying guppy/sets/test.py -> build/lib.linux-x86_64-cpython-311/guppy/sets
      copying guppy/sets/__init__.py -> build/lib.linux-x86_64-cpython-311/guppy/sets
      running build_ext
      building 'guppy.sets.setsc' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/src
      creating build/temp.linux-x86_64-cpython-311/src/sets
      gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/runner/work/core/core/venv/include -I/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/include/python3.11 -c src/sets/bitset.c -o build/temp.linux-x86_64-cpython-311/src/sets/bitset.o
      src/sets/bitset.c:7:10: fatal error: longintrepr.h: No such file or directory
          7 | #include "longintrepr.h"
            |          ^~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

Ref: cython/cython#4461

Profile Browser fails with AttributeError: 'bool' object has no attribute '_root'

I'm doing:

from guppy import hpy
hpy().pb()

and I get:

Traceback (most recent call last):
File "", line 1, in
File "/home/rlippmann/git/homeassistant/core/venv/lib/python3.10/site-packages/guppy/heapy/Prof.py", line 2998, in pb
pa.new_profile_browser(filename)
File "/home/rlippmann/git/homeassistant/core/venv/lib/python3.10/site-packages/guppy/heapy/Prof.py", line 1670, in new_profile_browser
return ProfileBrowser(self, filename)
File "/home/rlippmann/git/homeassistant/core/venv/lib/python3.10/site-packages/guppy/heapy/Prof.py", line 2418, in init
self.display = Display(self.disptab,
File "/home/rlippmann/git/homeassistant/core/venv/lib/python3.10/site-packages/guppy/heapy/Prof.py", line 821, in init
self.var_xgrid = BooleanVar(xgrid)
File "/usr/lib/python3.10/tkinter/init.py", line 610, in init
Variable.init(self, master, value, name)
File "/usr/lib/python3.10/tkinter/init.py", line 372, in init
self._root = master._root()
AttributeError: 'bool' object has no attribute '_root'

I get a window to pop up, but I can't do anything.

Am I doing something wrong?

pywin32<300 causes NULL pointer deference during referrer graph generation

guppy3==3.1.0

Using Windows 7. I ran this:

h = guppy.hpy()
he = h.heap()

It waited for a few seconds and then the process crashed. Here are the error details from the dialog:

Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	python.exe
  Application Version:	3.8.1150.1013
  Application Timestamp:	5dfab277
  Fault Module Name:	python38.dll
  Fault Module Version:	3.8.1150.1013
  Fault Module Timestamp:	5dfab24b
  Exception Code:	c0000005
  Exception Offset:	000000000002feaf
  OS Version:	6.1.7601.2.1.0.256.1
  Locale ID:	1033
  Additional Information 1:	e8ad
  Additional Information 2:	e8adaaf2e9f8209565e4a118f3c1cb38
  Additional Information 3:	359a
  Additional Information 4:	359a43a120d45186052c5a12b0526d2d

Getting text output from tool

Hi! Thanks for this tool. I'm trying to figure out why some Django Gunicorn processes are so large.

I'd like to either log the heap or even possibly make the heap into a view in Django so I can see the heap -- but I can't figure out how to get a textual representation of the output instead of the interactive one.

Is this possible?

hpy() fails when "coverage" is loaded

hpy() raises an AttributeError when coverage is running code that uses hpy, because the CoverageDebugFile module has no dict attribute:

.tox/py37/lib/python3.7/site-packages/guppy/__init__.py:36: in hpy
    return r.guppy.heapy.Use
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:50: in __getattr__
    return self._share.getattr(self, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:218: in getattr
    d = self.getattr2(inter, cache, owner, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:235: in getattr2
    x = self.getattr_package(inter, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:283: in getattr_package
    x = self.makeModule(x, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:349: in makeModule
    return Share(module, self, module.__name__, Clamp)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:207: in __init__
    getattr(inter, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:50: in __getattr__
    return self._share.getattr(self, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:218: in getattr
    d = self.getattr2(inter, cache, owner, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:237: in getattr2
    x = self.getattr3(inter, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:322: in getattr3
    pa = getattr(pa, at)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:50: in __getattr__
    return self._share.getattr(self, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:218: in getattr
    d = self.getattr2(inter, cache, owner, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:235: in getattr2
    x = self.getattr_package(inter, name)
.tox/py37/lib/python3.7/site-packages/guppy/etc/Glue.py:272: in getattr_package
    x = __import__(self.makeName(name))
.tox/py37/lib/python3.7/site-packages/guppy/heapy/View.py:571: in <module>
    prime_builtin_types()
.tox/py37/lib/python3.7/site-packages/guppy/heapy/View.py:551: in prime_builtin_types
    for t in list(mod.__dict__.values()):
E   AttributeError: 'tuple' object has no attribute '__dict__'

The offending module is a tuple:

(<coverage.debug.DebugOutputFile object at 0x7fec9fb98668>, False)

which has no dict attribute.

Would you accept a PR to look for the dict attribute in View.py and skip anything that doesn't have that attribute?

Thanks

Feature: monitor external python process, possibly by injecting a stub?

Hello @zhuyifei1999,

I wanted to learn if I can use guppy to monitor an external process? My use case is that a python process is already running. I want to monitor the heap size over hours/days. It is a daemon, a long-running process, and based on how to use the guppy3, it seems I need to be inside that process to query the heap, this is not desirable for my use case.

Could you help to share your thoughts with me?
Thank you

[Question] How does `theone` return the Python object?

I am trying to understand the inner workings of guppy. I am curious how the following method in IdentitySetSingleton actually works to return the actual Python object:

    def _get_theone(self):
        return self._node

Looking through the C extension code in hv.c, I see in hv_heap() function that it calls hv_heap_rec() which in turn calls NyNodeSet_setobj(). In NyNodeSet_setobj(), Py_INCREF() is called to increase the reference count of each obj. But in the code, I don't see where obj is stored in a node?

Thank you for your explanation.

Is it possible to dump the memory snapshot for offline analysis?

Hi,

I got this question after checking out #24 .

In fact, I have some memory issues with my Django server, but the online environment is not quite reachable for some reason.

Therefore, I wonder whether guppy3 provides any approaches to dump the memory/heap snapshot so that I can retrieve the snapshot via an API and do the rest of the analysis offline.

Thanks.

Idea: Save the entire reference graph (to make profile browsers more useful)

Right now it just saves all the rows in the stats. That's someone one could use .all and just save the string.

guppy.heapy.test.test_gsl: FileNotFoundError (docexample.gsl)

Hi,

I pulled the last checkin after v3.0.6 from Master, installed it and tested it. Unfortunately, it seems to be something wrong with how the test skips the test in main() of SpecNodes.py.

There is a check for file not found in SpecNodes.py:

        if not root.os.path.exists(main_dt_name):
            print('%s does not exist, skipping test' % main_dt_name)

However, it seems to be missing a return statement there. I get:

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.6/dist-packages/guppy3-3.0.6-py3.6-linux-i686.egg/guppy/gsl/../../specs/docexample.gsl'

I put a return after the print above and then the test was skipped and it could continue with the other tests.

As a side note, would it be a bad idea to install the specs/*.gsl files?
I see that I am also not installing those files in Guppy-PE/0.1.11.
But maybe we could consider installing them?

Regards
Sverker

Usage with JAX

I would like to dive deeper into the memory consumption of a program using JAX's numpy. However, with guppy I am unfortunately not able to get out any information since the formatting of the result fails during stringifying and printing the result. One can easily reproduce the error using the following simple program:

from jax import numpy as jnp

from guppy import hpy

a = jnp.linspace(0.0, 1.0)

h = hpy()  # succeeds
print(h.heap())

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/guppy/heapy/UniSet.py", line 349, in __str__
    return self.fam.c_str(self)
  File "/usr/lib/python3.9/site-packages/guppy/heapy/UniSet.py", line 1616, in c_str
    return a.more._oh_printer.get_str_of_top()
  File "/usr/lib/python3.9/site-packages/guppy/heapy/UniSet.py", line 74, in __getattr__
    return self.fam.mod.View.enter(lambda: self.fam.c_getattr(self, other))
  File "/usr/lib/python3.9/site-packages/guppy/heapy/View.py", line 256, in enter
    retval = func()
  File "/usr/lib/python3.9/site-packages/guppy/heapy/UniSet.py", line 74, in <lambda>
    return self.fam.mod.View.enter(lambda: self.fam.c_getattr(self, other))
  File "/usr/lib/python3.9/site-packages/guppy/heapy/UniSet.py", line 800, in c_getattr
    return self.c_getattr2(a, b)
  File "/usr/lib/python3.9/site-packages/guppy/heapy/UniSet.py", line 803, in c_getattr2
    raise AttributeError(b)
AttributeError: more

I know it might be quite a stretch to ask guppy to support custom libraries like JAX so I would already be happy about insights into how to resolve it myself or at least how to work-around it.

Dubious assignment to be a check in hv_cli_and.c

This assignment that will be a check is dubious. It is the same for the two cases. Suggest to fix and also have a test case that catches it, if possible. Does the code in nodetuple_richcompare work as intended? I don't understand it after these years.

hv_cli_and.c, lines 235-236

       vi = (Py_ssize_t)vt->ob_item[i];
       **wi = (Py_ssize_t)vt->ob_item[i];**

Missing files

sets directory is empty
Unable to import
from guppy.sets.setsc import BitSet # base bitset type
from guppy.sets.setsc import ImmBitSet # immutable bitset type
from guppy.sets.setsc import immbit # immutable bitset singleton constructor
from guppy.sets.setsc import immbitrange # immutable bitset range constructor
from guppy.sets.setsc import immbitset # immutable bitset constructor
from guppy.sets.setsc import MutBitSet # mutable bitset
from guppy.sets.setsc import NodeSet # base nodeset type
from guppy.sets.setsc import ImmNodeSet # immmutable nodeset type
from guppy.sets.setsc import MutNodeSet # mutable nodeset type

Feature: replace/patch an imported class at runtime

Suppose we have the following file structure：

files/
    a.py
    b.py
main.py

And each file contains codes like:

# a.py
class A():
    pass

# b.py
from .a import A
class B():
    self.some_attrs = A()

# main.py
from files import B

obj = B()

Is this possible to replace the imported class A with another class using guppy3? And then when execute obj = B(), the new class will be constructed and attached to self.some_attrs

CPython tracer / profiler hooks completely break tests

To reproduce, with just a dummy tracer:

$ python -c 'trace = lambda frame, event, arg: trace; __import__("sys").settrace(trace); __import__("guppy").hpy().test()'
[...]
imported: guppy.heapy.test.test_Classifiers
test_base_classes (guppy.heapy.test.test_Classifiers.BaseCase) ... GC hook object was referred to from somebody!
ok
test_fancy_list_args (guppy.heapy.test.test_Classifiers.BaseCase) ... GC hook object was referred to from somebody!
ok
test_fancy_type_conversions (guppy.heapy.test.test_Classifiers.BaseCase) ... GC hook object was referred to from somebody!
ok
test_invalid_operations (guppy.heapy.test.test_Classifiers.BaseCase) ... GC hook object was referred to from somebody!
ok
test_minmax (guppy.heapy.test.test_Classifiers.BaseCase) ... GC hook object was referred to from somebody!
ok

----------------------------------------------------------------------
Ran 5 tests in 0.182s

OK
test_classification (guppy.heapy.test.test_Classifiers.ClassificationCase) ... GC hook object was referred to from somebody!
ok
test_selection (guppy.heapy.test.test_Classifiers.ClassificationCase) ... GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
ok

----------------------------------------------------------------------
Ran 2 tests in 19.371s

OK
test_laws (guppy.heapy.test.test_Classifiers.LawsCase) ... GC hook object was referred to from somebody!
ok

----------------------------------------------------------------------
Ran 1 test in 0.872s

OK
test_rendering (guppy.heapy.test.test_Classifiers.RenderCase) ... GC hook object was referred to from somebody!
ok

----------------------------------------------------------------------
Ran 1 test in 0.190s

OK
test_1 (guppy.heapy.test.test_Classifiers.SpecialCases) ... GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
GC hook object was referred to from somebody!
ok
test_2 (guppy.heapy.test.test_Classifiers.SpecialCases) ... GC hook object was referred to from somebody!
ok
test_alt_retclaset (guppy.heapy.test.test_Classifiers.SpecialCases) ... GC hook object was referred to from somebody!
ok
test_dictowner (guppy.heapy.test.test_Classifiers.SpecialCases) ... GC hook object was referred to from somebody!
ok
test_retclaset (guppy.heapy.test.test_Classifiers.SpecialCases) ... GC hook object was referred to from somebody!
aseq: Expected: b =  8
Got actually  : a =  22
FAIL
test_via (guppy.heapy.test.test_Classifiers.SpecialCases) ... GC hook object was referred to from somebody!
FAIL

======================================================================
FAIL: test_retclaset (guppy.heapy.test.test_Classifiers.SpecialCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/test_Classifiers.py", line 347, in test_retclaset
    self.aseq(grc(C1), rcC1)        # (A)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/support.py", line 249, in aseq
    self.assertTrue(0)
AssertionError: 0 is not true

======================================================================
FAIL: test_via (guppy.heapy.test.test_Classifiers.SpecialCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/test_Classifiers.py", line 425, in test_via
    s.byvia.kind == hp.Via("_.f_locals['v']", "_[('k',)]", "_[('v',)]", '_.keys()[0]'))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 6 tests in 2.720s

FAILED (failures=2)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/zhuyifei1999/guppy3/guppy/heapy/Use.py", line 363, in test
    self._parent.test.test_all.test_main(debug)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/test_all.py", line 35, in test_main
    f(debug=debug)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/test_Classifiers.py", line 959, in test_main
    support.run_unittest(SpecialCases, debug)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/support.py", line 186, in run_unittest
    run_suite(suite, testclass)
  File "/home/zhuyifei1999/guppy3/guppy/heapy/test/support.py", line 176, in run_suite
    raise TestFailed(msg)
guppy.heapy.test.support.TestFailed: errors occurred in guppy.heapy.test.test_Classifiers.SpecialCases

The reference chains seem to contain unexpected values.

If the tracer / profiler is completely C then it works fine (try using profile and cProfile to profile the tests).

@svenil found that this is probably related to the use of PyFrame_FastToLocalsWithError in call_trampoline, and test.test_sys.SysModuleTest.test_refcount & test.support.refcount_test may explain the reason.

Remote monitor mode not available for python >3.8 ?

Hi, I'm using python 3.10 in one of my projects and wanted to use the remote monitoring mode from guppy3. However, after adding
import guppy.heapy.RM I get an error saying AttributeError: module 'guppy.heapy.heapyc' has no attribute 'interpreter'.

After some digging, I found that the interpreter is not available for python version above 3.8 (found here)

Is there any specific reason for this ? In the Readme, it's specified that the package works with python 3.9, 3.10 and 3.11.

Any help here would be appreciated :)