pythonspeed / profila Goto Github PK

Right now, it seems profiling is done at a practically infinite depth, so my calls usually end up going way deeper than is important to me (i.e. I don't need to see the deepest numpy source code). Is it possible to limit the depth to which the profiling is done, either by simply doing x levels away from the executed file or (possibly even better) only profiling calls in files contained in a certain directory/namespace?

Release requirements

• Tests
• Documentation

UX requirements

• Fail if loaded after Numba is imported
• Fail if gdb can't be found
• Fail if not on Linux
• After loading warn that it shouldn't be on by default for performance reasons

Implementation notes

On Linux prctl(PR_SET_PTRACER, pid_of_subprocess)/prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY) might allow gdb to access current process without needing root. prctl(PR_SET_PTRACER, 0) to reset.

Would probably need controller to be an additional Python process to prevent deadlocks.

The controlling Python subprocess would need to communicate with the parent process:

1. Command-line args can tell subprocess where to attach.
2. Subprocess needs to know when to stop profiling (via stdin closing?).
3. Subprocess needs to send results to Jupyter (via stdout?).
4. Subprocess needs to tell Jupyter when it's ready so Jupyter can proceed with code execution.

Profiling-wise, this is probably straightforward, should be able to do the equivalent of apply thread all bt in GDB MI and get samples across multiple threads.

Whether the line annotation visualization actually makes sense for multiple threads is something I have to think about.

UPDATE: This may take a while...

The problem

1. Profila currently uses GDB to get stacktraces, using the Machine Interface ("MI") protocol.
2. GDB doesn't work on macOS ARM, only lldb currently works.
3. lldb-mi is third-party tool supposedly compatible with GDB MI, but not readily available on macOS. E.g. VS Code plugins will ship it themselves.

Potential solutions

1. Switch to non-debugger method of getting stack traces; need something that relies on debuginfo in the way debuggers do. Initial experiments have failed, this is why there hasn't been profiling support for Numba until now! But will keep trying.
2. Ship lldb-mi in the wheel.
3. Convince some kind soul to package lldb-mi for Homebrew.
4. GDB adds support for macOS ARM.
5. Figure out a different way than MI to control lldb. The remote server protocol (same as GDB?) might work. Or there's https://pypi.org/project/hilda/ which either might be usable or might be a reasonable example. https://lib.rs/crates/lldb is another potential solution.

Workaround

On macOS, use Docker, Podman, or a VM to get a Linux environment.

I installed this onto a venv ran into this error:

{'type': 'result', 'message': 'error', 'payload': {'msg': '"/home/jje/.pyenv/shims/python": not in executable format: file format not recognized'}, 'token': None}

But /home/jje/.pyenv/shims/python is executable, so I'm not sure what the issue is.

ls /home/jje/.pyenv/shims/python -l
# -rwxr-xr-x 1 jje jje 176 Sep  8 22:12 /home/jje/.pyenv/shims/python*

Does it need to be a proper binary?

cat /home/jje/.pyenv/shims/python
# #!/usr/bin/env bash
# set -e
# [ -n "$PYENV_DEBUG" ] && set -x
# 
# program="${0##*/}"
# 
# export PYENV_ROOT="/home/jje/.pyenv"
# exec "/usr/share/pyenv/libexec/pyenv" exec "$program" "$@"

But it's not the executable I wanted anyway. I wanted the one from the venv I was calling python -m profilia in. Changing this line got it working the way I needed in my particular case:

-    process.stdin.write(b"-file-exec-file python\n")
+    process.stdin.write(bytes(f"-file-exec-file {sys.executable}\n", encoding="utf-8"))