1a2e438 can traverse asyncio.shield, i.e. show what's behind the shield that's blocking a given task.

This ticket is to explicitly show shield as an element in the task traceback or as a separate graph node.

When a task is (logically) blocked on a Future, it's actually blocked on a FutureIterator.

A simple Future may block N tasks, through N iterators.

We should figure out how to get from FutureIterator to the Future itself, so that, if 2 tasks are waiting on the very same thing, it's reflected in the graph.

Current state:

Task 1 → FutureIterator → ?
Task 2 → FutureIterator → ?

Desired state:

Task 1 → FutureIterator ┱→ Future
Task 2 → FutureIterator ┛

I'm thinking of integrating this repository with CI (Travis, Jenkins, CircleCI, etc) and readthedocs. For each commit to master, we will run the tests and update the documentation with the latest generated svg files.

stack should be a -> b -> sleep(...) -> sentinel

but we're only getting a -> sentinel

[red herring] somehow, _what reports that stack pointer is null?

stack stops on AttributeError: '_asyncio.FutureIter' object has no attribute 'cr_frame'

🤷‍♂

A helper function that dumps task state on a UNIX signal.
My fav. signal is WINCH.

It's probably easier to develop like this.

For comparison current approach is to spawn a dummy task whose purpose is only to report useful tasks's state:

What concerns me

• doesn't compile for Py 3.10 (frame struct was changed), fixable
• no users (via cursory github search)

Option 1

Work with cpython core devs to decide if it's a good idea, and if so, integrate the core feature (suspended task coro stack traversal) into cpyhton itself

Option 2

Abandon ship, give up

Option 3

Your idea, please?

Show remaining time in asyncio.sleep (when some task is blocked on it).
Maybe show original sleep time too (?)

I'm using python 3.7.6 (AMD64) on MacOS, I used version (baa4c73)

I took the examples/test_signal.py changed it to a while loop, and changed the signal, see below:

--- a/examples/test_signal.py
+++ b/examples/test_signal.py
@@ -12,10 +12,10 @@ async def job():


 async def foo():
-    signal.alarm(1)
-    await asyncio.sleep(1)
+    while True:
+        await asyncio.sleep(1)


 if __name__ == "__main__":
-    awaitwhat.helpers.register_signal(signal.SIGALRM)
+    awaitwhat.helpers.register_signal(signal.SIGINFO)
     asyncio.run(main())

When I'm sending SIGINFO to the process, the output is the following image:

Is this intended? How should I interpret this image?

asyncio.wait_for waits for something with a timeout:
https://docs.python.org/3/library/asyncio-task.html#timeouts

we should traverse wait_for (i.e. show awaitable behind it), and report remaining timeout.

see test_wait.py

• coroutines bba748c
• Tasks asyncio.all_tasks
• current task 670b138
• Futures #3
• Events
• asyncio.gather 74d5e76..606a965
• asyncio.wait #6
• asyncio.wait_for #5
• asyncio.shield 1a2e438
• asyncio.sleep #7
• network sockets #8
• what else is missing? read the docs

    Hi group,

    I'm recently debugging a long-running asyncio program that appears to get stuck about once a week.

    The tools I've discovered so far are:

        high level: asyncio.all_tasks() + asyncio.Task.get_stack()
        low level: loop._selector._fd_to_key

    What's missing is the middle level, i.e. stack-like linkage of what is waiting for what. For a practical example, consider:

    async def leaf(): await somesocket.recv()
    async def baz(): await leaf()
    async def bar(): await baz()
    async def foo(): await bar()
    async def job(): await foo()
    async def work(): await asyncio.gather(..., job())
    async def main(): asyncio.run(work())

    The task stack will contain:

        main and body of work with line number
        job task with line number pointing to foo

    The file descriptor mapping, socket fd, loop._recv() and a Future.

    What's missing are connections foo->bar->baz->leaf. That is, I can't tell which task is waiting for what terminal Future.

    Is this problem solved in some way that I'm not aware of? Is there a library or external tool for this already?

    Perhaps, if I could get a list of all pending coroutines, I could figure out what's wrong.

    If no such API exists, I'm thinking of the following:

    async def foo():
        await bar()

    In [37]: dis.dis(foo)
      1           0 LOAD_GLOBAL              0 (bar)
                  2 CALL_FUNCTION            0
                  4 GET_AWAITABLE
                  6 LOAD_CONST               0 (None)
                  8 YIELD_FROM
                 10 POP_TOP
                 12 LOAD_CONST               0 (None)
                 14 RETURN_VALUE

    Starting from a pending task, I'd get it's coroutine and:

    Get the coroutine frame, and if current instruction is YIELD_FROM, then the reference to the awaitable should be on the top of the stack. If that reference points to a pending coroutine, I'd add that to the "forward trace" and repeat.

    At some point I'd reach an awaitable that's not a pending coroutine, which may be: another Task (I already got those), a low-level Future (can be looked up in event loop), an Event (tough luck, shoulda logged all Event's on creation) or a dozen other corner cases.

    What do y'all think of this approach?

    Thanks, D.

It appears that the initial issue you mentioned, experiencing a stuck process once a week, could be related to disk I/O problems. In my experience, regular disk checks, like the ones my NAS performs on Saturdays, can cause significant delays and even crashes if there are processes running that heavily rely on disk I/O. This issue is also compounded by high user activity on the disks.

Regarding your async implementation, it sounds like you might benefit from considering threading if you need a higher level of parallelism. However, if you are tracking your tasks in a linear manner, creating a sequential flow of events through your async functions using await statements can work effectively. By properly ordering the dependencies with await, you ensure the tasks execute in the desired sequence.

In situations where you have a large number of tasks that can be completed mostly in parallel but have blocking dependencies, it's crucial to order those dependencies correctly using await statements, making sure the conditions are within the scope of your functions.

To troubleshoot potential bottlenecks and optimize your async program, consider incorporating clever insights for creating await points. Techniques such as using decorators to log await points, setting up breakpoints with a debugger, or implementing custom tracing functions can help identify and analyze the execution flow of your tasks.

By gaining a deeper understanding of the await points and their relationships, you can effectively troubleshoot and resolve any performance issues, including potential disk-related delays.

I'll be happy to share insights over a beer.

edit:
was curious if the readme was up to date and this got fixed

Currently the task dependency graph is generated in one go.

It would be awesome to split this in 2 stages:

1. generate plain data (nodes, edges, text)
2. convert plain data into a dot graph

When that's done, it becomes possible to:

• write tests over 1. :)
• output 1. as JSON
• output 1. as plain text, like Python traceback

guts:

What's required to build awaitwhat in a python:alpine container:

• apk update
• apk upgrade (dunno)
• apk add build-base (to get gcc)
• apk add libffi-dev (I guess I get it?)
• apk add openssl-dev (why?)

Event loop maintains a set of sockets that are being waited for with callbacks.
It should be possible to figure out which task is waiting for which socket.
Sockets could be reported with their type, state, direction (e.g. TCP, connected to 1.2.3.4:8080, read).

Today, awaitwhat code runs in-process, in-interpreter; the dependency must be added in advance.
Linux has tools to instrument other (already running) processes: bpftrace and oprofile

Links: https://github.com/iovisor/bcc
What's proper: usdt/uprobe/uretprobe?
Example: https://github.com/iovisor/bcc/blob/master/examples/cpp/RecordMySQLQuery.cc
More links: http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html

https://stackoverflow.com/questions/13103584/graphviz-how-do-i-make-the-text-in-labels-left-aligned

today, it looks like this:

Text text yada yada blah blah
  text blah yada tex
Text text yada blah blah yada
                       text

Apparently \l works a little different from e.g. \n;
Perhaps a trailing \l would solve the problem...

asyncio.wait is similar to asyncio.gather but supports more functionality:

return_when (operation mode) can be "first completed" "first exception" or "all completed".

we should report the operation mode and what's still pending behind wait.