Question about sharing data beween async and non ansync tasks about micropython-async HOT 7 CLOSED

Hi, thanks for your explanation. Now I understand when to use a lock or not.
Best regards!

from micropython-async.

I can see one obvious problem.

    def sync_get(self, keys=None):
        print(f"sync_get {keys}")
        return asyncio.run(self.get(keys))    

The function asyncio.run should be used once only to start asyncio. I think the method should read

    def sync_get(self, keys=None):
        print(f"sync_get {keys}")
        return await(self.get(keys))    

In general it's a lot to ask for someone to debug a substantial body of code. If you think you've found a bug you should systematically reduce your code until you have a minimal "reproducer" that a maintainer can run.

from micropython-async.

Hi,

i can't change it this way because if i want to use the await the function needs to be an async function. But i want to be able to call it from a non async context. Is this even possible?

Do you have any suggestion on how to update data in a dictionary from async and non async tasks? My idea was to use asyncio.lock.

I have stripped down my testcode but so far i could not figure out why or where exactly it crashes, but i quess you are correct that it happens because asyncio.run is run multiple times.

Thanks & best regards

from micropython-async.

i can't change it this way because if i want to use the await the function needs to be an async function.

Good point.

In general I'm puzzled by your code. There doesn't seem to be a clear distinction between synchronous and asynchronous code. For example the following is declared as async but is in fact synchronous - it never yields to the scheduler.

    async def _get(self, keys=None):
        if keys is None:
            return self.data.copy()
        d = self.data
        for key in keys:
            d = d[key]
        return d

This function is inherently synchronous. It could be made asynchronous by adding await asyncio.sleep(0) statements but there would be no practical benefit. This is because I would expect it to execute very quickly - probably in a very few ms or less. There is no practical gain in yielding to the scheduler unless a routine is likely to block for high tens of ms or longer.

The only point in yielding is to enable other tasks to run. Say you had a routine that, implemented as synchronous code, might block for (say) one second. You might well decide that having other tasks stall for 1s is unacceptable. So you change the routine to an async one, splitting the task into ten segments, with await asyncio.sleep(0) after each segment. Then the blocking time is reduced to 100ms.

My approach would be to review the code in this light, writing as async only those routines that threaten to harm the responsiveness of other async tasks.

To answer your question on running an async task from a synchronous function, you are correct in saying this is impossible. You can launch it:

def foo():  # Synchronous
    task = asyncio.create_task(bar())
    print('got here')

but the print statement is executed immediately. You wouldn't want the synchronous function to block while bar() was running because a blocking synchronous function locks up the scheduler.

In such cases there are various ways to check whether bar() has completed. It might run a callback or trigger an Event for example.

from micropython-async.

To add to the above, I suspect that you are unclear about the way asyncio works. Consider this function, which reads 1 million bytes from a binary file and returns their sum.

def add(file_path):
    with open(file_path, 'rb') as f:
        result = 0
        for _ in range(1_000_000):
            result += int(f.read(1))
    return result

The execution time is heavily dependent on the hardware hosting the file, but let's assume it takes 5 seconds. Let's also assume that you want other tasks to run while this is in progress. A naive user might do this:

async def add(file_path):
    with open(file_path, 'rb') as f:
        result = 0
        for _ in range(1_000_000):
            result += int(f.read(1))
    return result

with another task calling this as follows:

async def foo():
    total = await add("my_file")

This would run, but the scheduler would lock up for 5s. No other work would be accomplished while the file read was in progress. async def simply tells the compiler that the coroutine might include await statements. As written, there are none, so the routine would block until it completed. It is still a synchronous function with the annoying feature that it can't be used with a simple function call.

To actually achieve concurrency you need to do something like:

async def add(file_path):
    with open(file_path, 'rb') as f:
        result = 0
        for _ in range(100):
            for _ in range(10_000):
                result += int(f.read(1))
            await asyncio.sleep(0)  # Other tasks get some runtime here
    return result

This is now a true coroutine. Other tasks get some execution time about every 50ms (5s/100). The calling task

async def foo():
    total = await add("my_file")

sees a 5s pause before it gets the total - this is inevitable - but other tasks continue to run.

I hope this provides some clarity.

To add to this, a dictionary access is so fast that it isn't worth attempting to implement as asynchronous code.

from micropython-async.

Hi,

thank you for your detailed explanations, that helped a lot of better understanding asyncio.
I used it already to split up longer running tasks with await asyncio.sleep(0) statements to improve responsiveness of my application.

In general I'm puzzled by your code. There doesn't seem to be a clear distinction between synchronous and asynchronous code. For example the following is declared as async but is in fact synchronous - it never yields to the scheduler.

    async def _get(self, keys=None):
         if keys is None:
            return self.data.copy()
      d = self.data
         for key in keys:
             d = d[key]
        return d

Yes you are correct, that makes no sense here. The dictionary access is really fast enough.

The reason why I made the async get() function in the first place is, because of the async with self.lock: My fear was that if two async tasks access the dictionary the same time it could create problems. But from what I have understood now is, this is not possible on ESP32 because everything runs only on one core and dictionary access is atomic. So I guess I could just use a regular dict to share data between different running async tasks? Is my assumption correct?

from micropython-async.

Asyncio is a cooperative scheduler. A task only gets to run when another has yielded control to the scheduler. This means that synchronous code is never pre-empted. This includes synchronous code running as part of a coroutine. This is a major benefit in that locks are not required to protect synchronous code - see this tutorial section.

If multiple tasks are accessing a dict nothing will break (even if tasks are updating the dict). A lock might be required if it's important that several asynchronous updates must be read as a coherent set.

async def dict_update(d, lock):
    while True:
        async with lock:
            d['x'] = await read_sensor_x()   # Function yields to scheduler while waiting for a device
            d['y'] = await read_sensor_y()  # ditto
        await asyncio.sleep(1)

A task using those dict keys will get a consistent pair of values if it protects the accesses with the lock. Without protection it might get a new x value with a prior y value. Whether this matters, and whether a lock is required, is application dependent.

from micropython-async.