StreamReader.iter_chunked(n) yields chunk_sizes < n before end of stream about aiohttp HOT 5 OPEN

Not sure if this behavior is intended but it was not indicated by the documentation or the source code.

It says 'with maximum size limit', which atleast to me, suggests that chunks may be smaller. But, feel free to make a PR to make it clearer.

I read that as a deliberate design. To have chunks of a fixed size would require not sending data that is already available, which could cause unnecessary delays.

from aiohttp.

Got it, thanks! So in order to get precise chunks of size N, is it recommended to use readexactly(N)? This was several orders of magnitude slower for my use case. I just ended up writing to a tempfile and reading out exactly sized chunks.

I think it's important to address these since the chunked reading feature is a widely used part of requests and a lot of people expect exact analogues in aiohttp.

I can make a PR updating docs. Let me know if there's any other relevant information/context.

from aiohttp.

I guess readexactly() works, or just adding chunks together yourself, maybe something like:

for new_chunk in ...:
    chunk += new_chunk
    if len(chunk) >= n:
        process(chunk[:n])
        chunk = chunk[n:]

I guess it's possible we could add an option if there's some solid use cases for it, but I'm not clear what those are yet. I think the main use case for the chunk size is to limit the amount of memory the application uses (e.g. downloading a large file and writing it to disk, on a tiny embedded system you may want to limit the amount of memory used for this to 1KB or something, while a desktop application might be good with 10MB as a limit).

from aiohttp.

My use case was for blowfish decryption, which was done on a chunk by chunk basis. This behavior was an issue because for any chunk with length n < chunk_size

# from iteration i   from iteration i+1
decrypt(chunk[:n]) + decrypt(chunk[n:])

is not the same as

# from iteration i
decrypt(chunk)

which is what I want. I don't know any other use cases off the top of my head but I feel as though reading constant size blocks from a stream should be common enough.

from aiohttp.

This feels to me like an itertools job.
Something like batched(chain.from_iterable(resp.content.iter_chunked(n)), n) should do it.

Looks like there are async versions of itertools at https://asyncstdlib.readthedocs.io/en/stable/source/api/itertools.html and https://aioitertools.omnilib.dev/en/stable/api.html (the former doesn't look like it has batched() yet, and the latter appears to have named it chunked()).

from aiohttp.