Pass max_length parameter to ZLibDecompressor about aiohttp HOT 5 OPEN

At a glance, if I've understood that correctly, changing the parameter would just stop you from receiving the full response (as that method is only going to be called when new data is received). Isn't the problem more that it shouldn't be calling that method with too much data? As you are using the streaming API, I'd expect it to only receive a few KBs of data each time and call that decompress method with it each time. So, if you see high memory usage, I wonder if the issue is actually with the streaming API instead...

@mykola-mokhnach might have some ideas as they last worked on the compression code.

from aiohttp.

At a glance, if I've understood that correctly, changing the parameter would just stop you from receiving the full response (as that method is only going to be called when new data is received). Isn't the problem more that it shouldn't be calling that method with too much data? As you are using the streaming API, I'd expect it to only receive a few KBs of data each time and call that decompress method with it each time. So, if you see high memory usage, I wonder if the issue is actually with the streaming API instead...

@mykola-mokhnach might have some ideas as they last worked on the compression code.

For example 1Kb after ungzip will be about 1Mb or bigger. Because the buffer is for incoming from the socket, not after ungzip.

from aiohttp.

Right, good point. So, if it's data with a very high compression ratio it could become a problem. I suspect this would still need some refactoring in order to get it to return the full data.

i.e. Instead of something like:

for chunk in data_received:
    yield decompress(chunk)

we'd need something more along the lines of:

for chunk in data_received:
    while chunk:
        result = decompress(chunk)
        yield result.data
        chunk = result.unconsumed_tail

from aiohttp.

Feel free to try and create a PR. I'd assume this would be fine changing the default behaviour, as the purpose of the streaming API is to limit memory usage, so this looks like an oversight to me.

Ideally, this would need to be implemented in the brotli decompressor too.

from aiohttp.

@Dreamsorcerer thank you! I'll work on it. I created issue in brotli. it doesn't have the option.

from aiohttp.