totoro2205 / aioreq Goto Github PK

Aioreq is a Python low-level asynchronous HTTP client library. It is built on top of TCP sockets and implements the HTTP protocol entirely on his own.

Click here

$ pip install aioreq

>>> import aioreq
>>> import asyncio
>>>
>>> cl = aioreq.Client()
>>>
>>> resp = asyncio.run(
...	cl.get('https://www.google.com')
...	)
>>> resp
<Response 200 OK>
>>> resp.status
200
>>> resp.status_message
'OK'
>>> resp.request
<Request GET https://www.google.com>
>>> headers = resp.headers # dict
>>> body = resp.content # bytes object

Alternatively, the best practice is to use a Context manager.

>>> import aioreq
>>> import asyncio
>>>
>>> async def main():
...     async with aioreq.Client() as cl:
...         return await cl.get('https://google.com')
>>> asyncio.run(main())
<Response 200 OK>

This code will asynchronously send 100 get requests to google.com, which is much faster than synchronous libraries.

>>> import asyncio
>>> import aioreq
>>>
>>> async def main():
...     async with aioreq.http.Client() as cl:
...         tasks = []
...         for j in range(100):
...             tasks.append(
...                 asyncio.create_task(
...                     cl.get('https://www.google.com/', )
...                 )
...             )
...         await asyncio.gather(*tasks)
>>> asyncio.run(main())

We occasionally use the HTTP protocol to download videos, photos, and possibly files. When downloading very large files, Stream must be used instead of the default Client. When a client downloads videos or files, the server responds with all information including headers, status code, status message, and full body, which can be very large. As a result, we cannot store it in RAM. Stream only returns a portion of the body per iteration, allowing us to write it to disk, then receive another portion and solve the ram overflow problem.

There is some fundamental Stream usage.

>>> import aioreq
>>> import asyncio
>>> 
>>> async def main():
...        async with aioreq.StreamClient() as cl:
...                # This code iterates through the message and yields each received chunk separately.
...                async for chunk in cl.get('https://google.com'):
...                        ...
>>> asyncio.run(main())

Aioreq is a very fast library, and we compared it to other Python libraries to demonstrate its speed.

I used these libraries to compare speed.

• httpx
• requests

First, clone aioreq repository.

Then...

$ cd aioreq
$ python -m venv venv
$ source ./venv/bin/activate
$ pip install '.[benchmarks]'
$ cd benchmarks
$ python run_tests_functions.py

This is the average execution time of each library for 200 asynchronous requests where responses was received without chunked transfer encoding.

Benchmark settings.

• Url - https://www.github.com
• Requests count - 200 for async and 5 for sync libs

$ cd becnhmarks
$ python run_tests_functions.py
========================
Benchmark settings
        Async lib test requests count : 200
        Sync lib test requests count  : 5
=======================
Function test for aioreq completed. Total time: 1.2442591340004583
Received statuses
        {301: 200}
Function test for requests completed. Total time: 1.6835168350007734
Received statuses
        {200: 5}
Function test for httpx completed. Total time: 1.691718664000291
Received statuses
        {301: 200}

This is the average execution time of each library for 100 asynchronous requests where responses was received with chunked transfer encoding.

Benchmark settings.

• Url - https://www.youtube.com
• Requests count - 100 for async and 5 for sync libs

$ cd benchmarks
$ python run_tests_functions.py
========================
Benchmark settings
        Async lib test requests count : 100
        Sync lib test requests count  : 5
=======================
Function test for aioreq completed. Total time: 3.837283965000097
Received statuses
        {200: 100}
Function test for requests completed. Total time: 6.098562907998712
Received statuses
        {200: 5}
Function test for httpx completed. Total time: 6.467480723000335
Received statuses
        {200: 100}

As you can see, the synchronous code lags far behind when we make many requests at the same time.

If the SSLKEYLOGFILE environment variable is set, Aioreq will write keylogs to it.

$ export SSLKEYLOGFILE=logs

Then just run python script.

$ python aioreq_app.py
$ ls -l
total 8
-rw-r--r-- 1 user user  94 Dec  5 17:19 aioreq_app.py
-rw-r--r-- 1 user user 406 Dec  5 17:19 logs

Now, the 'logs' file contains keylogs that can be used to decrypt your TLS/SSL traffic with a tool such as 'wireshark'.

Aioreq support basic features to work with HTTP/1.1.
More functionality will be avaliable in future realeases.
This is the latest version features.

• Keep-Alive (Persistent Connections)
• Automatic accepting and decoding responses. Using Accept-Encoding header
• HTTPS support, TLS/SSL Verification using certifi library
• Request Timeouts