etianen / aiohttp-wsgi Goto Github PK
View Code? Open in Web Editor NEWWSGI adapter for aiohttp.
Home Page: https://aiohttp-wsgi.readthedocs.io
License: BSD 3-Clause "New" or "Revised" License
WSGI adapter for aiohttp.
Home Page: https://aiohttp-wsgi.readthedocs.io
License: BSD 3-Clause "New" or "Revised" License
We've upgraded to aiohttp>=3.8.0 recently in nixpkgs and I noticed this package now fails the following test:
___________________________ EnvironTest.testEnviron ____________________________
self = <tests.test_environ.EnvironTest testMethod=testEnviron>
def testEnviron(self) -> None:
with self.run_server(assert_environ) as client:
> client.assert_response(headers={
"Foo": "bar",
})
tests/test_environ.py:99:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/base.py:43: in assert_response
self._test_case.assertEqual(response.status, 200)
E AssertionError: 500 != 200
------------------------------ Captured log call -------------------------------
ERROR aiohttp.server:web_protocol.py:405 Error handling request
Traceback (most recent call last):
File "/nix/store/cpi0v7m6l2y47banahzihgpca47diryv-python3.9-aiohttp-3.8.1/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
resp = await request_handler(request)
File "/nix/store/cpi0v7m6l2y47banahzihgpca47diryv-python3.9-aiohttp-3.8.1/lib/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
resp = await handler(request)
File "/build/source/aiohttp_wsgi/wsgi.py", line 294, in handle_request
return await self._loop.run_in_executor(
File "/nix/store/nffa4bgyi43wcrw3g3z0cmwy9zzz528y-python3-3.9.6/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/build/source/aiohttp_wsgi/wsgi.py", line 147, in _run_application
body_iterable = application(environ, start_response)
File "/build/source/tests/test_environ.py", line 13, in do_environ_application
func(environ)
File "/build/source/tests/test_environ.py", line 23, in assert_environ
assert environ["CONTENT_TYPE"] == ""
AssertionError: assert 'application/octet-stream' == ''
+ application/octet-stream
aiohttp-wsgi
has a way to serve WSGI apps from the CLI, but there is no easy way to do it from Python code. By easy I mean with a single function call.
The serve
function in __main__
is almost good for this use case, except that it expects args from the CLI.
For inspiration, I really like how waitress does it with waitress.serve
.
I return an iterable for WSGI to stream to the client. The iterable can provide data for hours (radio streaming).
If the client disconnects before the iterable is done, I see this message printing on stderr: "socket.send() raised exception.". Worse, the iterable keeps iterating until is done, sucking CPU and RAM. Doing this several times I can bring down my server, just leaving broken connections behind me sucking resources.
Expected result: If a client disconects, the iterable I provided WSGI to send data to the client must be disposed, not called anymore and, eventually, garbage collected. If my iterables had a "close" method, call it before disposing the reference.
Code example showing this issue:
#!/usr/bin/env python3
import time
import aiohttp_wsgi
def application(environ, start_response) :
def iterable() :
while True :
time.sleep(1)
yield b'1234\r\n'
start_response('200 OK', [
('Content-Length', '9999999999'),
('Content-Type', 'text/plain'),
])
return iterable()
aiohttp_wsgi.serve(application, host="127.0.0.1", port=18333)
After making a request to port 18333 and force a disconnection, I will get a "socket.send() raised exception." message in the main window every second, showing that the iterator is still running instead of being disposed.
It would be great to have a version compatible with the most recent builds of aiohttp.
I am requesting this URL to my WSGI application: "http://127.0.0.1:8080/buffy/Test%20%C3%A1%20%C3%B3".
What I am getting in PATH_INFO is: "/buffy/Test á ó". This have two problems:
Hi,
I have Django project using python 3.7.0
Due to minimum example in the documentation, I'm trying to follow this docs and edit my wsgi like this.
import os
from aiohttp import web
from aiohttp_wsgi import WSGIHandler
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
application = get_wsgi_application()
wsgi_handler = WSGIHandler(application)
app = web.Application()
app.router.add_route('*', '/{path_info:.*}', wsgi_handler)
But when I run ./manage.py runserver
I got these error
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x106324158>
Traceback (most recent call last):
File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/utils/autoreload.py", line 225, in wrapper
fn(*args, **kwargs)
File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 137, in inner_run
handler = self.get_handler(*args, **options)
File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands/runserver.py", line 27, in get_handler
handler = super().get_handler(*args, **options)
File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 64, in get_handler
return get_internal_wsgi_application()
File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/core/servers/basehttp.py", line 45, in get_internal_wsgi_application
return import_string(app_path)
File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/utils/module_loading.py", line 17, in import_string
module = import_module(module_path)
File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/adiyatmubarak/Documents/Koding/Dewe/rumah-dongeng-pelangi-api/src/rumah_dongeng_pelangi/wsgi.py", line 20, in <module>
wsgi_handler = WSGIHandler(application)
File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/aiohttp_wsgi/wsgi.py", line 178, in __init__
self._loop = loop or asyncio.get_event_loop()
File "/Users/adiyatmubarak/.pyenv/versions/3.7.0/lib/python3.7/asyncio/events.py", line 644, in get_event_loop
% threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Dummy-1'.
To reproduce:
from concurrent.futures.process import ProcessPoolExecutor
from concurrent.futures.thread import ThreadPoolExecutor
from aiohttp import web
from aiohttp_wsgi import WSGIHandler
from flask import Flask
flaskapp = Flask(__name__)
@flaskapp.route('/')
def index():
return 'hello from flask'
executor = ProcessPoolExecutor() # ThreadPoolExecutor() works
aioapp = web.Application()
aioapp.router.add_route('*', '/{path_info:.*}', WSGIHandler(flaskapp, executor=executor))
if __name__ == '__main__':
web.run_app(aioapp)
Traceback:
======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)
Error handling request
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "F:\envs\_april\lib\multiprocessing\queues.py", line 236, in _feed
obj = _ForkingPickler.dumps(obj)
File "F:\envs\_april\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "F:\envs\_april\lib\site-packages\aiohttp\web_protocol.py", line 418, in start
resp = await task
File "F:\envs\_april\lib\site-packages\aiohttp\web_app.py", line 458, in _handle
resp = await handler(request)
File "F:\envs\_april\lib\site-packages\aiohttp\web_urldispatcher.py", line 158, in handler_wrapper
return await result
File "F:\envs\_april\lib\site-packages\aiohttp_wsgi\wsgi.py", line 266, in handle_request
environ,
File "F:\envs\_april\lib\multiprocessing\queues.py", line 236, in _feed
obj = _ForkingPickler.dumps(obj)
File "F:\envs\_april\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects
The WSGI adapter reads the complete request body into memory. This results in extreme memory consumption for large file uploads and makes it very easy to run denial-of-service attacks.
curl --data-binary "@/dev/zero" http://example.com/wsgi
It seems that aiohttp-wsgi
just runs an application in an executor, which by default is just a thread pool, with a default pool size of 5 threads.
So if the application is designed with blocking APIs, such as flask and django, it will actually be executed via multi-threading and could not take advantage of co-routines. Then what's the benefit for using aiohttp-wsgi
?
aiohttp-wsgi
shouldn't touch the logging configuration or at least provide an option to instruct it not to.
Here is a real life scenario where the setup done by aiohttp-wsgi
is problematic: a Django user has a custom configuration for the aiohttp
logger:
# in Django settings.py
LOGGING = {
"loggers": {
"aiohttp": {
"level": "ERROR"
}
}
}
What happens is:
aiohttp-wsgi
logging.basicConfig
does nothing because logging is already configuredaiohttp
logger level gets overridden by the level ofaiohttp-wsgi
It may not sound too bad, but it's tiresome when a user has to track down all third-party libraries doing such things, thus I recommend libraries authors to avoid touching logging
and let the user configure this the way he wants in a central place.
Just wondering if there are any caveats/instructions to deployment? (eg. in my case, I'm hoping to set up a quick development server with Heroku and wondering if I can use this wrapper with waitress). I'm assuming no, since aiohttp has it's own http server...but I know there is support for aiohttp with Gunicorn...
Let me know if this makes sense. I'm sure I may not be making the most sense as I'm just getting started with asyncio...and trying to make it work with Django...
what is wrong with my code
import asyncio
from aiohttp_wsgi import wsgi
from aiohttp import web
async def http_server_handler(request):
headers = {}
headers['Content-Type'] = 'text/html'
response = web.StreamResponse(
status=200,
reason='OK',
headers=headers,
)
await response.prepare(request)#没有这行 RuntimeError: Cannot call write() before prepare()
data=b'2333333'
await response.write(data) #AssertionError: data argument must be byte-ish
await response.write_eof()
return response
app = web.Application()
app.router.add_route('*', '/', http_server_handler)
# web.run_app(app, host='0.0.0.0', port=8080)
# loop=asyncio.get_event_loop()
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(4) # > 1
application=wsgi.WSGIHandler(
app,
loop=None,
executor=executor,
)
if __name__ == '__main__':
from gevent.pywsgi import WSGIServer
http_server = WSGIServer(('0.0.0.0', 8080), application)
http_server.serve_forever()
got error
>python3 wsgi.py
Traceback (most recent call last):
File "C:\QGB\Anaconda3\lib\site-packages\gevent\pywsgi.py", line 999, in handle_one_response
self.run_application()
File "C:\QGB\Anaconda3\lib\site-packages\gevent\pywsgi.py", line 945, in run_application
self.result = self.application(self.environ, self.start_response)
TypeError: handle_request() takes 2 positional arguments but 3 were given
2021-05-11T10:52:03Z {'REMOTE_ADDR': '192.168.43.162', 'REMOTE_PORT': '54957', 'HTTP_HOST': '192.168.43.162:8080', (hidden keys: 25)} failed with TypeError
Is it possible to run this under multiple processes using a ProcessPoolExecutor
to allow scaling past a single process? I have tried to replace the default executor with the ProcessPoolExecutor
but get some pickling errors.
If not, is there a way to expose the whole app to apache/nginx/gunicorn?
This looks interesting but I have no idea how to implement it into a project.
Is there any specific reason aiohttp-wsgi is pinned to aiohttp<3
(in setup.py) @etianen ?
I do not see any dependency information. Is python3.4 a requisite for this because aiohttp requires it? And if so, can I use this with an app that was written in python2.7?
Hi Dave,
if I remember correctly you wrote the article on why to use Waitress on Heroku. Now since you came up with this server, how does it do on heroku? Can you recommend using it?
I'd love to hear your thoughts on this.
Cheers
-Joe
I would like to use the new testing functionality in aiohttp 0.22. But the setup.py for aiohttp-wsgi disallows it.
I ran the aiohttp-wsgi tests against aiohttp 0.22.2 and all tests passed. My application also runs correctly with aiohttp-wsgi and aiohttp 0.22.2.
What is the reason for not allowing aiohttp 0.22 in your setup.py. If you let me know, I will happily work on these issues.
File "/Users/antonogorodnikov/repo/partner/venv3/lib/python3.5/site-packages/django/utils/functional.py", line 33, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/Users/antonogorodnikov/repo/partner/venv3/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 121, in GET
raw_query_string = get_bytes_from_wsgi(self.environ, 'QUERY_STRING', '')
File "/Users/antonogorodnikov/repo/partner/venv3/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 241, in get_bytes_from_wsgi
return value.encode(ISO_8859_1) if six.PY3 else value
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 62-64: ordinal not in range(256)
The problem is here:
"QUERY_STRING": request.rel_url.query_string,
https://github.com/etianen/aiohttp-wsgi/blob/master/aiohttp_wsgi/wsgi.py#L230
The solution is:
"QUERY_STRING": request.rel_url.raw_query_string,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.