Giter Club home page Giter Club logo

uvloop's Introduction

image

PyPI - Downloads

uvloop is a fast, drop-in replacement of the built-in asyncio event loop. uvloop is implemented in Cython and uses libuv under the hood.

The project documentation can be found here. Please also check out the wiki.

Performance

uvloop makes asyncio 2-4x faster.

image

The above chart shows the performance of an echo server with different message sizes. The sockets benchmark uses loop.sock_recv() and loop.sock_sendall() methods; the streams benchmark uses asyncio high-level streams, created by the asyncio.start_server() function; and the protocol benchmark uses loop.create_server() with a simple echo protocol. Read more about uvloop in a blog post about it.

Installation

uvloop requires Python 3.8 or greater and is available on PyPI. Use pip to install it:

$ pip install uvloop

Note that it is highly recommended to upgrade pip before installing uvloop with:

$ pip install -U pip

Using uvloop

As of uvloop 0.18, the preferred way of using it is via the uvloop.run() helper function:

import uvloop

async def main():
    # Main entry-point.
    ...

uvloop.run(main())

uvloop.run() works by simply configuring asyncio.run() to use uvloop, passing all of the arguments to it, such as debug, e.g. uvloop.run(main(), debug=True).

With Python 3.11 and earlier the following alternative snippet can be used:

import asyncio
import sys

import uvloop

async def main():
    # Main entry-point.
    ...

if sys.version_info >= (3, 11):
    with asyncio.Runner(loop_factory=uvloop.new_event_loop) as runner:
        runner.run(main())
else:
    uvloop.install()
    asyncio.run(main())

Building From Source

To build uvloop, you'll need Python 3.8 or greater:

  1. Clone the repository:

    $ git clone --recursive [email protected]:MagicStack/uvloop.git
    $ cd uvloop
  2. Create a virtual environment and activate it:

    $ python3.7 -m venv uvloop-dev
    $ source uvloop-dev/bin/activate
  3. Install development dependencies:

    $ pip install -e .[dev]
  4. Build and run tests:

    $ make
    $ make test

License

uvloop is dual-licensed under MIT and Apache 2.0 licenses.

uvloop's People

Contributors

1st1 avatar asfaltboy avatar asvetlov avatar cclauss avatar ciscorn avatar claws avatar elprans avatar fantix avatar graingert avatar hikoz avatar hroncok avatar jackenmen avatar jensbjorgensen avatar jlaine avatar kaniini avatar lovasoa avatar mtorromeo avatar nothing4you avatar ofek avatar petriborg avatar pfreixes avatar pranavtbhat avatar ronindesign avatar shuuji3 avatar tardyp avatar tobotimus avatar versusvoid avatar vladima avatar vodik avatar yostealth avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

uvloop's Issues

uvloop + aiohttp: AttributeError: '_GatheringFuture' object has no attribute '_blocking'

  • uvloop version:
    uvloop (0.5.3)
    aiohttp (1.0.3)
  • python version:
    Python 3.5.2+
  • platform:
    Linux ubuntu 4.8.0-14-generic #15-Ubuntu SMP Tue Sep 20 22:02:02 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Code:

import asyncio

from aiohttp import web
import uvloop


asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())


async def handle(request):
    name = request.match_info.get('name', "Anonymous")
    text = "Hello, " + name
    return web.Response(text=text)

app = web.Application()
app.router.add_get('/', handle)
app.router.add_get('/{name}', handle)

web.run_app(app)

traceback:

Exception in callback Task._step
handle: <Handle Task._step>
Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run (uvloop/loop.c:38343)
  File "uvloop/task.pyx", line 159, in uvloop.loop.BaseTask._fast_step (uvloop/loop.c:99783)
AttributeError: '_GatheringFuture' object has no attribute '_blocking'

Asynchronous remove_writer and remove_reader.

  • e7bedb8:
  • Python 3.5.2:
  • Linux 4.8.6:

I notice that remove_reader and remove_writer are asynchronous with respect to when the file descriptor is actually removed from its epoll set (linux terminology). This is a byproduct of how libuv works.

This is different from the asyncio module and causes troubles with at least dup() based FDs. Namely serving static files with aiohttp under uvloop will induce a 100% CPU loop inside libuv because it can't remove the FD. Strace attached..

18:25:45.050832 open("/home/mayfield/project/aiocluster/aiocluster/diag/ui/node_modules/semantic-ui-css/semantic.css", O_RDONLY|O_CLOEXEC) = 24 <0.000031>
18:25:45.050911 ioctl(24, FIOCLEX)      = 0 <0.000018>
18:25:45.050970 fstat(24, {st_mode=S_IFREG|0775, st_size=752358, ...}) = 0 <0.000019>
18:25:45.051036 ioctl(24, TCGETS, 0x7ffd90f38cb0) = -1 ENOTTY (Inappropriate ioctl for device) <0.000019>
18:25:45.051104 lseek(24, 0, SEEK_CUR)  = 0 <0.000020>
18:25:45.051853 fcntl(23, F_DUPFD_CLOEXEC, 0) = 25 <0.000024>
18:25:45.051972 ioctl(25, FIONBIO, [0]) = 0 <0.000018>
18:25:45.052067 write(1, "\nCREATED", 8) = 8 <0.000039>
18:25:45.052225 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000023>
18:25:45.052336 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000022>
18:25:45.052443 write(1, " 25 <socket.socket fd=25, family"..., 159) = 159 <0.000026>
18:25:45.052521 ioctl(25, FIONBIO, [1]) = 0 <0.000017>
18:25:45.052603 sendto(25, "HTTP/1.1 200 OK\r\nContent-Type: t"..., 184, 0, NULL, 0) = 184 <0.000068>
18:25:45.052777 sendfile(25, 24, [0] => [143352], 752358) = 143352 <0.000083>
18:25:45.052970 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000019>
18:25:45.053048 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000017>
18:25:45.053131 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000024>
18:25:45.053228 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = 0 <0.000021>
18:25:45.053289 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f38db0) = 0 <0.000048>
18:25:45.053384 ioctl(25, FIONBIO, [1]) = 0 <0.000021>
18:25:45.053499 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000022>
18:25:45.053584 epoll_wait(6, [], 1024, 0) = 0 <0.000019>
18:25:45.053651 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 43) = 1 <0.015515>
18:25:45.069554 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000181>
18:25:45.070023 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000158>
18:25:45.070464 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000069>
18:25:45.070709 sendfile(25, 24, [143352] => [273672], 609006) = 130320 <0.000131>
18:25:45.071063 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000026>
18:25:45.071258 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000113>
18:25:45.071623 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000179>
18:25:45.071947 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000065>
18:25:45.072065 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f368d0) = 0 <0.000031>
18:25:45.072139 ioctl(25, FIONBIO, [1]) = 0 <0.000018>
18:25:45.072225 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000022>
18:25:45.072302 epoll_wait(6, [], 1024, 24) = 0 <0.024234>
18:25:45.097030 epoll_wait(6, [], 1024, 0) = 0 <0.000166>
18:25:45.097609 epoll_ctl(6, EPOLL_CTL_ADD, 15, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000145>
18:25:45.098014 epoll_ctl(6, EPOLL_CTL_DEL, 15, 0x7ffd90f39590) = 0 <0.000049>
18:25:45.098193 ioctl(15, FIONBIO, [1]) = 0 <0.000030>
18:25:45.098361 epoll_ctl(6, EPOLL_CTL_ADD, 15, {EPOLLIN, {u32=15, u64=15}}) = 0 <0.000085>
18:25:45.098540 epoll_wait(6, [], 1024, 0) = 0 <0.000018>
18:25:45.098630 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 99) = 1 <0.004728>
18:25:45.103551 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000026>
18:25:45.103711 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000022>
18:25:45.103850 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000039>
18:25:45.103979 sendfile(25, 24, [273672] => [403992], 478686) = 130320 <0.000060>
18:25:45.104165 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000018>
18:25:45.104260 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000019>
18:25:45.104367 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000026>
18:25:45.104473 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000017>
18:25:45.104541 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f368d0) = 0 <0.000023>
18:25:45.104601 ioctl(25, FIONBIO, [1]) = 0 <0.000016>
18:25:45.104678 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000022>
18:25:45.104769 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 93) = 1 <0.057713>
18:25:45.162769 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000030>
18:25:45.162926 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000026>
18:25:45.163065 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000036>
18:25:45.163203 sendfile(25, 24, [403992] => [534312], 348366) = 130320 <0.000060>
18:25:45.163391 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000017>
18:25:45.163484 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000017>
18:25:45.163583 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000025>
18:25:45.163711 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000021>
18:25:45.163792 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f368d0) = 0 <0.000019>
18:25:45.163850 ioctl(25, FIONBIO, [1]) = 0 <0.000016>
18:25:45.163927 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000020>
18:25:45.163989 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 33) = 1 <0.017369>
18:25:45.181732 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000050>
18:25:45.181960 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000026>
18:25:45.182252 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000049>
18:25:45.182423 sendfile(25, 24, [534312] => [664632], 218046) = 130320 <0.000133>
18:25:45.182797 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000027>
18:25:45.183088 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000164>
18:25:45.183530 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000143>
18:25:45.183813 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000022>
18:25:45.183929 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f368d0) = 0 <0.000167>
18:25:45.184256 ioctl(25, FIONBIO, [1]) = 0 <0.000048>
18:25:45.184526 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000142>
18:25:45.184805 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 13) = 1 <0.012150>
18:25:45.197446 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000051>
18:25:45.197749 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000114>
18:25:45.198159 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000166>
18:25:45.198559 sendfile(25, 24, [664632] => [752358], 87726) = 87726 <0.000114>
18:25:45.198925 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000020>
18:25:45.199062 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000026>
18:25:45.199239 write(1, "CLOSE 25 <socket.socket fd=25, f"..., 146) = 146 <0.000049>
18:25:45.199385 write(1, "\n", 1)       = 1 <0.000025>
18:25:45.199494 close(25)               = 0 <0.000021>
18:25:45.199595 close(24)               = 0 <0.000024>
18:25:45.200256 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2403, ...}) = 0 <0.000032>
18:25:45.200889 write(2, "[\33[34m2016-11-19 18:25:45,200\33[0"..., 333) = 333 <0.000031>
18:25:45.201266 epoll_wait(6, [], 1024, 0) = 0 <0.000036>
18:25:45.201492 epoll_ctl(6, EPOLL_CTL_ADD, 15, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000025>
18:25:45.201601 epoll_ctl(6, EPOLL_CTL_DEL, 15, 0x7ffd90f39590) = 0 <0.000036>
18:25:45.201713 ioctl(15, FIONBIO, [1]) = 0 <0.000026>
18:25:45.201844 epoll_ctl(6, EPOLL_CTL_ADD, 15, {EPOLLIN, {u32=15, u64=15}}) = 0 <0.000026>
18:25:45.201949 epoll_wait(6, [], 1024, 0) = 0 <0.000018>
18:25:45.202011 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 81) = 1 <0.008192>
18:25:45.210303 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f37060) = -1 EBADF (Bad file descriptor) <0.000101>
18:25:45.210532 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 72) = 1 <0.000178>
18:25:45.210950 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f37060) = -1 EBADF (Bad file descriptor) <0.000159>
18:25:45.211247 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 63) = 1 <0.000064>
18:25:45.211420 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f37060) = -1 EBADF (Bad file descriptor) <0.000045>
18:25:45.211531 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 54) = 1 <0.000048>
18:25:45.211620 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f37060) = -1 EBADF (Bad file descriptor) <0.000018>
18:25:45.211675 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 45) = 1 <0.000022>

I could go into more detail about this but it's rather complex and I believe there are a number of issues you could tie back to the fact that remove_* isn't syncronous as it is with the stdlib implementation of EventLoop.

taking uvloop into production?

This is not an issue but perhaps a request to maybe expand on documentation on how to use uvloop with other frameworks. The performance on it is very impressive, however, it's so low level, taking this into a production application would be quite a challenge.

What would be the best practice approach to doing this? Combine with tornado or any other async libary? Leverage other smaller libraries to handle things like user authentication and http parsing. Or does uvloop just need these features introduced directly into it to maintain the performance it currently has?

If I am using the wrong forum for asking this question I apologize and happy to move this discussion somewhere else.

Mismatch between port arguments to the asyncio and uvloop loop.create_server method

There appears to be a discrepancy in valid port arguments to the event loop's create_server method. With asyncio I can use 0 and None to obtain a server bound to an ephemeral port. With uvloop it appears that None is invalid and raises an error from the _getaddrinfo method.

File "uvloop/loop.pyx", line 1026, in create_server (uvloop/loop.c:19287)
File "uvloop/loop.pyx", line 505, in uvloop.loop.Loop._getaddrinfo (uvloop/loop.c:11976)
TypeError: port must be a str, bytes or int

I came across this because in my use case I create many servers on ephemeral ports that then register their actual bound endpoint details with a service discovery mechanism. In my asyncio based implementation I leave the port at its default value (e.g. port=None) but this failed when I was testing switching over to uvloop. I was able to modify my code to explicitly use port=0 to get it to work with both event loops.

It would be nice if uvloop was a true drop in replacement for asyncio. I expect that a check in the _getaddrinfo method could accomodate this discrepancy.

I have created a short gist that can be used to demonstrate the difference.

uvloop alpha release

Some ToDo items we need completed before we make an alpha release:

  • Implement loop.create_connection
  • Add few more tests for loop.create_server to increase the coverage
  • Add a few high-level tests involving aiohttp or another real codebase
  • Implement loop.create_unix_server
  • Implement loop.create_unix_connection
  • Implement loop.subprocess_shell and loop.subprocess_exec
  • Implement loop.getnameinfo
  • Implement loop.add_signal_handler and loop.remove_signal_handler
  • Implement loop.connect_read_pipe and loop.connect_write_pipe
  • Implement SSL
  • Implement loop.create_datagram_endpoint

UVTimer leaks

  • uvloop version: 0.4.29
  • python version: 3.5.1
  • platform: FreeBSD

We are seeing a serious memory leak that only shows up when using UVLoop on Gunicorn. Examining the GC state does not show anything useful, which indicates to me that it is likely inside native code, so I report this here. Under moderate load, we see each Gunicorn worker taking 2GB ram within 18-24 hours.

We are presently retesting under 0.4.32 and will update if we still see it. If you have any suggestions for debugging it that may be helpful, that would be useful too. We only see this leak under UVLoop and not the default asyncio loop, so it has to be related to UVLoop.

asyncio.get_child_watcher Not Working

  • uvloop version: Latest PIP
  • python version: Python 3.5.2
  • platform: Ubuntu Server 14.04

Using uvloop and asyncio.SubprocessProtocol to run shell commands asynchronously, these create zombie processes which are not garbage collected properly unless you run asyncio.get_child_watcher().attach_loop(loop).

Since I am using uvloop as the event loop policy, for some reason it is not calling the unix_events function from the asyncio module and returning this error:

Traceback (most recent call last): File "/root/discord/bot0.py", line 18, in <module> bot = NotSoBot(loop=loop, shard_id=shard_id, shard_count=shard_count, dev_mode=dev_mode, max_messages=10000) File "/root/discord/bot.py", line 126, in __init__ asyncio.get_child_watcher().attach_loop(self.loop) File "/usr/lib/python3.5/asyncio/events.py", line 647, in get_child_watcher return get_event_loop_policy().get_child_watcher() File "/usr/lib/python3.5/asyncio/events.py", line 538, in get_child_watcher raise NotImplementedError NotImplementedError

The function is only available for unix systems which I am on but with uvloop it doesn't work.

uvloop does not implement _write_to_self

  • uvloop version: 0.7.0
  • python version: 3.6.0
  • platform: OS X El Capitan 10.11.6

With asyncio default event loop the following works fine:

import asyncio

async def async_gen():
    yield 5

async def run():
    val = await async_gen().__anext__()
    print(val)

asyncio.get_event_loop().run_until_complete(run())

With uvloop, the following will raise a AttributeError: 'Loop' object has no attribute '_write_to_self'

import asyncio
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

async def async_gen():
    yield 5

async def run():
    val = await async_gen().__anext__()
    print(val)

asyncio.get_event_loop().run_until_complete(run())

After a quick glance it looks like some of the new async generator code is calling _write_to_self, in particular the _asyncgen_finalizer_hook method. However it doesn't seem like the uvloop event loop has implemented that method yet.

Mac OS X installation

Requires brew install libtool because of autogen.sh: line 43: glibtoolize: command not found

Could not build uvloop: uvloop/loop.c no such file or directory

Hi,

I tried to build the project to run the test on my machine (Ubuntu 14.04, 64b).

First I had to install automake and libtool (you may want to add this step on the readme), then running make I got:

    building 'uvloop.loop' extension
    creating build
    creating build/temp.linux-x86_64-3.5
    creating build/temp.linux-x86_64-3.5/uvloop
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/user/.local/share/virtualenvs/ace813a9968fde5/include/python3.5m -Ivendor/libuv/include -c uvloop/loop.c -o build/temp.linux-x86_64-3.5/uvloop/loop.o
    x86_64-linux-gnu-gcc: error: uvloop/loop.c: No such file or directory
    x86_64-linux-gnu-gcc: fatal error: no input files
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 4
    make: *** [compile] Erreur 1

Feature request: add support for Tornado ioloop

uvloop targets asyncio specifically but given that Tornado can use asyncio as the main event loop [1] [2] it looks like integrating the 3 (tornado + asyncio loop + uvloop) could be possible, and perhaps also fairly easily.

I haven't looked into uvloop code specifically but given that the tornado -> asyncio loop bridge is already in place in tornado's code, such an integration could already be possible in principle, in which case the proposal is to investigate how to "enable it" and document it (possibly in both projects).

[1] http://www.tornadoweb.org/en/stable/asyncio.html
[2] #9

Asyncio incompatibility: address already in use

Hello,

I thought I'd give the aiofiles tests a go on uvloop and had a few failing. Here's a minimal program that works under the default asyncio, and doesn't with uvloop:

import asyncio
import socket

from os.path import join, dirname

import uvloop

asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

async def test_sendfile_socket():
    """Test the sendfile functionality, file-to-socket."""
    port = 50000

    @asyncio.coroutine
    def serve_file(_, writer):
        pass

    server = await asyncio.start_server(serve_file, port=port)

    await server.wait_closed()


loop = asyncio.get_event_loop()

loop.run_until_complete(test_sendfile_socket())
> python t.py
Traceback (most recent call last):
  File "uvloop/loop.pyx", line 1168, in uvloop.loop.Loop.create_server (uvloop/loop.c:19802)
  File "uvloop/handles/streamserver.pyx", line 38, in uvloop.loop.UVStreamServer.listen (uvloop/loop.c:56668)
  File "uvloop/handles/streamserver.pyx", line 75, in uvloop.loop.UVStreamServer._fatal_error (uvloop/loop.c:57149)
OSError: [Errno 98] Address already in use

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "t.py", line 25, in <module>
    loop.run_until_complete(test_sendfile_socket())
  File "uvloop/loop.pyx", line 1015, in uvloop.loop.Loop.run_until_complete (uvloop/loop.c:18124)
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 240, in _step
    result = coro.send(None)
  File "t.py", line 18, in test_sendfile_socket
    server = await asyncio.start_server(serve_file, port=port)
  File "/usr/lib/python3.5/asyncio/streams.py", line 116, in start_server
    return (yield from loop.create_server(factory, host, port, **kwds))
  File "uvloop/loop.pyx", line 1172, in create_server (uvloop/loop.c:19927)
OSError: [Errno 98] error while attempting to bind on address ('::', 50000, 0, 0): address already in use

64-bit Ubuntu 16.04, Python 3.5. Using reuse_port=True will solve it, not sure if that's the difference.

Transport extra info "socket" is not upgraded to SSL after STARTTLS

  • uvloop version: uvloop (0.5.4)
  • python version: Python 3.5.2
  • platform: Linux 4.7.6-1-ARCH x86_64

After performing STARTTLS, transp.get_extra_info("socket") still returns the original socket and not the SSL socket. In the builtin asyncio event loop the transports extra info is upgraded to point to the SSL socket. I am unsure if this happens after a direct TLS connection or not.

I also don't have an easy to reproduce example without requiring an account somewhere; using the latest version of slixmpp to connect to any XMPP server should attempt STARTTLS and reproduce the issue, but I'm afraid that's not the most helpful advice in a bug report. I'll try to put together a minimal working example.

Hang waiting for create_server

  • uvloop 0.4.33:
  • python 3.5.1, 3.5.2:
  • platform Mac OS X, Ubuntu 16.04/4.4.12-boot2docker:

I need some suggestions for debugging server startup hangs.

Startup code looks like:

 cr = loop.create_server(self.factory, addr[0], addr[1],
                                reuse_address=True, ssl=ssl)
f = asyncio.async(cr, loop=loop)
server = loop.run_until_complete(f)

What's sometimes happening, in the context of the ZEO tests, is that the run_loop_until_complete call above never returns.

This is in a port of ZEO, github.com/zopefoundation/ZEO, to asyncio. The ZEO tests are exhaustive, and I suspect in uvloop's case, exhausting. :) The tests start and stop servers 10s, maybe 100s of times in a test runs that typically lasts a few minutes. If I run a smaller subset of the tests I don't get a hang.

Most tests run the server as a subprocess using multiprocessing. Some tests run the server using threading. It appears that the server isn't exiting, but merely hanging.

Do you have any suggestions for how to debug this? I've tried setting PYTHONASYNCIODEBUG=1 and enabling debug logging, but that isn't yielding any additional information..

Pulsar in multiprocessing mode and Ctrl-C issue

  • uvloop version: 0.5.3
  • python version: 3.5.2
  • platform: Mac OS X 10.11.6

Hi, I'm testing uvloop integration with pulsar and it works a treat!
However, I have one problem when running in multiprocessing mode.

If I Ctrl-C on the shell, the master process receives the SIGINT signal but the subprocesses do not and do not respond to the master process sending QUIT signals either.
The subprocesses get eventually killed by pulsar master process using SIGKILL.
This only happen with uvloop.
Is there something I need to be aware about Ctrl-C with uvloop?

uvloop + aiohttp RuntimeError: Timeout context manager should be used inside a task

  • uvloop version: 0.6.7
  • python version: 3.5.2
  • platform: CentOS 6.6
  • aiohttp version: 1.0.5

Hi guys,

My program runs normally when using default asyncio policy. But if installing the uvloop event loop policy,

import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

the following exception will be raised:

Traceback (most recent call last):
  File "/opt/venv/websec/lib/python3.5/site-packages/aiohttp/client.py", line 565, in __aenter__
    self._resp = yield from self._coro
  File "/opt/venv/websec/lib/python3.5/site-packages/aiohttp/client.py", line 197, in _request
    with Timeout(timeout, loop=self._loop):
  File "/opt/venv/websec/lib/python3.5/site-packages/async_timeout/__init__.py", line 33, in __enter__
    raise RuntimeError('Timeout context manager should be used '
RuntimeError: Timeout context manager should be used inside a task

Thanks!

Solaris 11 build fix

  • uvloop version: 0.5.0 / master branch
  • python version: python 3.5
  • platform: Solaris 11

Build is successful on Solaris 11, but the uvloop.so file is missing references to kstat_* methods. To correct the issue the the gcc linker needs -lkstat

The below patch fixes the issue.

diff --git a/setup.py b/setup.py
index ba7bff2..e30783f 100644
--- a/setup.py
+++ b/setup.py
@@ -86,6 +86,8 @@ class libuv_build_ext(build_ext):
             self.compiler.add_library('rt')
         elif sys.platform.startswith('freebsd'):
             self.compiler.add_library('kvm')
+        elif sys.platform.startswith('sunos'):
+            self.compiler.add_library('kstat')

         super().build_extensions()

UDPTransport isn't sending traffic to remote_addr when local_addr is set

  • uvloop version: 0.5.2
  • python version: Python 3.5.2
  • platform: Archlinux

Using the asyncio example udp echo server with uvloop. Works perfectly without uvloop, but once uvloop is used, the remote addr isn't set on the transport. Fails to send packets to the remote machine.

Note: It works if you run the server/client on the same machine. To replicate this, try going between two machines.

Inspecting the socket of the created transport, I see <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 55216)>.

With asyncio, raddr is set (<socket.socket fd=6, family=AddressFamily.AF_INET, type=2050, proto=17, laddr=('10.200.0.9', 33074), raddr=('10.10.8.30', 9999)>).

I'd like to use this with aiosip, but I'm also seeing this problem there.

Using uvloop on Mac create a lot of CLOSE_WAIT

I'm trying to use uvloop in my proxy project. The diff is below (or see it online here):

diff -r 4c40523b6830 -r 575a8ff4284f wormhole/proxy.py
--- a/wormhole/proxy.py Fri May 06 14:41:45 2016 +0700
+++ b/wormhole/proxy.py Tue May 10 09:44:00 2016 +0700
@@ -62,6 +62,12 @@
     if not (1 <= args.port <= 65535):
         parser.error('port must be 1-65535')

+    try:
+        import uvloop
+    except ImportError:
+        pass
+    else:
+        asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
     loop = asyncio.get_event_loop()
     try:
         loop.run_until_complete(

It work pretty fast for a while, then everything just stop. I stop process by control-c. ps show no process running. However, netstat -an show a lot of CLOSE_WAIT connections, and then I have to reboot my Mac to clear them.

Do I did something wrong in my code? It never happened when I use pure asyncio.

Provide wheels

Consider providing wheels for all the major platforms. They install much more quickly and do no require your users to have a C toolchain and development libs installed.

aiohttp and uvloop - crash

  • uvloop version: 0.4.31
  • python version: Python 3.5.2rc1
  • platform: Linux debian-7 3.2.0-4-amd64

I'm not sure that this is a problem uvloop.
aio-libs/aiohttp#925

import asyncio
import aiohttp
import uvloop
from aiohttp.resolver import AsyncResolver

async def run(_loop):
    r = AsyncResolver(loop=_loop)
    conn = aiohttp.TCPConnector(loop=_loop, resolver=r)

    with aiohttp.ClientSession(connector=conn, loop=_loop) as ses:
        async with ses.get('http://vk.com') as resp:
            if resp.status == 200:
                print(await resp.read())
            else:
                print(resp.status)

if __name__ == '__main__':
    asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
    loop = asyncio.get_event_loop()
    loop.run_until_complete(run(loop))
    loop.close()

Result:

python3: src/unix/core.c:888: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.
bash: line 1:  4020 aborted  /usr/bin/python3 -u /vagrant/test.py
python finished with exit code 134

Unable to install uvloop

pip install uvloop throws this error

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-3p6l9uj5/uvloop/setup.py", line 81, in <module>
        include_package_data=True
      File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/anand/.virtualenvs/35/lib/python3.5/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.5/distutils/command/install.py", line 583, in run
        self.run_command('build')
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/anand/.virtualenvs/35/lib/python3.5/site-packages/setuptools/command/build_ext.py", line 49, in run
        _build_ext.run(self)
      File "/usr/lib/python3.5/distutils/command/build_ext.py", line 338, in run
        self.build_extensions()
      File "/tmp/pip-build-3p6l9uj5/uvloop/setup.py", line 40, in build_extensions
        self.build_libuv()
      File "/tmp/pip-build-3p6l9uj5/uvloop/setup.py", line 35, in build_libuv
        subprocess.run(['make', j_flag], cwd=LIBUV_DIR, env=env, check=True)
      File "/usr/lib/python3.5/subprocess.py", line 711, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['make', '-j4']' returned non-zero exit status 2

    ----------------------------------------

uvloop crashed and core dumped

  • uvloop version: 0.4.33
  • python version: 3.5.1
  • platform: VMware Photon Linux 1.0

I just rebuild my wormhole proxy docker image today (Dockerfile). After I use it for a few minutes, it crashed and show error messages like these (randomly on each time it crashed).

python3.5: src/unix/core.c:236: uv__finish_close: Assertion `handle->flags & UV_CLOSING' failed.

or

python3.5: src/unix/core.c:264: uv__finish_close: Assertion `0' failed.

or

python3.5: src/unix/stream.c:444: uv__stream_destroy: Assertion `(((const QUEUE ) (&(stream->loop)->active_reqs) == (const QUEUE *) ((QUEUE *) &(((&(stream->loop)->active_reqs))[0]))) == 0)' failed.

I also tried to build it on Alpine Linux image (Dockerfile) the result is also the same.

BTW, I have no problem with the previous image that was built around 2 weeks ago, the uvloop version on that image was 0.4.30.

libuv filesystem async calls

AFAIK, libuv also support async filesystem calls. Can this be implemented in uvloop (or based on uvloop) with nice await with and some pathlib awaitable coroutines? If so, «not good» executors for filesystem calls can be dropped 👏

pulsar socket server in multiprocessing with uvloop

  • uvloop version: 0.6.7
  • python version: 3.5.2 & 3.6.0
  • platform: Mac OS x

When running pulsar in multiprocessing mode I get an error when using uvloop as the event loop.
To reproduce the error try the same code in issue #54 with pulsar from master branch:

pip install git+http://github.com/quantmind/pulsar.git@master#egg=pulsar uvloop
py scripy.py --io uv --concurrency multi -w 2

uvloop complains about the socket being not valid (closed probably).

11:24:10 [p=13310, t=140736205472704, ERROR, asyncio] Task exception was never retrieved
future: <Task finished coro=<SocketServer.worker_start() done, defined at /Users/lsbardel/.pyenv/versions/3.5.2/lib/python3.5/site-packages/pulsar/apps/socket/__init__.py:270> exception=OSError(38, 'Socket operation on non-socket')>
Traceback (most recent call last):
  File "uvloop/future.pyx", line 372, in uvloop.loop.BaseTask._fast_step (uvloop/loop.c:105885)
  File "/Users/lsbardel/.pyenv/versions/3.5.2/lib/python3.5/site-packages/pulsar/apps/socket/__init__.py", line 274, in worker_start
    server = await self.create_server(worker)
  File "/Users/lsbardel/.pyenv/versions/3.5.2/lib/python3.5/site-packages/pulsar/apps/socket/__init__.py", line 323, in create_server
    await server.start_serving(cfg.backlog, sslcontext=self.sslcontext())
  File "/Users/lsbardel/.pyenv/versions/3.5.2/lib/python3.5/site-packages/pulsar/async/protocols.py", line 666, in start_serving
    ssl=sslcontext)
  File "uvloop/loop.pyx", line 1376, in create_server (uvloop/loop.c:26228)
  File "uvloop/loop.pyx", line 1373, in uvloop.loop.Loop.create_server (uvloop/loop.c:26167)
  File "uvloop/handles/streamserver.pyx", line 43, in uvloop.loop.UVStreamServer.listen (uvloop/loop.c:72688)
  File "uvloop/handles/streamserver.pyx", line 80, in uvloop.loop.UVStreamServer._fatal_error (uvloop/loop.c:73173)
OSError: [Errno 38] Socket operation on non-socket

This occurs when using asyncio subprocess as process concurrency.
It looks like uvloop has closed the socket in the main process.

Pulsar create the socket server in the main process, than loop.remove_reader and pass the socket to subprocesses (implementation is https://github.com/quantmind/pulsar/blob/master/pulsar/apps/socket/__init__.py#L220 last five lines in the method).
This works with vanilla asyncio but it may not be the best way of doing it.

Can't leave host=None in loop.create_server

With release 0.4.11, if you leave host=None while setting the port with create_sever() you end up with:

File "uvloop/loop.pyx", line 1028, in create_server (uvloop/loop.c:19317)
File "uvloop/loop.pyx", line 511, in uvloop.loop.Loop._getaddrinfo (uvloop/loop.c:12089)
TypeError: host must be a str or bytes

This is different behavior from the base asyncio create_server that allows host=None while setting the port.

Production ready?

Is UVLoop production ready? Getting ready to implement some new microservices on top of asyncio using UVLoop.

TCPTransport object has no attribute _protocol

  • uvloop version: 0.5.0
  • python version: 3.5.2
  • platform: os x 10.11.6

Tried running aiopyramid websocket example, it raises AttributeError: 'uvloop.loop.TCPTransport' object has no attribute '_protocol'.

With asyncio event loop it's working fine.

aiopyramid is using internal API, but is there any way of providing similar _protocol property in uvloop?

Compat for Python 2.7

Would love to look at adopting uvloop in a number of high-scale tornado apps we have in production. Those are running Python 2.7 - what's stopping this library from adding support?

uvloop beta release

  • Release alpha (see issue #3)
  • Add support for UDP
  • Support SSL in loop.create_connection, loop.create_unix_connection, loop.create_server, loop.create_unix_server

UNIX socket is automatically unlinked

When using the built-in event loop, you need to do something like this:

server = await asyncio.start_unix_server(client_connected, path=path)
try:
    # ...
finally:
    server.close()
    await server.wait_closed()
    os.unlink(path)

However, when I use uvloop, the os.unlink() call raises a FileNotFoundError exception.

I was wondering if you already know about this and if it is intentional?

I don't mind adding a try/except clause to silence the exception, but if you're aiming for this event loop to be a drop-in replacement for the built-in event loop, I guess you're going to want to change this?

UVLoop sets FD_CLOEXEC on descriptors it doesn't own

  • uvloop version: 0.4.33
  • python version: 3.5
  • platform: Mac OSX 10.11

Creating a uvloop.Loop has the surprising side effect of marking file descriptors it doesn't own (notably including stdin/out/err) as FD_CLOEXEC. This causes any use of subprocesses which do not explicitly assign stdin/out/err to fail, sometimes in surprising ways. This toy example should print "hi", but does nothing if a uvloop.Loop has been created. In more complex examples a subprocess may open other files or sockets, those sockets get assigned the first three file descriptors, and its stdio output gets interleaved with its network traffic.

import uvloop
import subprocess

uvloop.Loop()
p = subprocess.Popen(["echo", "hi"])
p.wait()

Loop time does not pass when loop is blocked

  • uvloop version: 0.4.25
  • python version: 3.5
  • platform: Mac OS X, Linux

Some of my timing- and debouncing-related tests are failing when I replace asyncio loop with uvloop. I tracked it down to loop.time not increasing when the loop isn't running (e.g. when I'm blocking with time.sleep).

First, an example of call_later malfunctioning:

import asyncio
import time
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())


def f():
    print('f   ', time.time())


async def main(loop):
    await asyncio.sleep(0.001)
    time.sleep(1)
    print('main', time.time())
    loop.call_later(1, f)
    await asyncio.sleep(1)


loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))

There should be a 1-second difference between the two printed times, but actual output is

main 1463724375.72489
f    1463724375.725403

Second, a demonstration that this has something to do with loop.time:

import asyncio
import time
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())


async def test1(loop):
    T0 = loop.time()
    time.sleep(1)
    print(loop.time() - T0)


async def test2(loop):
    T0 = loop.time()
    time.sleep(1)
    await asyncio.sleep(0.001)
    print(loop.time() - T0)


async def test3(loop):
    T0 = loop.time()
    await asyncio.sleep(0.001)
    time.sleep(1)
    print(loop.time() - T0)


loop = asyncio.get_event_loop()
loop.run_until_complete(test1(loop))
loop.run_until_complete(test2(loop))
loop.run_until_complete(test3(loop))

In all three cases a difference of 1 second should be printed. Actual output is

0.0
1.0019999999785796
0.0010000000474974513

Note that test2 works as expected because the loop is given a chance to "catch up" during that millisecond wait. test3 shows that asyncio.sleep before the blocking call is accounted for.

Both examples behave "correctly" if you just comment out the set_event_loop_policy call.

I have no idea how to fix this but I hope this report helps you. Really excited to try uvloop in a stock trading platform I work on!

uvloop breaks aiohttp's capability of serving static files

  • uvloop version: 0.5.3
  • python version: 3.5.2
  • platform: Arch Linux / 4.7.1-1-ARCH / x86_64

uvloop breaks aiohttp's capability of serving static files. It makes the server send the response headers after the body.

To run this, have aiohttp and uvloop installed, then run the gist ( https://gist.github.com/diogobaeder/0558c43d785cd82a1ecff553bc583867 ):
$ python uvhttp.py

Then make a request to the app, making sure to get verbose output:
$ curl -v localhost:8765/jquery.js

These are the results I'm getting when serving jQuery:

  1. Without uvloop (correct response): http://pastebin.com/e6ep3g22
  2. With uvloop (bad response): http://pastebin.com/Kg37DXB8

Notice how with uvloop the response headers are sent after the body.

I'm not acquainted with the internals of aiohttp or uvloop, but it seems that the former relies on some parser from asyncio and uvloop seems to replace it by another one that behaves differently.

Any ideas of why this happens? At the moment this is preventing me to use uvloop, unfortunately...

Thanks!

BaseTransport._fatal_error should ignore Cancellation

  • uvloop version: 0.4.31
  • python version: 3.5.2
  • platform: ubuntu 14.04
import asyncio
import os
if os.getenv('USE_UVLOOP'):
    import uvloop
    asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())



HOST="10.99.99.99"  # should not exist on your network!
PORT=9090

async def doconnect(loop):
    try:
        return await asyncio.wait_for(asyncio.open_connection(HOST, PORT), timeout=2)
    except asyncio.TimeoutError:
        pass

loop=asyncio.get_event_loop()
loop.run_until_complete( doconnect(loop))

In classic asyncio eventloop this program has no output. With USE_UVLOOP=1 it logs the following at the ERROR level

Fatal error on transport TCPTransport (connect failed)
protocol: <asyncio.streams.StreamReaderProtocol object at 0x7fb567fa29e8>
transport: <TCPTransport closed=True reading=False 0xe823e8>
concurrent.futures._base.CancelledError

IMHO this should not be logged at this level.
Cancellation should be treated the same as the other exception types that are ignored in BaseTransport._fatal_error (around line 44)

Could uvloop be implemented using CFFI?

Could uvloop be implemented using CFFI? How hard would it be to do so?

I ask because at some point in the future PyPy3 will reach compatibility with Python 3.5 (It could be a while, but PyPy3 3.3 is in alpha right now so there is hope :-) and it would be really great if uvloop were usable on it :-)

move build logic to setup.py

Hi,

Could you move build logic from Makefile to setup.py? Like in asyncpg's 93733dba6c09673660614c014bae955dda9e8796?

TIA :)

aiohttp.worker.GunicornUVLoopWebWorker + unix sockets Nginx - error

  • uvloop version: 0.5.4
  • python version: 3.5.2
  • platform: Centos7

Aiohttp recomended deploy Gunicorn + Nginx
Gunicorn (aiohttp.worker.GunicornWebWorker) + unix socket Nginx - OK
Ginicorn (aiohttp.worker.GunicornUVLoopWebWorker) + unix sockets Nginx - error

Traceback (most recent call last):
File "/usr/local/python/3.5.2/lib/python3.5/site-packages/aiohttp/helpers.py", line 407, in log
[message, environ, response, transport, time]))
File "/usr/local/python/3.5.2/lib/python3.5/site-packages/aiohttp/helpers.py", line 394, in _format_line
return tuple(m(args) for m in self._methods)
File "/usr/local/python/3.5.2/lib/python3.5/site-packages/aiohttp/helpers.py", line 394, in
return tuple(m(args) for m in self._methods)
File "/usr/local/python/3.5.2/lib/python3.5/site-packages/aiohttp/helpers.py", line 347, in _format_a
peername = args[3].get_extra_info('peername')
File "uvloop/handles/tcp.pyx", line 150, in uvloop.loop.TCPTransport.get_extra_info (uvloop/loop.c:66876)
File "uvloop/dns.pyx", line 60, in uvloop.loop.__convert_sockaddr_to_pyaddr (uvloop/loop.c:82997)
RuntimeError: cannot convert sockaddr into Python object

Looks like loop is blocking with aiohttp.worker.GunicornWebWorker

When I deploy an aiohttp app with aiohttp.worker.GunicornWebWorker it looks like the loop is blocked and app never serves requests.

When I run without the worker (with web.run_app(app)), it works well.
I think the problem could be caused by the GunicornWebWorker itself since it's closing the default asyncio's loop and then creating new one. But I'm not sure.

Is this issue related to uvloop or I should post to aiohttp's issue tracker?

TypeError on Future declaration in Python 3.6.0b2

  • uvloop version: 89bf69f
  • python version: Python 3.6.0b2 (default, Oct 12 2016, 20:13:22) (From docker python:3.6 (e6d7f066d57f))
  • platform: Linux 4.7.6-200.fc24.x86_64

Trying out uvloop on the b2 build of python 3.6 fails to import..

Python 3.6.0b2 (default, Oct 12 2016, 20:13:22) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import uvloop
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mayfield/project/uvloop/uvloop/__init__.py", line 7, in <module>
    from .loop import Loop as __BaseLoop, Future
  File "uvloop/future.pyx", line 249, in init uvloop.loop (uvloop/loop.c:114059)
    class Future(BaseFuture, aio_Future):
TypeError: multiple bases have instance lay-out conflict

minor mismatch in add_signal_handler

I picked this up in my unit tests which are admittedly testing some edge cases. When trying to use the loop to register a signal handler for SIGKILL (which itself is invalid) asyncio and uvloop behave differently.

A simple test that demonstrates the difference is this:

import asyncio
import signal

# import uvloop
# asyncio.DefaultEventLoopPolicy = uvloop.EventLoopPolicy
# asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

def my_sig_handler():
    pass

loop = asyncio.get_event_loop()
loop.add_signal_handler(signal.SIGKILL, my_sig_handler)
loop.stop()
loop.close()

Running the script as is produces the asyncio output:

$ python3.5 uvloop-sig-test.py 
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/asyncio/unix_events.py", line 95, in add_signal_handler
    signal.signal(sig, _sighandler_noop)
  File "/usr/local/lib/python3.5/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "uvloop-sig-test.py", line 31, in <module>
    loop.add_signal_handler(signal.SIGKILL, my_sig_handler)
  File "/usr/local/lib/python3.5/asyncio/unix_events.py", line 108, in add_signal_handler
    raise RuntimeError('sig {} cannot be caught'.format(sig))
RuntimeError: sig 9 cannot be caught

By uncommenting the uvloop part and running the script will produce the uvloop output:

$ python3.5 uvloop-sig-test.py 
Traceback (most recent call last):
  File "uvloop-sig-test.py", line 31, in <module>
    loop.add_signal_handler(signal.SIGKILL, my_sig_handler)
  File "uvloop/loop.pyx", line 1632, in uvloop.loop.Loop.add_signal_handler (uvloop/loop.c:30442)
  File "uvloop/handles/signal.pyx", line 52, in uvloop.loop.UVSignal.start (uvloop/loop.c:42667)
  File "uvloop/handles/handle.pyx", line 145, in uvloop.loop.UVHandle._fatal_error (uvloop/loop.c:37619)
OSError: [Errno 22] Invalid argument

On the box that I ran this test on the Python version was 3.5.0 (not the latest 3.5.1) so the Python module line numbers may be a little out.

I actually think that uvloop does the right thing in reporting the first error. However it would appear that the asyncio developers observed that the error reported is not very informative and wrapped it with a more informative error report.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.