What steps will reproduce the problem?

1. Build gevent with libevent2 (I've used 2.0.3-alpha)
2. Run test__http.py.

What is the expected output? What do you see instead?
The test should pass but it does not.

The example scripts (wsgiserver.py and httpserver.py) work fine, so the
problem might be in test itself or somewhere else.

Reported by Denis.Bilenko.

earlier comments

PHusiatynski said, at 2010-04-11T18:00:54.000Z:

Works here, only some fail logs on stderr:

python -c 'import gevent; print gevent.version_info; print gevent.core.get_version()'

(0, 12, 3)
2.0.4-alpha

$ python greentest/test__http.py
..Traceback (most recent call last):
  File "/mnt/sda5/Projekty/virtualenvs/gevent/gevent/gevent/http.py", line 117, in
_cb_request
    self.handle(request)
  File "greentest/test__http.py", line 135, in handle
    raise Expected
Expected
Failed to handle request: 

.Traceback (most recent call last):
  File "/mnt/sda5/Projekty/virtualenvs/gevent/gevent/gevent/greenlet.py", line 388,
in run
    result = self._run(*self.args, **self.kwargs)
  File "/mnt/sda5/Projekty/virtualenvs/gevent/gevent/gevent/util.py", line 36, in
__call__
    return func(*args, **kwargs)
  File "greentest/test__http.py", line 135, in handle
    raise Expected
Expected
>( failed with Expected

.........
----------------------------------------------------------------------
Ran 12 tests in 0.314s

OK

Except that Greenlet is failing because of raise of Expected. In gevent/http.py,
line 119, there's Greenlet spawn with exception handler that does not match Expected:

greenlet = spawn(wrap_errors(core.HttpRequestDeleted, self.handle), request)</p><p><strong>Denis.Bilenko said, at 2011-12-20T17:55:21.000Z:</strong></p>

We no longer use libevent-http in 1.0.

Currently socket.getaddrinfo is built on top of evdns_resolve_ipv4/v6. It
does not parse /etc/hosts, instead it calls the real getaddrinfo if
hostname has no dots.

Libevent2 has async getaddrinfo with support for parsing /etc/hosts. We
should wrap that and use in gevent.socket module.

Reported by Denis.Bilenko.

earlier comments

Denis.Bilenko said, at 2011-01-28T11:49:37.000Z:

https://bitbucket.org/denis/gevent

Denis.Bilenko said, at 2011-04-07T08:34:31.000Z:

We use c-ares now which handles /etc/hosts

Glyph.Lefkowitz said, at 2011-09-24T03:35:25.000Z:

Does libevent2's async getaddrinfo also honor gai.conf and nsswitch.conf?

Denis.Bilenko said, at 2011-09-24T08:02:55.000Z:

I'm not sure, you have to check the libevent source.

Regardless, gevent.socket.socket.getaddrinfo in 0.13.6 does not use libevent's getaddrinfo, because it was written for libevent 1.4.

It would be nice if the pywsgi has support for optional platform specific file handling via the "wsgi.file_wrapper" key in the environ(http://www.python.org/dev/peps/pep-0333/#optional-platform-specific-file-handling). And allow sendfile to be used for sending files, or a file like objects, in a performant way.

Reported by galfyo.pundee.

earlier comments

Denis.Bilenko said, at 2010-06-28T10:05:21.000Z:

If someone wants to contribute, check out this example: http://bitbucket.org/denis/gevent/src/tip/examples/geventsendfile.py

churunmin said, at 2012-05-19T14:16:45.000Z:

"+1 Me too!".

There's gevent.signal but it has incompatible interface. We should create a function with the interface compatible to signal.signal and then have gevent.monkey replace it.

Reported by Denis.Bilenko.

earlier comments

o.persson said, at 2010-09-24T08:44:22.000Z:

Here's a simple patch to work-around the problem.

Denis.Bilenko said, at 2011-03-28T05:28:31.000Z:

The trunk works with Python's signal module without monkey patching.

Twisted has reactor.callFromThread() for this purpose, with gevent you
currently have to resort to using pipe().

Reported by Denis.Bilenko.

earlier comments

[email protected] said, at 2011-05-06T02:23:11.000Z:

The situation is more desperate on Windows where os.pipe() handles do not work with gevent as they're not sockets. Another option is socket.socketpair() but that is not available on Windows at all.

Denis.Bilenko said, at 2012-01-09T06:48:52.000Z:

In the trunk (soon to be released as 1.0b1), there are async watchers that do exactly that.

async = gevent.get_hub().loop.async()
async.start(my_callback)

in some other thread

async.send()

it works both on linux and on windows.

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?

What version of the product are you using? On what operating system?

Please provide any additional information below.

Reported by lukatmyshu.

gevent uses old libevent API that relies on a global variable to store the
event loop. While it's still supported in the latest versions of libevent,
the preferred way of doing things is to use the newer API that accepts
event_base* explicitly.

Reported by Denis.Bilenko.

earlier comments

8mayday said, at 2010-04-16T07:53:01.000Z:

It can't be reached with libevent 1.4.x, because of DNS subsystem which works only with current_base.

Denis.Bilenko said, at 2011-04-28T13:35:59.000Z:

Since the current development version uses libev, this ticket is no longer valid. We do not use any global variables though and creating multiple loops and hubs is now possible.

Libevent provides functions for making HTTP requests. Would be nice to have
access to it from Python, as it's probably much faster than httplib.

Reported by Denis.Bilenko.

earlier comments

gm41lu53r said, at 2010-02-19T19:18:29.000Z:

Would indeed be nice.

tgannert said, at 2010-03-22T08:06:21.000Z:

Patch to add a httplib like interface to evhttp.

The patch includes the regeneration of core.c,
but that should be easy enough to rip out if need be.

I haven't tested the functionality that much, just run
the httplib examples and a concurrency test (included in the patch).

Denis.Bilenko said, at 2010-03-26T19:05:28.000Z:

Thanks for the patch! Can you upload your repo on bitbucket (fork http://bitbucket.org/denis/gevent)?

o.persson said, at 2010-05-20T15:31:39.000Z:

Here's some small fixes to the previous patch. We've used this http client for a while and haven't found any problems so far.

Denis.Bilenko said, at 2010-06-23T02:31:08.000Z:

Thanks!

If anyone wants to help with this getting merged faster - rebase the patches against the current dev version.

Also, ideally there would be many individual patches that are as small as possible (e.g. each file committed individually) and the whole thing can be browsed online (e.g. on bitbucket).

jk.ladee said, at 2010-08-04T10:36:03.000Z:

The attached patch types ports as 'short' in a couple of places in evhttp.pxi, should be 'unsigned short'.

Denis.Bilenko said, at 2011-01-11T08:02:19.000Z:

I've pushed gevent.httplib module, based on your patches. https://bitbucket.org/denis/gevent/changeset/709bba6277b5 https://bitbucket.org/denis/gevent/changeset/c609d873a19a

Note, that trunk only supports libevent-2 for now.

I'm not totally happy with it, sometimes it fails where regular httplib does not (although I think it avoids crashes). This could be libevent2-specific issue as libevent-http is known to be worse here than on 1.4 and I have not tested it on 1.4 yet.

The one from stdlib will raise ValueError if its current value exceeds its
initialized value. The one from gevent.coros will block instead.

Reported by Denis.Bilenko.

What steps will reproduce the problem?

1. monkey patch socket
2. urllib2.urlopen('https://sf.net/')
3. Enjoy <urlopen error (2, 'The operation did not complete (read)')>

What is the expected output? What do you see instead?
It's expected to fail with a more clear explanation, "ssl package is required"

Reported by Denis.Bilenko.

earlier comments

Denis.Bilenko said, at 2010-06-04T14:54:57.000Z:

Fixed. monkey.patch_socket() now removes 'ssl' from stdlib's socket module if not provided by gevent. Thus httplib raises the following error for https urls: URLError:

gevent 0.12.2, libevent 1.4.13, greenlet 0.3, Mac OS X 10.6.3

I'm attempting to use gevent.wsgi with itty, a web app mini-framework. itty uses cgi.FieldStorage to
parse request data, which calls readline(1<<16) on WSGIHandler.request.input_buffer, but this
throws a TypeError because input_buffer doesn't support readline with an argument.

I tried to work around this by removing the argument to the readline calls, but then things break in
other strange ways, so I'm looking for some other angle of attack.

Reported by levity.

earlier comments

Denis.Bilenko said, at 2010-04-03T03:47:35.000Z:

This is a known bug that has been fixed in the development version.

It will be released as 0.12.3 sometime soon, but in the meantime you can get it from

http://bitbucket.org/denis/gevent

This will eliminate the class of bugs when gevent.monkey was not updated properly.

Reported by Denis.Bilenko.

earlier comments

Denis.Bilenko said, at 2011-04-04T08:43:07.000Z:

Fixed in https://bitbucket.org/denis/gevent/changeset/3e14f226b8dd

Add windows binary packages (egg/msi) for Python 2.5

Reported by Denis.Bilenko.

earlier comments

Denis.Bilenko said, at 2011-04-28T15:46:49.000Z:

I don't think I'll find time for this (this requires installing yet another Microsoft Visual Studio version), but there are 3rdparty builds (untested, unverified, use at your own risk) for Python2.5: http://www.lfd.uci.edu/~gohlke/pythonlibs/#gevent

there is a method waitany that waits for every event to set,waitany should
wait for any event to set and switch back to the current greenlet.
a possible implementation:

def waitany(events):
  switch = getcurrent().switch
  try:
    for ev in events:
      ev.rawlink(switch)
    return get_hub().switch()
  finally:
    for ev in events:
      ev.unlink(switch) 

Reported by uriel.katz.

earlier comments

uriel.katz said, at 2010-04-28T11:50:47.000Z:

a possible test

Denis.Bilenko said, at 2010-07-01T06:50:38.000Z:

here's an alternative implementation (it requires the latest trunk). it works on Event, AsyncResult, Greenlet (everything with ready() and rawlink())

def wait(events, timeout=None):
    result = AsyncResult()
    update = result.set
    try:
        for event in events:
            if event.ready():
                return event
            else:
                event.rawlink(update)
        return result.get(timeout=timeout)
    finally:
        for event in events:
            event.unlink(update)

What steps will reproduce the problem?

1. gevent.sleep(-1.0)

What is the expected output? What do you see instead?

exception, or return immediately.

Reported by schmir.

There are actually two bugs. The first one is using buggy Python library code, which was fixed in http://svn.python.org/view?view=rev&amp;revision=80453. You shouldn't override user timeouts with None. Code hangs up on something as simple as:

import ssl
import socket

sock = socket.socket()
sock.settimeout(5)
sock.connect(('www.google.com', 80))
sock = ssl.wrap_socket(sock)

The second bug is because the timeout value is not copied from the original gevent socket, so the internal timeout value happens to be None. This makes ssl sockets without a timeout and they can hang up very easily. For example:

import ssl
import socket

sock = socket.socket()
sock.settimeout(5)
sock.connect(('www.google.com', 80))
sock = ssl.wrap_socket(sock, do_handshake_on_connect=False)
print sock.gettimeout()

Would print 5.0 on Python 2.6, but it prints None when ssl is monkey patched.

Reported by snaury.

earlier comments

snaury said, at 2010-09-09T21:51:53.000Z:

And here's a possible patch. I'm not sure if it causes any tests to fail though...

snaury said, at 2010-09-09T22:21:34.000Z:

Ok, I found that the previous patch was actually causing test_ssl.py to timeout. Here's a better patch.

Denis.Bilenko said, at 2010-09-15T15:01:34.000Z:

I believe this is fixed in the current trunk (http://bitbucket.org/denis/gevent/src). Could you try it and confirm that?

snaury said, at 2010-09-15T18:51:28.000Z:

Yes, it appears it doesn't block anymore. Thank you.

gevent.subprocess would be nice to have as well as gevent.os.popen*.

Here's an example that can be used as a start:
http://bitbucket.org/denis/gevent/src/tip/examples/processes.py

Reported by Denis.Bilenko.

earlier comments

[email protected] said, at 2012-02-08T18:42:57.000Z:

Perhaps this project can be joined in: https://bitbucket.org/eriks5/gevent-subprocess ?

pacemkr said, at 2012-02-28T19:26:23.000Z:

I've looked at the gevent-subprocess project, and it looked slightly too complicated for the task. A special pipe class that is a "queue that can be closed," no doubt it works, but I felt we could do better:

https://bitbucket.org/denis/gevent-0.13/pull-request/1/subprocess-support-well-tested

Largely made possible by this discussion: http://groups.google.com/group/gevent/browse_thread/thread/dba1a5d29e0a60ff/3efce8d3cc76dc9d?lnk=gst&q=subprocess#3efce8d3cc76dc9d

Forgive my cross-posting this to the list and at bitbucket itself, just making it easier for people to find.

Denis.Bilenko said, at 2012-05-11T21:48:57.000Z:

The trunk now does have gevent.subprocess module: https://bitbucket.org/denis/gevent/src/beb730937d9f/gevent/subprocess.py

What steps will reproduce the problem?

1. Run a pywsgi server which reads data from wsgi.input
2. Send a request contain body but without content-length
3. Here you are

What is the expected output? What do you see instead?

Traceback (most recent call last):
File "c:\python26\lib\site-packages\gevent\greenlet.py", line 403, in run
result = self._run(_self.args, *_self.kwargs)
File "H:\workspace\middleman\proxy.py", line 69, in _serverToClient
data = req.body_file.read(self.chunkSize)
File "c:\python26\lib\site-packages\gevent\pywsgi.py", line 130, in read
return self._do_read(self.rfile.read, length)
File "c:\python26\lib\site-packages\gevent\pywsgi.py", line 72, in _do_read
if length and length > self.content_length - self.position:
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

What version of the gevent are you using?
0.13.0-beta1

On what operating system?
Windows 7

On what Python?
Python2.6

Which event loop are you running? (Paste the output of python -c "from
gevent.core import *; print get_method(), get_version()")

Please provide any additional information below.

Reported by Bornstub.

earlier comments

schmir said, at 2010-09-17T20:39:06.000Z:

let me quote http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4: "For compatibility with HTTP/1.0 applications, HTTP/1.1 requests containing a message-body MUST include a valid Content-Length header field unless the server is known to be HTTP/1.1 compliant. If a request contains a message-body and a Content-Length is not given, the server SHOULD respond with 400 (bad request) if it cannot determine the length of the message, or with 411 (length required) if it wishes to insist on receiving a valid Content-Length."

emil.vladev said, at 2010-10-15T11:19:31.000Z:

Here is simple patch that will just make pywsgi work and not throw the aforementioned exception when there is no Content-Length header.

emil.vladev said, at 2010-10-15T11:39:53.000Z:

Here is a new patch, the same as the previous one, but with a test:

Denis.Bilenko said, at 2011-04-28T15:36:59.000Z:

I think this was fixed in release 0.13.4: https://bitbucket.org/denis/gevent-0.13/changeset/09cef9a9d28c (the same patch is also available in the trunk).

I think, absence of Content-Length or Transfer-Encoding header means that the request does not include message-body, not "read until EOF".

Please confirm if it's fixed for you.

What steps will reproduce the problem?

1. monkey.patch_all()
2. daemonize the process
3. use any socket function that resolves a hostname

What is the expected output? What do you see instead?
It should just work, however dns resolution hangs for some time and then
fails with DNSError 67 timeout.

The workaround is not to patch dns functions:
monkey.patch_all(dns=False)

Reported by Denis.Bilenko.

earlier comments

bchesneau said, at 2010-04-14T09:56:27.000Z:

same bug on gunicorn. We use same fix. Any idea what causing the problem ?

uriel.katz said, at 2010-04-27T12:26:38.000Z:

i don`t think it happen to me when i use python-daemon,how are you daemonizing?

Denis.Bilenko said, at 2010-04-27T12:35:02.000Z:

I've seen it happen with python-daemon.

If it works for you, then it's probably because you import gevent (and thus initialize
libevent) after daemonizing. (This is another workaround).

Denis.Bilenko said, at 2010-04-28T18:38:36.000Z:

libevent-dns creates a socket and has no way of re-creating it after it was closed. Daemonizing libraries like to close all the files, thus screwing with libevent-dns socket.

Not closing any files seems to alleviate the problem. Here's how to do it with
python-daemon:

class All(object):
def contains(self, item):
return True

class MyDaemonContext(DaemonContext):

def _get_exclude_file_descriptors(self):
    return All()

uriel.katz said, at 2010-04-29T22:07:58.000Z:

i using python-daemon and telling it to not close all files,like this: with DaemonContext(files_preserve=range(maxfd() + 2)): my code daemonized

what we can do is make a variable in gevent.core which is the fd number of the socket
that libevent-dns makes and pass that fd to the daemon library.

Denis.Bilenko said, at 2011-04-20T04:16:57.000Z:

This issue was closed by revision 5c4bd86c3b7d.

Denis.Bilenko said, at 2011-04-20T04:36:23.000Z:

The trunk, which uses libev + c-ares, instead of libevent + libevent-dns now works correctly after fork();

If there are outstanding DNS requests during fork(), they will be resolved as usual in the parent but will raise an exception in the child. All new requests will succeed.

Here's the code that does re-initialization after fork(): https://bitbucket.org/denis/gevent/src/1fa4b2344cf6/gevent/resolver_ares.py#cl-22

Please test and give feedback!

Denis.Bilenko said, at 2011-04-20T04:39:00.000Z:

In fact, we can make the code catch that exception and retry the request. What do you think?

Denis.Bilenko said, at 2011-04-23T07:57:15.000Z:

OK, I did that: https://bitbucket.org/denis/gevent/changeset/2baff666de4e

Even outstanding DNS requests complete after fork() in both parent and child.

What steps will reproduce the problem?
1.Using Mac OSX Snow leopard
2. I have version Gevent 0.12.2 LibEvent 1.4.13-stable
3. Python crashes

What is the expected output? What do you see instead?
The program is supposed to download pictures for storage from smugmug. I am
getting a massive core dump here http://pastebin.com/s82ARj9m

What version of the product are you using? On what operating system?

Please provide any additional information below.

Reported by nytrokiss.

earlier comments

nytrokiss said, at 2010-05-06T18:37:05.000Z:

Sorry I missed the traceback Traceback (most recent call last): File "core.pyx", line 136, in gevent.core.__event_handler (gevent/core.c:1160) File "build/bdist.macosx-10.6-universal/egg/gevent/socket.py", line 133, in _wait_helper current.switch(ev) AttributeError: 'readwrite_event' object has no attribute 'switch'

thruflo said, at 2010-05-18T17:33:53.000Z:

I have this issue too with http://gist.github.com/405271

Not sure if using fork() or this compilation mismatch warning may be the cause, or if
it's something else::

UserWarning: libevent version mismatch: system version is '1.4.13-stable' but

this gevent is compiled against '1.4.9-stable'
#wT1####[^_]ËGT#P#$#R####Z

On OSX 10.4.

Denis.Bilenko said, at 2010-05-19T04:12:42.000Z:

The version mismatch could certainly cause the crash. Please get the libevent headers for 1.4.13-stable and rebuild gevent. Report here if it solved the problem or not.

thruflo said, at 2010-05-19T12:53:55.000Z:

Reinstalled libevent 1.4.13-stable, rebuilt gevent from tip against it, also reinstalled greenlet, still get the same error:

(thruflo)terry:/env/thruflo james$ ./bin/thruflo
Traceback (most recent call last):
File "core.pyx", line 136, in gevent.core.__event_handler (gevent/core.c:1181)
File "/env/thruflo/lib/python2.6/site-packages/gevent/socket.py", line 135, in
_wait_helper
current.switch(ev)
AttributeError: 'readwrite_event' object has no attribute 'switch'
Segmentation fault

I get this crash log: http://gist.github.com/406266

N.b. I tried with gevent-0.12.2 and got:

Traceback (most recent call last):
File "./bin/thruflo", line 8, in
load_entry_point('thruflo==0.2', 'console_scripts', 'thruflo')()
File "/env/thruflo/src/thruflo/app.py", line 60, in main
wsgi.WSGIServer(('', 6543 + i), app_factory()).serve_forever()
File "/env/thruflo/lib/python2.6/site-packages/gevent/baseserver.py", line 177, in
serve_forever
self.start()
File "/env/thruflo/lib/python2.6/site-packages/gevent/baseserver.py", line 141, in
start
self.start_accepting()
File "/env/thruflo/lib/python2.6/site-packages/gevent/http.py", line 34, in
start_accepting
self.http = core.http(self._on_request)
self.http = core.http(self._on_request)
TypeError: init() takes exactly 0 positional arguments (1 given)

Not sure if that's relevant. Using the latest gevent made it go away.

Denis.Bilenko said, at 2010-06-04T15:03:10.000Z:

thruflo: Regarding the latest traceback, it looks like you have more than one
installation of gevent conflicting with each other. core.http.init accepts
argument in the current development version but does not in 0.12.2. Somehow the two
got mixed in your system.

Regarding this AttributeError: 'readwrite_event' object has no attribute 'switch'
I'd love to reproduce it but I don't know how.

Denis.Bilenko said, at 2010-06-04T15:23:30.000Z:

could anyone provide the complete minimal example that reproduces the issue?

Denis.Bilenko said, at 2010-06-04T15:51:18.000Z:

Could you also confirm that setting EVENT_NOKQUEUE=1 environment variable "fixes" the issue?

thruflo said, at 2010-06-06T20:21:54.000Z:

EVENT_NOKQUEUE=1 indeed 'fixes' it -- will try to report back more soon, no time now sorry

Denis.Bilenko said, at 2011-04-28T15:54:01.000Z:

What's the status of this on the latest gevent?

Denis.Bilenko said, at 2011-12-20T17:59:46.000Z:

This seems to be

a) related to libevent
b) too old and no longer bothers anyone

So I'm closing it. If someone still has this problem with 1.0, please leave a comment here or create a new issue.

What steps will reproduce the problem?

1. Run a wsgi script to serve a tracking image (1x1 gif)
2. Hit server with full production load (~25 requests/second)
3. Drown in error logs

What is the expected output? What do you see instead?

I expect the small gif to be returned very quickly, but the response time
jumps up to an unacceptable level.

Right at the start of the process:
$ time curl "http://.../tracker.gif"
GIF89a????????!??,D;
real 0m0.055s
user 0m0.003s
sys 0m0.005s

Again at T +1 minute
$ time curl "http://.../tracker.gif"
GIF89a????????!??,D;
real 0m6.366s
user 0m0.003s
sys 0m0.008s

What version of the product are you using? On what operating system?

Gevent 0.12.2
Ubuntu 9.10

Please provide any additional information below.

The service runs fine for like 30 seconds then it just starts spitting out:

"[warn] accept_socket: bad accept: Too many open files"

Which definitely interferes with response time. Is there a way to tweak the
gevent server or libevent to remedy this issue? I believe it occurs with long
lasting http connections not being closed properly.

The server script simply uses the WSGIServer

from gevent.wsgi import WSGIServer
WSGIServer((host, port), application).serve_forever()

Reported by justquick.

earlier comments

justquick said, at 2010-03-19T15:00:02.000Z:

Just tried it again with the most recent tip and the test__http unittest passed fine but the problem still appears after like 30 seconds or so. I am using libevent-1.4-2

Denis.Bilenko said, at 2010-03-25T08:19:39.000Z:

This does not seem like a gevent's issue to me. Instead it's a limitation set by your operating system.

For example, take a look at these articles, where the author mentions how he raised
the relevant OS limits:
http://nichol.as/asynchronous-servers-in-python
http://nichol.as/benchmark-of-python-web-servers

dragonfear said, at 2011-04-15T21:27:52.000Z:

Please re-open this issue. This is not about raising operating system limits at all -- it s a bug in gevent's libevent usage. The relevant HTTP sockets are NEVER closed by gevent after they have been used and the other side has closed the socket as well, therefore this causes a massive fd leak (which is easily hit in about 2 seconds).

Denis.Bilenko said, at 2011-04-15T21:43:35.000Z:

can you give an example script that reproduces it?

justquick said, at 2011-04-18T02:20:53.000Z:

thats not what went wrong on my end. the following OS tweaks fixed the problem for me. try em out

echo “10152 65535″ > /proc/sys/net/ipv4/ip_local_port_range
sysctl -w fs.file-max=128000
sysctl -w net.ipv4.tcp_keepalive_time=300
sysctl -w net.core.somaxconn=250000
sysctl -w net.ipv4.tcp_max_syn_backlog=2500
sysctl -w net.core.netdev_max_backlog=2500
ulimit -n 10240

commands taken from http://nichol.as/benchmark-of-python-web-servers

Denis.Bilenko said, at 2011-12-20T17:57:48.000Z:

OK, so it seems there's no issue with gevent here.

What steps will reproduce the problem?

1. Run the following code:
   import gevent.queue
   q = gevent.queue.JoinableQueue()
   q.join()

What is the expected output? What do you see instead?

I expect q.join() to return immediately since the queue is empty.

Instead, the call blocks forever.

It looks like JoinableQueue creates an Event called _cond that's initially set to False, and waits for it to be True in join(). A fix for this issue might be to call _cond.set() in the JoinableQueue init method.

A patch is attached.

What version of the gevent are you using?

0.13.0
On what operating system?

On what Python?

Which event loop are you running? (Paste the output of python -c "from
gevent.core import *; print get_method(), get_version()")

Please provide any additional information below.

Reported by dmitry.chechik.

earlier comments

dmitry.chechik said, at 2010-09-20T22:24:58.000Z:

Forgot to provide these:

On what operating system?

Ubuntu
On what Python?

Python 2.6

Which event loop are you running? (Paste the output of python -c "from
gevent.core import *; print get_method(), get_version()")
epoll 1.4.13-stable

Denis.Bilenko said, at 2010-09-21T05:31:43.000Z:

You're right with your expectations. I'll look into it.

Denis.Bilenko said, at 2010-09-23T08:12:32.000Z:

Fixed in http://bitbucket.org/denis/gevent/changeset/dcae3cb87688 Thanks!

In case there's no other way to make library asynchronous (e.g. it's a
database driver written in C), thread pools are to the rescue.

Reported by Denis.Bilenko.

earlier comments

uriel.katz said, at 2010-04-29T21:46:21.000Z:

is there any chance this is going to move to gevent from gevent-playground? i want to build a module(async file using threads with monkey patching) that can benefit from having threadpool in gevent so we can include async file also in gevent(which i think is also important).

Denis.Bilenko said, at 2010-06-10T04:24:05.000Z:

In its current form threadpool from the playground cannot be added to gevent because it breaks when monkey patching is enabled. (It uses threading module which is replaced by monkey patching).

[email protected] said, at 2011-05-08T16:34:50.000Z:

It seems to me that the monkey patcher should save pristine copies of any module it patches. The threadpool module could then access that.

Denis.Bilenko said, at 2011-05-08T16:46:27.000Z:

In the trunk, gevent does save the pristine copies of the functions it patches.

However, for example, threading.Thread is not monkey patched, it just becomes "green" because components it relies upon are "green". So there's no original threading.Thread class that you can use.

There's no such problem with 'thread' module, that's why I suggest threadpool should use it.

Denis.Bilenko said, at 2012-01-09T20:10:03.000Z:

the trunk has a thread pool now: https://bitbucket.org/denis/gevent/src/5a431f359ca5/gevent/threadpool.py

Currently, wsgi.WSGIHandler collects the whole response and then sends it
over the network. It should send the data as soon as it gets it from the
wsgi application.

Reported by Denis.Bilenko.

earlier comments

tommi.virtanen said, at 2010-04-14T07:17:57.000Z:

This would be really nice to have.

Bornstub said, at 2010-06-29T15:23:29.000Z:

Agree, due to the PEP333, current WSGIHandler behavior violate what the specification said, the yield or written data should be passed to client immediately.

Denis.Bilenko said, at 2010-07-01T11:22:01.000Z:

We cannot change WSGIHandler to send the data as soon as the application writes it.

We can use libevent's send_reply_chunk() but that will still buffer the data and the buffer can grow indefinitely (there's no way to block when the buffer reaches a certain size with libevent-http)

I'm tempted to close this as out of scope for gevent. This is something that should be fixed in libevent first.

For streaming HTTP server see gevent.pywsgi module and gunicorn.

schmir said, at 2010-07-20T08:24:40.000Z:

Yes, go ahead and close it.

schmir said, at 2011-10-12T20:27:34.000Z:

closing it as wontfix, since gevent.wsgi is completely gone.

Testcase:
from gevent.coros import Semaphore
s = Semaphore(value = 0)
s.acquire(timeout = 0.01)

What is the expected output? What do you see instead?
I expect acquire to return False. Instead it threw an assertion failure (assert self.counter >= 0)

What version of the product are you using? On what operating system?
0.13.0 on Win7 64-bit.

Original code (file coros.py, method acquire):
if ex is not timer:
raise

Proposed patch:
if ex is timer:
return False
raise

Reported by teh.ekik.

earlier comments

Denis.Bilenko said, at 2010-08-17T09:48:19.000Z:

Applied in http://bitbucket.org/denis/gevent/changeset/e91e43554284

Thanks!

teh.ekik said, at 2010-08-17T11:24:55.000Z:

Thank YOU for gevent!

What steps will reproduce the problem?

1. use sys.stdin.readline()

What is the expected output? What do you see instead?
Expected that other greenlet continue to run while readline() blocks.
Instead, the whole process is blocked.

Reported by Denis.Bilenko.

earlier comments

ferenczi.viktor said, at 2011-02-12T15:28:54.000Z:

There is an example here, not exactly for your use case, but very close:

https://bitbucket.org/denis/gevent/src/23fd80c3be00/examples/processes.py

It might be written by you, actually.

Could we use the same fcntl call to make the process' own stdin non-blocking? It would allow for reimplementing readline in a cooperative way.

It won't work on Windows, however, since no way to wait on unnamed pipes without introducing threads. On Windows the only possibility is to use a reader thread for the blocking readline call, which is ugly, but better than nothing.

Denis.Bilenko said, at 2012-07-10T09:39:57.000Z:

You can do it now:

from gevent.fileobject import FileObject

# This makes stdin's FD non-blocking and replaces sys.stdin with
# a wrapper that is integrated into the event loop
sys.stdin = FileObject(sys.stdin)

for line in sys.stdin:
  ...

Only PUT and POST are the HTTP methods for which Content-Length is
mandatory. All other request may skip sending the Content-Length header.
This patch fixes the problem.

Reported by galfyo.pundee.

earlier comments

Denis.Bilenko said, at 2010-07-08T09:10:36.000Z:

Galfy reports that the issue is fixed with the latest pywsgi.

Run httplocalbug.py, load up the server in your browser, and mash the refresh button. I intermittently get a 500 error with "Internal Server Error" in the body, which seems to be coming from an error case in core.c or evhttp.pxi.

Commenting out the line that assigns the http request to local storage fixes the issue. Note that in neither case is the http request read back out of local storage.

gevent-0.13.0-beta1
libevent-1.4.13-stable

Reported by tsuzman.

earlier comments

Denis.Bilenko said, at 2010-06-25T14:09:30.000Z:

I get 200 OK with your test with that line uncommented. Weird.

Could you perhaps try it with a less fancy event loop (EVENT_NOKQUEUE=1, EVENT_NOEPOLL=1) ?

or make sure that you don't have older version of gevent lying around?

tsuzman said, at 2010-06-25T20:12:10.000Z:

It gets stranger. It turns out the issue has nothing to do with gevent.local; I've attached a simpler test case.

The test file has a dict and a list. Adding http_request to just the dict causes intermittent errors. Adding it to the list AND to the dict = no problems. Adding it to just the list = no problems. Adding it to neither = no problems. I'm seeing this reliably -- I've probably tried 30 times. gevent.local was tripping the problem because it puts http_request in a dict.

I've attached a sample http request/response I got from a tcmpdump that errored out. On errors, the server sends both the error response and the correct response, one appended to the other. I've only gotten the error response a couple times when testing with curl, and never with ab; it happens every 1 in 2 to 1 in 50 requests in chrome or firefox, depending on who knows what.

I've been careful about wiping all the old versions of both libraries before installing anything

I tried downloading libevent to 1.4.9; problem was still there.
Tried downgrading gevent to 0.12.2; problem went away entirely. So the problem seems new in gevent 0.13.0-beta1
I'm running on python 2.6.4

I suspect it must be some kind of GC/memory issue. Based on how weird it is, I doubt you'll be able to replicate it easily... any thoughts on changes in 0.13.0-beta1 that could be causing this?

Denis.Bilenko said, at 2010-06-26T08:37:20.000Z:

The issue is originated from libevent-http not implementing pipelining properly. The workaround is to add "Connection: close" header to disable it.

I'll make gevent.http and gevent.wsgi use "connection: close" by default and add a note in the documentation why it should be like that.

Ted, thanks for helping figuring it out.

What steps will reproduce the problem?

1. Run the concurrency.py
2. Run eg. ab -k -v 3 -r -c 1 -n 2 http://127.0.0.1:8080/

What is the expected output? What do you see instead?
Since default headers are connection close, keep-alives does not work.

Maybe default_headers can be moved into pywsgi and set empty in wsgi?

Reported by o.persson.

earlier comments

Denis.Bilenko said, at 2010-07-29T08:47:42.000Z:

Keep alives are disabled in gevent.wsgi because of bugs in libevent-http see http://code.google.com/p/gevent/issues/detail?id=32

You can re-enable them, by setting default_headers to [].

But unless you're sure that the bug that it protect against is fixed in your version of libevent, I wouldn't recommend that.

Modified the wsgiserver.py (attached) to call stop_accepting while we're actually processing a request. Calling that appears to actually throw away the entire libevent http object (which means the request fails).

What version of the product are you using? On what operating system?
gevent-trunk on linux.

I don't think there is a way to implement stop_accepting w/o modifying libevent appropriately.
I've attached a patch for libevent that shutdown the accepting/binding sockets as well as patch for gevent. I have not submitted this to libevent yet (wanted to get feedback here first).

Reported by lukatmyshu.

earlier comments

Denis.Bilenko said, at 2010-06-10T04:21:24.000Z:

For libevent-1.4, you're right there's no way to do it gevent without modifying libevent.

The gevent patches look OK to me, if libevent patch gets accepted we can apply it.

For libevent-2 there's evhttp_del_accept_socket which gevent currently does not use but should. It does what you want.

lukatmyshu said, at 2010-06-10T13:54:38.000Z:

submitted patch to libevent. https://sourceforge.net/tracker/?func=detail&aid=3014372&group_id=50884&atid=461325

Denis.Bilenko said, at 2011-12-20T18:02:28.000Z:

OK, so since this was related to libevent, I'm closing it here.

This bug is a follow-up to issue #26. I sometimes get these errors:

File "/home/cyril/gevent/gevent/socket.py", line 364, in connect
wait_readwrite(sock.fileno(), timeout=timeleft)
File "/home/cyril/gevent/gevent/socket.py", line 192, in wait_readwrite
event = core.readwrite_event(fileno, _wait_helper, timeout, (getcurrent(), timeout_exc))
File "core.pyx", line 363, in gevent.core.readwrite_event.init (gevent/core.c:4203)
File "core.pyx", line 286, in gevent.core.event.add (gevent/core.c:2884)
IOError: event_add(fileno=26) returned -1

There's a race condition in socket.close. If I have 2 sockets, one waiting on read(), the second calling close(). Since changeset 32c26ce69eaf, which fixes issue #26, this should work flawlessly. It does, most of the time.

In socket.close(), we have:

self._sock = _closedsocket()

This means the real socket is freed and can be garbage-collected anytime. If it is indeed garbage-collected before our second greenlet (the one waiting for reading) has a chance to run, Python will call close() on the real socket, which will eventually remove the fd from epoll.

If another (new) socket gets the same fd number as the one we just closed, the IOError is raised. What happens is basically the scenario Denis exhibited in this thread:

http://archives.seul.org/libevent/users/Apr-2010/msg00021.html

As a (dirty) workaround, adding:

self.__sock = self._sock

just before:

self._sock = _closedsocket()

makes the bug goes away, since a reference is kept on the real socket, preventing it from being garbage collected (and closed) until the other greenlet has no reference on the gevent socket.

Reported by cyril.bay.

earlier comments

Denis.Bilenko said, at 2010-07-01T11:05:25.000Z:

Thanks again, Cyril. Here's the fix: http://bitbucket.org/denis/gevent/changeset/12f0230acd03

It's really the same issue as #26 except the patch for #26 did not cover connect() method.

Please confirm that this fixes your app.

cyril.bay said, at 2010-07-01T11:20:05.000Z:

Hello Denis. I've just tested your patch, it doesn't fix it.

Denis.Bilenko said, at 2010-07-01T14:00:26.000Z:

OK, how about this one: http://bitbucket.org/denis/gevent/changeset/3db36f332605 ?

cyril.bay said, at 2010-07-01T14:35:23.000Z:

This latest patch seems to fix the issue. Thanks!

cyril.bay said, at 2010-07-03T23:11:53.000Z:

I've just experienced the bug again on the latest trunk. There seems to be another race condition that triggers it, although it's much rarer than before. Problem is, once it occurs, it will occur indefinitely as a specific fd (26 in my example) is unusable with gevent, but gets reused by the system for new connections.

Denis.Bilenko said, at 2010-07-04T03:37:17.000Z:

Are you sure you're not closing the descriptor bypassing socket's close method? e.g. by doing sock._sock.close(), or os.close(sock.fileno()) If you do that error is unavoidable with epoll method.

If you can provide more details, e.g. traceback, please do.

cyril.bay said, at 2010-07-16T09:09:23.000Z:

I'm sure I'm not bypassing the socket's close method or manipulating self._sock myself. My dirty fix (self.__sock = self._sock) fixes the problem, but I can't figure out why your patch doesn't, despite re-reading gevent's socket.py.

By the way, is your fix supposed to work with SSL sockets too? Because I'm using both SSL and non-SSL sockets in my code.

Denis.Bilenko said, at 2010-07-18T05:57:35.000Z:

Actually no, the fix for sockets does not cover SSLObject. I think they are still susceptible to the problem. At least now we've got a lead to something!

Denis.Bilenko said, at 2010-07-24T22:29:00.000Z:

I'm attaching gevent/ssl.py with a fix. Could try it and confirm that it works?

cyril.bay said, at 2010-07-26T11:23:35.000Z:

Thanks Denis. I'm testing it right now, I'll let you know if the bug happens again or not.

cyril.bay said, at 2010-07-26T14:44:57.000Z:

Here is an error I got:

[...]
File "/home/cyril/gevent-0.13-ssl/gevent/socket.py", line 484, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/home/cyril/gevent-0.13-ssl/gevent/ssl.py", line 202, in send
except error, ex: NameError: global name 'error' is not defined

Denis.Bilenko said, at 2010-07-26T20:41:10.000Z:

oops, here's a fix for that.

cyril.bay said, at 2010-07-27T13:43:49.000Z:

Thanks. Unfortunately, there's another problem: suddenly, my Python process eventually starts using up to 100% CPU. It doesn't start right away: my process (a HTTP proxy) ran fine for 2 hours until it suddenly used up to 100% CPU, and it didn't go down.

I first thought it was a bug in my code, but it did it a second time, I really don't think it's a coincidence. The very same code ran for several weeks without this issue.

I tried profiling the program and I think the problem is related to socket writes. I can't be more specific unfortunately; I really wish I had a test case for that.

cyril.bay said, at 2010-07-28T09:38:59.000Z:

Wrong alert, it happened again without your fix. I'll investigate more and let you know. Sorry about that!

Denis.Bilenko said, at 2010-08-01T23:41:33.000Z:

The last patch also had problems. This one should be alright: http://bitbucket.org/denis/gevent/changeset/b425c83d62d2

Please let me know if it fixes the issue.

cyril.bay said, at 2010-08-02T08:57:16.000Z:

Thanks Denis. I'll try it ASAP and let you know, but I have to figure out my CPU bug first.

cyril.bay said, at 2010-08-14T09:22:55.000Z:

I've tested my application for a few days with the latest gevent trunk (with your incorporated patch), and I'm glad to say the bug is finally fixed. Hurray! Thanks again, Denis.

Denis.Bilenko said, at 2010-08-14T10:06:05.000Z:

Thanks for the report, Cyril!

To be completely clear, 0.13.0 fixed the issue for gevent.socket.socket and upcoming 0.13.1 will fix the issue for gevent.ssl.SSLObject.

There's also gevent.sslold which still has the issue, but I think it's going to be deprecated before 1.0 so there's no point touching it.

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?

What version of the gevent are you using?
0.13

On what operating system?
CentOS 4.4

On what Python?
Python 2.6.4

Which event loop are you running? (Paste the output of python -c "from
gevent.core import *; print get_method(), get_version()")
epoll 1.4.12-stable

Please provide any additional information below.
mimic the StreamServer to provide DataGramServer.

Reported by shenyute.

earlier comments

nagy.attila said, at 2011-10-23T19:37:05.000Z:

It would be nice to have UDP support. Are there any intentions to clear this up and merge it into gevent?

BTW, running the example I get:
File "udp_server_example.py", line 18, in
server = DgramServer(udp_sock, echo)
File "dgram_server.py", line 34, in init
BaseServer.init(self, listener, handle=handle, backlog=backlog, spawn=spawn)
File "build/bdist.freebsd-8.2-STABLE-amd64/egg/gevent/baseserver.py", line 47, in init
AttributeError: can't set attribute

shenyute said, at 2011-11-10T03:35:19.000Z:

http://stackoverflow.com/questions/4183432/cant-set-attribute-with-new-style-properties-in-python

It should work after you change the name of started method like the following:

55 @Property
56 def isStarted(self):
57 return self._recv_event is not None or self._start_receving_timer is not None

Denis.Bilenko said, at 2012-01-06T17:37:27.000Z:

Added DatagramServer here: https://bitbucket.org/denis/gevent/changeset/04b71a2280d8#chg-gevent/server.py

Example: https://bitbucket.org/denis/gevent/src/f70a1cc2e5d3/examples/udp_server.py

Please try it out and let me know if it works for you.

Would be cool to have python3 support, ideally as single source.

Here's an old python3 fork: http://bitbucket.org/jjonte/gevent

Reported by Denis.Bilenko.

earlier comments

Denis.Bilenko said, at 2011-01-28T05:36:07.000Z:

There's some work going on at https://bitbucket.org/Edmund_Ogban/gevent-py3k

[email protected] said, at 2011-03-20T18:40:50.000Z:

Exception handling is a problem here. With Python2.5 there is only except Exception, e, whereas Python3.x requires except Exception as e.

Making this single source supporting 2.5 and higher is therefore quite difficult. Or is there some future that can be used to simulate the new syntax included with the language since 2.6?

Otherwise I would say: drop support for Python 2.5 with the next release version, and make that one compatible with 2.6 .. 3.2.

anacrolix said, at 2011-09-13T00:03:14.000Z:

I'm keen to try gevent, but I sailed for Python3 over a year ago.

anacrolix said, at 2011-09-20T00:23:26.000Z:

Denis can you report on the status of Python3 support for gevent?

Denis.Bilenko said, at 2011-12-20T18:14:35.000Z:

Issue 89 has been merged into this issue.

luchenue said, at 2011-12-23T09:14:52.000Z:

Any news ?

anacrolix said, at 2011-12-23T11:09:10.000Z:

From what I've observed, I don't think Denis wants to break support for Python 2 yet, and so won't accept patches that don't support both 2 *and* 3, which is a complex obstacle.

k.cherkasoff said, at 2011-12-23T11:42:53.000Z:

Is there any up to date forks for python 3.x ?

whitelynx said, at 2012-02-01T07:13:18.000Z:

I think the general approach for maintaining anything that's supposed to work on 2.5 - 3.x is to use the 2to3 tool to automatically generate a python3 version from the python2 source. The exception handling issue mentioned by carsten above (comment 2) is treated by the 'except' fixer in 2to3. (http://docs.python.org/library/2to3.html?highlight=except#2to3fixer-except) Since fixers can be run individually, it shouldn't be too difficult to put together a 2to3 command line that would be able to translate gevent with a minimum amount of breakage, and then just manually clean up the results. I do wish there was a 3to2 or similar that would translate some of the python3-style idioms to something compatible with 2.5, as with the 'except _ as _' syntax.

amcnabb8 said, at 2012-02-09T19:14:00.000Z:

The six library makes it very easy to support both Python 2 (>=2.6) and Python 3 without needing 2to3. With a couple of workarounds, Michael Foord has shown that it's entirely possible to support Python 2.4 without 2to3. But really, it's not as hard as it sounds to support both Python 2 and 3, and there are two reasonable approaches.

What steps will reproduce the problem?

Unknown. It is triggered occasionally by running our test suite.

What is the expected output? What do you see instead?

Traceback (most recent call last):
File "core.pyx", line 152, in gevent.core.__event_handler (gevent/core.c:1204)
File "/Users/aat/Projects/gevent/gevent/socket.py", line 134, in _wait_helper
current, timeout_exc = ev.arg
TypeError: 'NoneType' object is not iterable

This also ends in a segfault...

What version of the product are you using? On what operating system?

tip 1273:881f49ad1b31 on OSX 10.5, with

Please provide any additional information below.

I think this is caused by gevent/socket.py lines 278-279.

Reported by [email protected].

earlier comments

Denis.Bilenko said, at 2010-06-21T02:03:58.000Z:

Could you check if the error goes away when you run it with EVENT_NOKQUEUE=1 env var set?

alecathomas said, at 2010-06-21T03:04:21.000Z:

That does seem to fix the issue, yes. It's hard to be 100% sure as the error is intermittent, but it was occurring fairly frequently prior to setting the envar and I haven't seen it since.

Denis.Bilenko said, at 2010-06-21T03:14:16.000Z:

So it seems this is the same as issue #25 (http://code.google.com/p/gevent/issues/detail?id=25).

I thought it's a kqueue problem, so I tried to trigger this bug on FreeBSD. However, kqueue works fine there. So it's MacOSX-specific, which I cannot test on.

alecathomas said, at 2010-06-21T13:24:50.000Z:

Here is a gdb backtrace from a core dump of one of these segfaults. It may be of help...

(gdb) bt
@0  0x0023e694 in PyString_Format ()
@1  0x001e2392 in binary_op1 ()
@2  0x001e369d in PyNumber_Remainder ()
@3  0x01431bb2 in __pyx_f_6gevent_4core___event_handler ()
@4  0x0144ed2e in event_base_loop ()
#5  0x0144f002 in event_loop ()
#6  0x0144f022 in event_dispatch ()
#7  0x01412261 in __pyx_pf_6gevent_4core_dispatch ()
#8  0x00295494 in PyEval_EvalFrameEx ()
#9  0x002975ed in PyEval_EvalCodeEx ()
#10 0x00214486 in function_call ()
#11 0x001e2ea5 in PyObject_Call ()
#12 0x001f5486 in instancemethod_call ()
#13 0x001e2ea5 in PyObject_Call ()
#14 0x0028f8ce in PyEval_CallObjectWithKeywords ()
#15 0x014ab22a in g_initialstub ()
#16 0x014aa583 in g_switch ()
#17 0x0022afcd in PyCFunction_Call ()
Cannot access memory at address 0x3

alecathomas said, at 2010-06-23T01:15:10.000Z:

Also, we didn't see this issue with gevent 0.12, if that's helpful.

lexasa2904 said, at 2010-08-23T20:15:31.000Z:

I experience same issue with FreeBSD7.3, libevent 1.14b and gevent 0.13

@400000004c72d5f10d22bd54   File "core.pyx", line 152, in gevent.core.__event_handler (gevent/core.c:1307)
@400000004c72d5f10d25323c   File "/usr/local/lib/python2.6/site-packages/gevent/socket.py", line 134, in _wait_helper
@400000004c72d5f10d25a76c     current, timeout_exc = ev.arg
@400000004c72d5f10d25f58c TypeError: 'NoneType' object is not iterable
@400000004c72d5f10d26d04c Failed to execute callback for event fd=96 READ|WRITE flags=INIT
@400000004c72d5f10d26d81c   cb  = 
@400000004c72d5f10d26dfec   arg = None

equand said, at 2010-09-11T22:53:41.000Z:

I experience the same on FreeBSD 8.1

Traceback (most recent call last):
  File "core.pyx", line 152, in gevent.core.__event_handler (gevent/core.c:1307)
  File "/usr/local/lib/python2.6/site-packages/gevent/socket.py", line 137, in _wait_helper
    current, timeout_exc = ev.arg
TypeError: 'NoneType' object is not iterable
Failed to execute callback for event fd=21 READ|WRITE flags=INIT
  cb  = 
  arg = None

snaury said, at 2010-11-11T07:47:15.000Z:

Here's a script that reproduces the problem for me

wolever said, at 2010-11-16T15:45:49.000Z:

I've had this issue too, and @equand's script reproduces it every time. For what it's worth, I'm on OS X 10.6.4, and I started to notice it after I updated Celery to a development release.

snaury said, at 2010-11-16T20:10:41.000Z:

Ok, I think I found what's the issue. The problem is in libevent. :)

When event without EV_PERSIST is added to kqueue, there are actually two events added: one for read and one for write. When write event (connected) hits, kqueue dispatcher sees the event in the udata pointer and removes EVLIST_X_KQINKERNEL flag, even though the read event is still there. Because the flag is cleared kq_del is a no-op and doesn't even bother removing it. My example never reads from the socket (thus it never overwrites the udata pointer), so it seems that it just sits there until socket finally disconnects. Then it seems kernel finally generates read event, for which _rw_event is still registered as the target, and it becomes activated. That's why we see exceptions.

snaury said, at 2010-11-16T20:12:59.000Z:

Btw, if I do sock._rw_event = None just after connect call then I see a marvelous segmentation fault instead of the exception. %)

Denis.Bilenko said, at 2010-11-25T10:07:09.000Z:

snaury, thanks for figuring this out! Could you create an issue in the libevent's bugtracker? http://sourceforge.net/tracker/?group_id=50884&atid=461322

Please post a link to it here as well.

snaury said, at 2010-11-25T19:51:03.000Z:

Denis, here it is: https://sourceforge.net/tracker/?func=detail&aid=3118741&group_id=50884&atid=461322

snaury said, at 2010-11-27T21:37:50.000Z:

Denis, actually, I just realized... Why do you use wait_readwrite in socket.connect, and for that matter _rw_event at all? I've been reading man pages on connect and it seems both FreeBSD, Mac OS X and Linux man pages specify that you should select for writing when connect returns EINPROGRESS. I also looked in eventlet source code and sure enough, it uses trampoline with write=True, no waiting for read.

This bug can be worked around in gevent by simply moving from wait_readwrite and _rw_event to wait_write and _write_event. What do you think?

Denis.Bilenko said, at 2010-11-28T11:56:21.000Z:

If I remember correctly, I made it 'readwrite' rather than 'write' to make it work on Windows.

However, if that's the source of the problem, we can switch it back to 'write' for non-win systems.

Did you verify that using write event instead of read/write fixes the issues?

snaury said, at 2010-11-28T12:43:00.000Z:

Yes, I tested it and my example script no longer raises an exception (not sure about the test suite though, for some reason I have a lot of failures even without my patch). As for Windows, from MSDN (http://msdn.microsoft.com/en-us/library/ms740141(VS.85).aspx):

"If a socket is processing a connect call (nonblocking), a socket is writeable if the connection establishment successfully completes."

So I'm wondering what was the real issue on Windows. Can't test on Windows right now though, would have to wait for Monday when I'm at work...

snaury said, at 2010-11-29T18:12:45.000Z:

Denis, I found why you had to use EV_READ for connect() on Windows. It is yet another libevent 1.4 bug, see https://sourceforge.net/tracker/?func=detail&aid=3122560&group_id=50884&atid=461322

Denis.Bilenko said, at 2011-01-09T13:10:04.000Z:

I've pushed a patch that uses write event instead of read/write on anything except windows: https://bitbucket.org/denis/gevent/changeset/5f128257d02c

Thanks to snaury for figuring it out.

this would greatly help to solve some issues i have with running on pypy and
running inside the glib mainloop

• pypy
• gtk

  Reported by ronny.pfannschmidt.

  ---

  earlier comments

  Denis.Bilenko said, at 2010-02-25T04:00:08.000Z:

  Pluggable reactors would be a tough project as gevent depends on libevent in more areas than just io multiplexing (signal handling, dns resolution, http server).

Also, making a reactor interface that works equally well with python's select module
as well as with event loop written in C is hard. (Eventlet has pyevent hub but it is
disabled by default for a reason.).

gevent has good integration with libevent at the expense of other possible options.
This was intentional design decision.

What about running pygtk loop in another thread? I think it is a viable option,
provided both threads communicate by message passing.

As for PyPy, reimplementing gevent.core in ctypes seems like it would help, given
that PyPy does have ctypes.

ronny.pfannschmidt said, at 2010-02-26T20:27:43.000Z:

unfortunately a multithreaded solution makes it harder to reason about what to pass how

i would prefer just to have a single mainloop

[email protected] said, at 2010-03-04T22:19:19.000Z:

Just something to think about - ctypes often does not compile on non gcc compilers. If you move to use ctypes - you may reduce the number of platforms you can use gevent on. This maybe a mute point, as greenlets themselves are often gcc bound anyway.

Denis.Bilenko said, at 2010-03-04T23:14:57.000Z:

I didn't propose we move to ctypes completely as it'll probably have more overhead than Cython-generated extension. It's just an idea how to make it work at all on something like PyPy.

Denis.Bilenko said, at 2010-04-10T05:57:46.000Z:

BTW, pypy seems to support CPython extensions now: http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html gevent should work here

Denis.Bilenko said, at 2011-04-07T08:44:46.000Z:

gevent no longer relies on libevent1.4 which uses a global shared event loop. Instead the 'loop' is always explicitly referenced and used to create new watchers/events.

Theoretically at least, it is possible to use an alternative event loop implementation, just point gevent.hub.Hub.loop_class to this new class to enable it.

Stackless also provides microthreads for Python, as well as a number of
other features. It's also better maintained than greenlet extension and
could yield better performance. gevent should run on it.

Reported by Denis.Bilenko.

earlier comments

8mayday said, at 2010-02-19T11:33:32.000Z:

What about support for preemptive scheduling with stackless, is it possible? I think some applications can get benefits from that.

Denis.Bilenko said, at 2010-02-19T12:12:08.000Z:

It seems to me that preemptive scheduling would be harder to get right compared to cooperative. I'm also not sure what are the benefits are for an I/O-bound application.

anacrolix said, at 2011-09-20T00:22:27.000Z:

I'm also not sure of the value of preemptive scheduling with microthreads. Why exactly did you opt out of kernel scheduling with greenlet/gevent if you want preemption? :P

hong.minhee said, at 2012-01-05T09:32:09.000Z:

If gevent can run on stackless also it would become more portable. The stackless module is not provided by only Stackless Python but also PyPy.

Denis.Bilenko said, at 2012-01-05T09:33:49.000Z:

@hong.minhee: PyPy also has greenlet module, so it does not matter here.

hong.minhee said, at 2012-01-05T09:35:55.000Z:

There is one I missed: PyPy cannot use Cython yet. It woud anymore portable (for PyPy at least) even if gevent support stackless module. :-(

See http://code.google.com/r/mattgoodall-socket-constants/ (couldn't find
anyway to request a merge).

Reported by matt.goodall.

earlier comments

Denis.Bilenko said, at 2009-12-29T07:32:26.000Z:

Thanks Matt! Fixed.

They can either be compiled on 64bit Windows (maybe via amazon micro instance) or cross-compiled on 32bit Windows via cross-compiled Python (http://docs.python.org/distutils/builtdist.html#cross-compiling-on-windows)

Note, that there are two 64 bit architectures: amd64 and ia64

Reported by Denis.Bilenko.

If I call pool.join(timeout=xxx, raise_error=True) and one of the greenlets times out, I get the following exception:

Traceback (most recent call last):
File "./test_pool.py", line 50, in <module>
p.join(timeout=3, raise_error=True)
File "/usr/local/lib/python2.6/dist-packages/gevent-0.13.0-py2.6-linux-i686.egg/gevent/pool.py", line 101, in join
raise greenlet.exception
TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType

Changing line 101 in pool.py to
raise greenlet._exception
fixes the error.

Reported by qixludo.

earlier comments

Denis.Bilenko said, at 2010-07-29T08:49:20.000Z:

Could you provide the code that produces unexpected results?

[email protected] said, at 2010-08-31T21:53:25.000Z:

I have run into this problem as well. Here's some code that reproduces the problem, and a patch to pool.py that fixes it.

Denis.Bilenko said, at 2010-09-15T15:43:18.000Z:

Thanks for the patches! Applied in the trunk (http://bitbucket.org/denis/gevent/changeset/f9bebeaaecce)

I've seen tlmonkey and tpool by Ralf Schmitt at http://github.com/schmir/gevent/commit/93ede5bae2b3784d051a8367270aa1de2a26c206 and it introduce issue where a call to gevent.shutdown hangs up:

gevent.spawn(lambda: None).join()
print "Shutting down..."
gevent.shutdown()

I commented on that commit about a possible need for tpool.shutdown() function that would cancel the event, but I think better solution might be to add support for callbacks in Hub.shutdown. This way any library that introduces persistent events would have a chance to tear them down. Cancelling SIGINT handler, which is currently hard-coded, can be refactored into such a callback too.

Reported by snaury.

earlier comments

snaury said, at 2010-09-05T21:32:07.000Z:

An example implementation of what I have in mind.

Denis.Bilenko said, at 2010-09-06T07:49:35.000Z:

Why do you need shutdown() in the first place? I've found it to be one of those things I've added for completeness that are never actually used.

Regarding tpool's persistent event, I think hub should provide wakeup() method which tpool would then use instead of it's own event.

Thus, I don't see a use case for a generalization like this.

snaury said, at 2010-09-06T17:02:15.000Z:

Well, initially I though to use it for something like this:

app = App()
app.start()
try:
# work until interrupted
app.join()
except KeyboardInterrupt:
# mark it stopped/send event
app.stop()
# wait until everything stops
gevent.shutdown()

But of course it's possible to rewrite it like this:

app = App()
app.start()
try:
app.join()
except KeyboardInterrupt:
app.stop()
while True:
try:
app.join() # wait until it stops
except KeyboardInterrupt:
print "Already stopping!"

Maybe the second variant is even better as I could add forceful shutdown after lots of Ctrl-C presses (so I don't have to kill a process if something goes really wrong). And no possible hangs because of some unaccounted greenlet doing events.

Denis.Bilenko said, at 2010-09-06T17:08:24.000Z:

Better yet, you can make app.stop() to block until the app is actually stopped.

For explanation about what bufferevent is, read
http://www.wangafu.net/~nickm/libevent-book/Ref6_bufferevent.html

Reported by Denis.Bilenko.

earlier comments

Denis.Bilenko said, at 2011-12-20T17:56:07.000Z:

We have switched to libev 1.0 which does not have bufferevent.

What steps will reproduce the problem?

1. add a host alias to /etc/hosts that contains dots
2. call gevent.socket.getaddrinfo(host alias)

What is the expected output? What do you see instead?

Expected the host to resolve to the ip entered in /etc/hosts
Instead get DNSError(3, 'name does not exist')

The problem is is that gevent.socket.getaddrinfo does not implement parsing
of /etc/hosts at all. Later versions of libevent have async getaddrinfo
that gevent should use (see issue #16), but on the older versions of
libevent we should parse /etc/hosts ourselves.

Reported by Denis.Bilenko.

earlier comments

drprofessor said, at 2010-06-25T23:02:18.000Z:

I made a quick change to dns.py to get this working as it was preventing me from testing an application I'm building that makes requests to a locally run webserver whose host is faked in /etc/hosts. Here it is in case anyone needs a fix before gevent works with libevent 2. I'm sure it has issues as I'm new to Python (for one, it won't work on Windows).

Denis.Bilenko said, at 2011-04-07T08:33:38.000Z:

The trunk now uses c-ares which handles /etc/hosts and /etc/resolv.conf https://bitbucket.org/denis/gevent/changeset/9d3a45d85486

Denis.Bilenko said, at 2011-06-08T10:13:51.000Z:

Issue 87 has been merged into this issue.

Denis.Bilenko said, at 2011-12-29T09:28:53.000Z:

Issue 110 has been merged into this issue.

arab.gawen said, at 2012-04-24T13:54:42.000Z:

Same problem with gevent 0.13.6 (currently last version) installed through PyPI. I think c-ares is not used.

Here's a patch I did to make it work. I modified the comment #2's dns.py

Changelog:

• FIRST check host file, and if still unresvoled, THEN check online
• FIX: if line was malformed (i.e. blank, no IPv4, syntax error, ...), crashed.

  Denis.Bilenko said, at 2012-04-24T14:37:35.000Z:

c-ares is used starting with 1.0 version which is currently in beta.

Please get it from http://code.google.com/p/gevent/downloads/list

arab.gawen said, at 2012-04-24T14:57:32.000Z:

It works with 1.0b2. Thanks !

[email protected] said, at 2012-04-24T16:10:34.000Z:

Understanding that this patch will not be applied I want to comment that not supporting IPv6 in 2012 is not ok. Please stop writing code that uses inet_aton and use inet_pton instead. IPv6 and this API (RFC 2133) have been around for over a decade.

arab.gawen said, at 2012-04-24T16:21:58.000Z:

My bad, I just copied/pasted & fixed the comment #2's dns.py without paying attention of inet_aton and its deprecation. Call it "patch-of-extreme-urgency-that-can-save-you-a-sleepless-night".

C:\Files\Checkouts\gevent\gevent>python wsgi.py
Traceback (most recent call last):
File "wsgi.py", line 4, in <module>
from urllib import unquote
File "C:\Python26\lib\urllib.py", line 26, in <module>
import socket
File "C:\Files\Checkouts\gevent\gevent\socket.py", line 59, in <module>
is_windows = platform.system() == 'Windows'
File "C:\Python26\lib\platform.py", line 1227, in system
return uname()[0]
File "C:\Python26\lib\platform.py", line 1116, in uname
node = _node()
File "C:\Python26\lib\platform.py", line 908, in _node
except socket.error:
AttributeError: 'module' object has no attribute 'error'

Reported by maluke.

earlier comments

Denis.Bilenko said, at 2010-03-14T19:43:13.000Z:

To run gevent.wsgi as a module, do python -m gevent.wsgi

When you run it as gevent/wsgi.py Python adds 'gevent' directory to sys.path, which
makes modules inside gevent package conflict with the modules in stdlib (e.g.
gevent.socket is loaded instead of socket).

What steps will reproduce the problem?
1.
python2.7 test_patched_socket.py --verbose

on Windows

What is the expected output? What do you see instead?
Few test fails because the symbolic fromfd and socketpair are non present in Windows version on Python (and python documentation says they are not present on all platform)

What version of the product are you using? On what operating system?
gevent tip with python 2.6 and pyton 2.7rc1 on windows platform.
on ubuntu the test run successfully

Please provide any additional information below.
monkey.patch_socket should now add unconditionally fromfd and socketpair but should check if this method are present in the python socket module (something similar is already present in eventlet)

The atacched patch run the test successfully both on Windows and on ubuntu

On Windows there is also the problem that WSAStartup in not called and thus the test fails in any case. I patched libevent to call WSAStartup in event_init() and as a conseguence I'm able to run the test. I don't think this is the right solution, probably gevent should initialize the windows socket system if needed

Reported by luigi.pugnetti.

earlier comments

Denis.Bilenko said, at 2010-06-21T12:30:11.000Z:

The socketpair/fromfd issue has been fixed in the trunk for some time now. The WSAStartup not called is also fixed now: http://bitbucket.org/denis/gevent/changeset/f90e4d487093

Please verify. Thanks.

luigi.pugnetti said, at 2010-06-27T17:46:21.000Z:

Hello,

it seems to work with today tip.

Thank you

I have one greenlet blocking on sock.recv() and another one closing this very same socket (with
sock.close()). The first greenlet will block forever on recv() whereas the socket is already
closed. I would expect recv() to return as soon as close() is called.

Moreover, if I want to call sock.recv() on another socket that happens to have the same fd than
the one blocked, I get this exception:

File "core.pyx", line 243, in gevent.core.event.add (gevent/core.c:2414)
IOError: [Errno 2] No such file or directory

The following test case exhibits the problem:

import gevent
from gevent import spawn, socket

def close(sock):
gevent.sleep(1)
sock.shutdown(socket.SHUT_RDWR)
sock.close()
print 'Socket closed.'

def run():
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('www.google.com', 80))
sock.sendall('GET / HTTP/1.1\r\n\r\n')
response = sock.recv(4096)
print 'Response:\n', response
gevent.spawn(close, sock)
s = sock.recv(4096)
# Once the socket is closed, I would expect sock.recv to return immediately.
# Instead, it waits forever.
print 'Received:', s

gevent.spawn(run).join()

I've tested it on Debian Lenny/amd64. Part of the problem is that epoll_wait doesn't return
anything when the socket is closed, but I guess gevent.socket.close may be overriden in that
case to stop any pending operation.

Reported by cyril.bay.

earlier comments

Denis.Bilenko said, at 2010-05-27T13:04:09.000Z:

Thanks for the report. Here's the fix: http://bitbucket.org/denis/gevent/changeset/32c26ce69eaf

I get this error in gunicorn:

[warn] kevent: Bad file descriptor
Restarting gevent.core.dispatch() after an error [2]: [Errno 9] Bad file descriptor

This happen after stopping the StreamServer, a WSGI application continue to run behind due to a time.sleep(). Time is patched to the gevent version.

How to reproduce :

1. gunicorn -k egg:gunicorn#gevent -w 1 longpoll:app
2. Kill -9 workerpid

(step 2 should be done immediatly). How could I make sure all processes are killed in the pool after timeout ?

• benoit

  Reported by bchesneau.

  ---

  earlier comments

  bchesneau said, at 2010-09-01T07:54:24.000Z:

  if I do kill -QUIT, it's worst :

2010-09-01 09:52:07 [478] [INFO] Parent changed, shutting down: <Worker 478>
2010-09-01 09:52:07 [478] [INFO] Worker exiting (pid: 478)
python(478) malloc: *** error for object 0x1011d0aa8: incorrect checksum for freed object - object was probably modified after being freed.

Seem like gevent doesn't detect we quit the current process.

equand said, at 2010-11-01T06:34:09.000Z:

is it gevent or gunicorn problem?

bchesneau said, at 2010-11-01T10:32:04.000Z:

gevent.

Denis.Bilenko said, at 2011-04-28T15:55:46.000Z:

What's the status of this on gevent 0.13.5 / gevent 0.14dev ?

Denis.Bilenko said, at 2011-12-20T18:04:11.000Z:

OK, this probably no longer happens in 1.0. If it is, feel free to leave a comment here or open a new issue.

The 'started' property is only define in StreamServer so it is not possible to derive a new class from BaseServer without declaring this property in the child class.

I'm using gevent version '0.13.0-beta1' from mercurial repo running on linux x86_64.

Reported by noel.cuillandre.

earlier comments

Denis.Bilenko said, at 2010-07-03T16:58:26.000Z:

yes, 'started' property is supposed to be provided by a subclass. BaseServer is an abstract base class in this regard.

Perhaps you should subclass gevent.server.StreamServer instead?

For API discussion it's better to use the google group: http://groups.google.com/group/gevent

Please join!

run threadlocaltest.py to see that the thread doesn't have its own local
data:

$ python2.6 local.py
func's view of localdata.dict: {'x': "hello, hope you can't see this"}

Comment out monkey.patch_thread() to see the correct behavior:
$ python2.6 local.py
func's view of localdata.dict: {}

gevent 0.12.2 + python 2.6 on centos

Reported by tsuzman.

earlier comments

tsuzman said, at 2010-04-30T07:49:42.000Z:

this example (which uses the same patch technique as monkey.patch_thread()) shows the underlying problem on my python 2.6.4 install:

import thread
import _threading_local

thread._local = _threading_local.local

import threading

print "threading.local is", threading.local, "instead of", _threading_local.local

I get:

threading.local is <type 'thread._local'> instead of <class '_threading_local.local'>

I added a "print thread._local" line to the top of my install's threading.py, and it
prints out <type 'thread._local'>, despite the line directly above 'import threading'
that sets thread._local to _threading_local.local. Somehow, threading.py in the
python install is magically finding the original <type 'thread._local'>.

I copied my python's threading.py to mythreading.py and put it in the same directory
as my python test script. If I import mythreading instead of threading, the problem
goes away. I don't know enough about python's import system to debug any further.
Hopefully this problem is only affecting my install. I'm working around it by adding
"threading.local = _threading_local.local" after I import threading.

thewrongboy said, at 2010-05-04T03:53:48.000Z:

Verified that this problem can be simply fixed by adding the following lines to the beginning of your program.

import gevent.monkey
gevent.monkey.patch_all()

import threading
import thread
threading.local = thread._local

thewrongboy said, at 2010-05-04T10:03:04.000Z:

Please note that _threading_local is broken in many Python versions. You may want to apply the patch in the following issue first.

http://bugs.python.org/issue1522237

Denis.Bilenko said, at 2010-05-20T09:35:50.000Z:

Fixed in the trunk.

gevent now has gevent.local module:
http://bitbucket.org/denis/gevent/src/tip/gevent/local.py

gevent.monkey uses it when patching thread and it also patches threading now.

[migrating this from mailing list]

seeing a case where proxies (e.g. akamai) append cookies to
request. for gevent wsgi handlers this effectively overwrites the
cookie header from downstream user-agent since
wsgi.WSGIHandler.prepare_env will build a dictionary for http headers
with keys like "HTTP_<header-name>". in that case last header wins
collision.

i think most headers cannot be combined and so this is fine but other
web frameworks (e.g. Rails) appear to handle cookies in a special way:
merge them.

here are some notes on this:

http://kristol.org/cookie/errata.html#fold

attached is patch against gevent-0.13.0

Reported by [email protected].

earlier comments

Denis.Bilenko said, at 2010-09-15T16:27:20.000Z:

Thanks for the report!

Fixed in the trunkg both for
wsgi: http://bitbucket.org/denis/gevent/changeset/6a0ee4e5c9cb
and pywsgi: http://bitbucket.org/denis/gevent/changeset/b8f8e79304a6

What steps will reproduce the problem?

import gevent
from gevent.socket import *
job = gevent.spawn(getaddrinfo, 'code.google.com', 80, AF_UNSPEC,
SOCK_STREAM, 0, 0)
gevent.joinall([job])
print job.value

Expected output:
[(2, 1, 6, '', ('74.125.53.138', 80)), (2, 1, 6, '', ('74.125.53.139',
80)), (2, 1, 6, '', ('74.125.53.113', 80)), (2, 1, 6, '', ('74.125.53.100',
80)), (2, 1, 6, '', ('74.125.53.101', 80)), (2, 1, 6, '', ('74.125.53.102',
80))]

Actual output:
[]

What version of the product are you using? On what operating system?
gevent 0.12.2
python 2.6.5
Linux 2.6.32 (x86_64)

The standard library socket module expects integer proto but gevent.socket
expects None when the user does not want to specify. This needlessly breaks
some monkey patching that would otherwise work without modification.

Reported by randall.leeds.

earlier comments

randall.leeds said, at 2010-06-02T01:39:44.000Z:

Here's a patch against the development tip that fixes this issue without breaking backwards compatibility with gevent code which passes None.

Denis.Bilenko said, at 2010-06-02T14:06:30.000Z:

Thanks for the patch! Applied.