Comments (7)
jamadden
commented on June 18, 2024
1
Your little dance with re-initting stuff has never been necessary or accomplished anything of note --- fork() already does that same stuff internally (except for destroying the loop, but if you're doing that, you should probably just be spawning, not forking). It sounds like you just have buggy code that can't deal with a fork. Most people would just fix that bug --- i.e., have your greenlet that's trying to interact with a dead thread check to see if it's still in the same process, and if not, don't do whatever it is you're doing that's crashing.
from gevent.
likecs
commented on June 18, 2024
Gevent version: 1.5.0
Python version: 2.7.18
from gevent import monkey
monkey.patch_all()
import gevent
print(gevent.version_info)
import os
import time
import threading
import traceback
from six import PY3
if PY3:
import _thread as thread
else:
import thread
def reset_coroutine_state():
if gevent.version_info[:3] >= (20, 0, ):
# return
def destroy():
gevent.reinit()
hub = gevent.get_hub()
del hub.threadpool
hub._threadpool = None
gevent.get_hub().loop.destroy()
t = threading.Thread(target=destroy)
t.start()
t.join()
return
else:
gevent.reinit()
hub = gevent.get_hub()
del hub.threadpool
hub._threadpool = None
hub.destroy(destroy_loop=True)
def test():
event = threading.Event()
def alive():
ctr = 0
while ctr < 20:
print("Alive in process: %s, event_set: %s" %
(os.getpid(), event.is_set()))
time.sleep(0.1)
ctr += 1
hub1 = gevent.get_hub()
print("HUB1: %s" % hub1)
thread.start_new_thread(alive)
pid = os.fork()
if pid == 0:
print("Inside child process %s" % os.getpid())
reset_coroutine_state()
hub2 = gevent.get_hub()
print("HUB2: %s" % hub2)
#assert(hub1 != hub2)
print("Waiting for event in child")
event.wait(0.5)
assert(not event.is_set())
event.set()
print("Sleeping for 0.5 seconds")
time.sleep(0.5)
else:
print("Inside parent process %s" % os.getpid())
print("Waiting for event in parent pre-waitpid")
event.wait(1.0)
assert(not event.is_set())
print("Waiting for child to exit in parent")
_, status = os.waitpid(pid, 0)
print("Parent process saw child status: %s" % status)
print("Waiting for event in parent")
event.wait(0.5)
assert(not event.is_set())
test()The output is
version_info(major=1, minor=5, micro=0, releaselevel='dev', serial=0)
HUB1: <Hub '' at 0x7fee8a159940 poll default pending=0 ref=0 thread_ident=0x7fee96c65740>
Inside parent process 67302
Waiting for event in parent pre-waitpid
Inside child process 67305
Alive in process: 67302, event_set: False
HUB2: <Hub '' at 0x7fee8a159b50 poll default pending=0 ref=1 thread_ident=0x7fee96c65740>
Waiting for event in child
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Sleeping for 0.5 seconds
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Waiting for child to exit in parent
Alive in process: 67302, event_set: False
Parent process saw child status: 0
Waiting for event in parent
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: False
Alive in process: 67302, event_set: FalseThis ensures there are no child threads apart from one waiting for the event to be set. If I uncomment assert(hub1 != hub2) it works as well.
But when I run the same script against:
Case 1:
Gevent version: 21.12.0
Python version: 2.7.18
version_info(major=20, minor=0, micro=0, releaselevel='dev', serial=0)
HUB1: <Hub '' at 0x7ff5b079c470 select default pending=0 ref=0 thread_ident=0x20114d280>
Inside parent process 2351
Waiting for event in parent pre-waitpid
Inside child process 2354
Alive in process: 2351, event_set: False
Alive in process: 2354, event_set: False
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774, in __bootstrap
self.__bootstrap_inner()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 849, in __bootstrap_inner
self.__stop()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 864, in __stop
self.__block.notify_all()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 407, in notifyAll
self.notify(len(self.__waiters))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 383, in notify
if not self._is_owned():
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 303, in _is_owned
if self.__lock.acquire(0):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gevent/thread.py", line 141, in acquire
sleep()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gevent/hub.py", line 158, in sleep
loop.run_callback(waiter.switch, None)
File "src/gevent/libev/corecext.pyx", line 722, in gevent.libev.corecext.loop.run_callback
File "src/gevent/libev/corecext.pyx", line 272, in gevent.libev.corecext._check_loop
ValueError: operation on destroyed loop
2023-09-03T05:42:00Z <Greenlet at 0x7ff5b07a65a0: <bound method Thread.__bootstrap of <Thread(Thread-1, stopped 140693204526496)>>> failed with ValueError
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Waiting for child to exit in parent
Parent process saw child status: 11
Waiting for event in parent
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: False
Alive in process: 2351, event_set: FalseCase 2:
Gevent version: 21.12.0
Python version: 3.9.18
version_info(major=20, minor=0, micro=0, releaselevel='dev', serial=0)
HUB1: <Hub '' at 0x1016e7f40 select default pending=0 ref=0 thread_ident=0x1f7d6de00>
Inside parent process 2508
Waiting for event in parent pre-waitpid
Alive in process: 2508, event_set: False
Inside child process 2509
Alive in process: 2509, event_set: False
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 910, in gevent._gevent_cgreenlet.Greenlet.run
File "src/gevent/greenlet.py", line 876, in gevent._gevent_cgreenlet.Greenlet._Greenlet__report_result
File "src/gevent/libev/corecext.pyx", line 722, in gevent.libev.corecext.loop.run_callback
File "src/gevent/libev/corecext.pyx", line 272, in gevent.libev.corecext._check_loop
ValueError: operation on destroyed loop
2023-09-03T05:44:14Z <callback at 0x1018bdec0 stopped> failed with ValueError
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Waiting for child to exit in parent
Parent process saw child status: 11
Waiting for event in parent
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: False
Alive in process: 2508, event_set: FalseThis means the code to destroy the loop is not working with gevent 21.12.0. It seems I am hitting an issue similar to #1350. If I uncomment assert(hub1 != hub2) it doesn't work as well.
It doesn't matter if in reset_coroutine_state I do the destruction in a thread or as is. If no destruction is called in reset_coroutine_state, I get the result as
version_info(major=20, minor=0, micro=0, releaselevel='dev', serial=0)
HUB1: <Hub '' at 0x101e87f40 select default pending=0 ref=0 thread_ident=0x1f7d6de00>
Inside parent process 2577
Waiting for event in parent pre-waitpid
Alive in process: 2577, event_set: False
Inside child process 2578
HUB2: <Hub '' at 0x101e87f40 select default pending=0 ref=1 thread_ident=0x1f7d6de00>
Waiting for event in child
Alive in process: 2578, event_set: False
Alive in process: 2578, event_set: False
Alive in process: 2577, event_set: False
Alive in process: 2578, event_set: False
Alive in process: 2577, event_set: False
Alive in process: 2578, event_set: False
Alive in process: 2577, event_set: False
Alive in process: 2578, event_set: False
Alive in process: 2577, event_set: False
Sleeping for 0.5 seconds
Alive in process: 2578, event_set: True
Alive in process: 2577, event_set: False
Alive in process: 2578, event_set: True
Alive in process: 2577, event_set: False
Alive in process: 2578, event_set: True
Alive in process: 2577, event_set: False
Alive in process: 2578, event_set: True
Alive in process: 2577, event_set: False
Alive in process: 2578, event_set: True
Alive in process: 2577, event_set: False
Waiting for child to exit in parent
Alive in process: 2577, event_set: False
Parent process saw child status: 0
Waiting for event in parent
Alive in process: 2577, event_set: False
Alive in process: 2577, event_set: False
Alive in process: 2577, event_set: False
Alive in process: 2577, event_set: FalseThe result is the same irrespective of python version. So, I am looking for ideas on how to destroy the hub and related greenlets in the child process and ensure no context switching to parent happens from the child process. OR
Is there a way to ensure the parent state (thread for example) is not inherited by the child as seen in the output of 1.5.0 compared to 21.12.0 where Alive message were seen only from parent in former and from both in latter?
from gevent.
likecs
commented on June 18, 2024
@jamadden can you please check this?
from gevent.
likecs
commented on June 18, 2024
I think it is related to this comment in hub.py
.. versionchanged:: 20.5.1
Attempt to ensure that Python stack frames and greenlets referenced by this
hub are cleaned up. This guarantees that switching to the hub again
is not safe after this. (It was never safe, but it's even less safe.)
Note that this only works if the hub is destroyed in the same thread it
is running in. If the hub is destroyed by a different thread
after a ``fork()``, for example, expect some garbage to leak.
from gevent.
jamadden
commented on June 18, 2024
Does gevent monkey patching of os.fork() ensure that child process greenlet and parent process greenlets are separately managed i.e. a greenlet context switch in child process cannot end up context switching to parent process?
Of course. That's what being in a separate process means.
If yes, from which gevent version is this guaranteed?
All of them, always.
Finally, if the gevent hub different for the child and parent process after os.fork()? If no, how if the above issue guaranteed to not happen?
They might appear to have the same address, because fork copies the address space, but because they're in different processes they are of course different objects.
So, I am looking for ideas on how to destroy the hub and related greenlets in the child process and ensure no context switching to parent happens from the child process.
There is never a context switch from a child process to the parent process in the sense that you mean. They are two separate process that have nothing to do with each other, aside from their shared file descriptors.
I stress again that the child's address space is a copy of the parents address space. Meaning that every greenlet object that existed in the parent exists in the child, and will happily run if switched to. That may or may not be what you want, and is one of the reasons that fork isn't generally recommended in multi-threaded or multi-greenlet processes --- forking a miltu-greenlet process is an advanced use case. Either do your forking early, immediately exec after forking, use posix_spawn, or take manual steps to be sure exactly what you want to run.
from gevent.
likecs
commented on June 18, 2024
Thanks for the super quick reply on the weekend.
I understand your response that child entities are different which is seen in the output for the event variables not being updated in the parent when it was updated in the child process.
Let me explain my problem in a bit more detail to see if I should be worried or what workaround I should use.
In my process, I am forking a cython pthread for some keepalive to ensure it is not impacted by gevent scheduling or due to any buggy greenlet consuming CPU for too long in the system where keepalive is not sent on time to the other side. The keepalive sending and receiving the response is done on the cython pthread side but I have Greenlet running in the process to update the responses from the keepalive ping-pongs. If a do a fork in such a process, the fork happens from the python interpreter and has per posix(2) policy the main thread is copied in to the child. So, in this case, as you mentioned (and we see in the output), all the greenlets are seen in the child but with a different address space but the cython pthread wouldn't be part of the child. Now the greenlet in the parent which was handling the keepalive counters logic is seen in child as well and can end up reading something that was meant for the parent process and not the child process and thus lead to panics/segfaults.
I think it is related to this comment in hub.py
Before 20.5.1, LoopExit error wasn't seen and the below workaround worked
gevent.reinit()
hub = gevent.get_hub()
del hub.threadpool
hub._threadpool = None
hub.destroy(destroy_loop=True)which I have shown in my example where in the child process, I don't see any alive() thread running at all. So, I am looking at possible solutions for gevent version >= 20.5.1.
from gevent.
likecs
commented on June 18, 2024
Thanks, let me close the issue for now. Will reach if I have further questions.
from gevent.
Related Issues (20)
- ssl.SSLError: [SSL: BAD_LENGTH] bad length (_ssl.c:2423)
- Fail to install v23.9.0 - tries to build from source instead of pre-compiled wheel HOT 6
- Missing manylinux2014 wheels for py3.8/py3.9? HOT 3
- 23.9.1: env name py311-cffi conflicting with base python python2.7 HOT 1
- No wheels for new version on python 3.10 docker image HOT 1
- gevent.pywsgi fails to inform which random port it uses when given port 0
- AbstractLinkable._notify_links under VS Code debugger HOT 2
- Wheels for Python 3.9 - ARM64 missing for new releases HOT 2
- this product dont works with flask 3.0 greenlet 3.0 on python 3.12 HOT 1
- How to disable verbose output of the monitoring thread? HOT 1
- client hang with gevent time out HOT 1
- gevent.exceptions.InvalidSwitchError: Invalid switch into AsyncResult.wait(): None HOT 1
- gevent.timeout.Timeout: 0.0 seconds HOT 3
- Assertion failure in _sempahore.py
- AttributeError: module 'select' has no attribute 'epoll' HOT 2
- TypeError: memoryview: a bytes-like object is required, not 'str' HOT 1
- gevent.hub.LoopExit: This operation would block forever HOT 3
- Bad use of gevent: the Thread has created before patch_all, is_alive() always True HOT 3
- Test units with 0 ran tests (skipped) exit with code 5 on Python 3.12 HOT 7
- Logging lock not released when held by OS threads HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gevent.