ethereum / asyncio-run-in-process Goto Github PK

View Code? Open in Web Editor NEW

12.0 8.0 9.0 148 KB

A simple asyncio friendly replacement for multiprocessing to run coroutines in a separate process.

License: MIT License

Shell 2.95% Makefile 1.83% Python 95.22%

asyncio-run-in-process's Introduction

asyncio-run-in-process

Simple asyncio friendly replacement for multiprocessing

Quickstart

pip install asyncio-run-in-process

Developer Setup

If you would like to hack on asyncio-run-in-process, please check out the Snake Charmers Tactical Manual for information on how we do:

Testing
Pull Requests
Code Style
Documentation

Development Environment Setup

You can set up your dev environment with:

git clone [email protected]:ethereum/asyncio-run-in-process.git
cd asyncio-run-in-process
virtualenv -p python3 venv
. venv/bin/activate
pip install -e .[dev]

Testing Setup

During development, you might like to have tests run on every file save.

Show flake8 errors on file change:

# Test flake8
when-changed -v -s -r -1 asyncio_run_in_process/ tests/ -c "clear; flake8 asyncio_run_in_process tests && echo 'flake8 success' || echo 'error'"

Run multi-process tests in one command, but without color:

# in the project root:
pytest --numprocesses=4 --looponfail --maxfail=1
# the same thing, succinctly:
pytest -n 4 -f --maxfail=1

Run in one thread, with color and desktop notifications:

cd venv
ptw --onfail "notify-send -t 5000 'Test failure ⚠⚠⚠⚠⚠' 'python 3 test on asyncio-run-in-process failed'" ../tests ../asyncio_run_in_process

Release setup

For Debian-like systems:

apt install pandoc

To release a new version:

make release bump=$$VERSION_PART_TO_BUMP$$

How to bumpversion

The version format for this repo is {major}.{minor}.{patch} for stable, and {major}.{minor}.{patch}-{stage}.{devnum} for unstable (stage can be alpha or beta).

To issue the next version in line, specify which part to bump, like make release bump=minor or make release bump=devnum. This is typically done from the master branch, except when releasing a beta (in which case the beta is released from master, and the previous stable branch is released from said branch). To include changes made with each release, update "docs/releases.rst" with the changes, and apply commit directly to master before release.

If you are in a beta version, make release bump=stage will switch to a stable.

To issue an unstable version when the current version is stable, specify the new version explicitly, like make release bump="--new-version 4.0.0-alpha.1 devnum"

asyncio-run-in-process's People

Contributors

Stargazers

Watchers

Forkers

pipermerriam ravachol70 metaver5o gsalgado tubbz-alt fetafernanda 0xelemental qblockq pyeth

asyncio-run-in-process's Issues

Consider using a separate session for every child process

Currently, every process we start is on the same process group as the parent process, and that means a Ctrl-C in the terminal will cause a SIGINT to be sent to every process we start, but our parent process also has logic to detect a SIGINT and forward it to every child, so by default every child ends up receiving two SIGINTs, unless one adds a start_new_session=True item to open_in_process()s subprocess_kwargs` dict, like we do in trinity

As Piper suggested, though, we might want to have that as the default behaviour here

Use `loop.connect_read_pipe` and `loop.connect_write_pipe`

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.connect_read_pipe

Currently, the parent process uses blocking IO and asyncio.run_in_executor to read/write to the pipes used to communicate with the child process. It appears that we can use the loop.connect_read_pipe to construct the appropriate StreamReader and StreamWriter instances to make this IO non-blocking and remove the need for a thread based executor.

Unpickleable exceptions raised in subprocess cause us to crash

Custom exceptions whose constructor require multiple arguments cannot be pickled (https://bugs.python.org/issue32696) and that currently cause us to crash

One example of such exception is rlp.exceptions.SerializationError

Some tests fail randomly when using the trio runner

test_open_proc_SIGINT_can_be_ignored, test_open_proc_SIGINT_while_running and test_open_proc_with_trio_KeyboardInterrupt_while_running all fail randomly when using the open_in_process_with_trio() runner. That happens because sometimes the runner raises a RuntimeError like shown below

<function test_open_proc_SIGINT_while_running.<locals>.do_sleep_forever at 0x7f214a0c10d0> raised an unexpected exception: Cancel scope stack corrupted: attempted to exit <trio.CancelScope at 0x7f214a0f6280, active, cancelled> in <Task '__main__._do_async_fn' at 0x7f214a0f28e0> that's still within its child <trio.CancelScope at 0x7f214a0f6460, active, cancelled>

This is probably a bug in your code, that has caused Trio's internal state to
become corrupted. We'll do our best to recover, but from now on there are
no guarantees.

Typically this is caused by one of the following:
  - yielding within a generator or async generator that's opened a cancel
    scope or nursery (unless the generator is a @contextmanager or
    @asynccontextmanager); see https://github.com/python-trio/trio/issues/638
  - manually calling __enter__ or __exit__ on a trio.CancelScope, or
    __aenter__ or __aexit__ on the object returned by trio.open_nursery();
    doing so correctly is difficult and you should use @[async]contextmanager
    instead, or maybe [Async]ExitStack
  - using [Async]ExitStack to interleave the entries/exits of cancel scopes
    and/or nurseries in a way that couldn't be achieved by some nesting of
    'with' and 'async with' blocks
  - using the low-level coroutine object protocol to execute some parts of
    an async function in a different cancel scope/nursery context than
    other parts
If you don't believe you're doing any of these things, please file a bug:
https://github.com/python-trio/trio/issues/new

Unpickleable exceptions raised in subprocess cause us to crash

Custom exceptions whose constructor require multiple arguments cannot be pickled (https://bugs.python.org/issue32696) and that currently cause us to crash

One example of such exception is rlp.exceptions.SerializationError

Intermittent test failures on CI

Got this failure today on master: https://circleci.com/gh/ethereum/asyncio-run-in-process/374

Retried and it passed. Need to investigate

Locally failing tests

What was wrong?

A few tests related to SIGINT and process killing are failing on my machine. I assume this is the cause of ethereum/trinity#1507.

Code that produced the error

The following two test cases are failing:

tests/core/test_open_in_process.py::test_open_proc_SIGINT_can_be_handled FAILED [ 15%]
tests/core/test_open_in_process.py::test_open_proc_SIGINT_can_be_ignored FAILED [ 18%]

Sometimes, they fail immediately, sometimes they block indefinitely and I have to kill them manually.

Full error output

================================================================================== FAILURES ==================================================================================
____________________________________________________________________ test_open_proc_SIGINT_can_be_handled ____________________________________________________________________

    @pytest.mark.asyncio
    async def test_open_proc_SIGINT_can_be_handled():
        async def do_sleep_forever():
            try:
                while True:
                    await asyncio.sleep(0)
            except KeyboardInterrupt:
                return 9999
    
        async with open_in_process(do_sleep_forever) as proc:
            proc.send_signal(signal.SIGINT)
>       assert proc.returncode == 0
E       assert 2 == 0
E         -2
E         +0

do_sleep_forever = <function test_open_proc_SIGINT_can_be_handled.<locals>.do_sleep_forever at 0x7f7cc9b84c10>
proc       = <asyncio_run_in_process.process.Process object at 0x7f7cc9b34430>

tests/core/test_open_in_process.py:58: AssertionError
____________________________________________________________________ test_open_proc_SIGINT_can_be_ignored ____________________________________________________________________

    @pytest.mark.asyncio
    async def test_open_proc_SIGINT_can_be_ignored():
        async def do_sleep_forever():
            try:
                while True:
                    await asyncio.sleep(0)
            except KeyboardInterrupt:
                # silence the first SIGINT
                pass
    
            try:
                while True:
                    await asyncio.sleep(0)
            except KeyboardInterrupt:
                return 9999
    
        async with open_in_process(do_sleep_forever) as proc:
            proc.send_signal(signal.SIGINT)
            await asyncio.sleep(0.01)
>           proc.send_signal(signal.SIGINT)

do_sleep_forever = <function test_open_proc_SIGINT_can_be_ignored.<locals>.do_sleep_forever at 0x7f7cc9b849d0>
proc       = <asyncio_run_in_process.process.Process object at 0x7f7cc9b4d670>

tests/core/test_open_in_process.py:81: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <asyncio_run_in_process.process.Process object at 0x7f7cc9b4d670>, signum = <Signals.SIGINT: 2>

    def send_signal(self, signum: signal.Signals) -> None:
        """
        Issues the provided signal to the process.
        """
>       os.kill(self.pid, signum.value)
E       ProcessLookupError: [Errno 3] No such process

self       = <asyncio_run_in_process.process.Process object at 0x7f7cc9b4d670>
signum     = <Signals.SIGINT: 2>

asyncio_run_in_process/process.py:277: ProcessLookupError

Environment

$ python -m eth_utils

Python version:
3.8.1 (default, Dec 21 2019, 20:57:38) 
[GCC 9.2.0]

Operating System: Linux-4.19.91-1-MANJARO-x86_64-with-glibc2.2.5

pip freeze result:
alabaster==0.7.12
apipkg==1.5
argh==0.26.2
async-generator==1.10
-e [email protected]:ethereum/asyncio-run-in-process.git@74f11319739c64c0fe8ff7c2aebee3ecf5a125e8#egg=asyncio_run_in_process
atomicwrites==1.3.0
attrs==19.3.0
Babel==2.8.0
backcall==0.1.0
bleach==3.1.0
bumpversion==0.5.3
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
cloudpickle==1.2.2
colorama==0.4.3
cryptography==2.8
cytoolz==0.10.1
decorator==4.4.1
docopt==0.6.2
docutils==0.16
eth-hash==0.2.0
eth-typing==2.2.1
eth-utils==1.8.4
execnet==1.7.1
flake8==3.4.1
idna==2.8
imagesize==1.2.0
ipython==7.11.1
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.15.2
jeepney==0.4.2
Jinja2==2.10.3
keyring==21.1.0
MarkupSafe==1.1.1
mccabe==0.6.1
more-itertools==8.1.0
mypy==0.740
mypy-extensions==0.4.3
packaging==20.0
parso==0.5.2
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
pkginfo==1.5.0.1
pluggy==0.13.1
prompt-toolkit==3.0.2
ptyprocess==0.6.0
py==1.8.1
pycodestyle==2.3.1
pycparser==2.19
pydocstyle==3.0.0
pyflakes==1.5.0
Pygments==2.5.2
pyparsing==2.4.6
pytest==5.2.2
pytest-asyncio==0.10.0
pytest-forked==1.1.3
pytest-watch==4.2.0
pytest-xdist==1.31.0
pytz==2019.3
PyYAML==5.3
readme-renderer==24.0
requests==2.22.0
requests-toolbelt==0.9.1
SecretStorage==3.1.2
six==1.14.0
snowballstemmer==2.0.0
Sphinx==1.8.5
sphinx-rtd-theme==0.4.3
sphinxcontrib-websupport==1.1.2
toolz==0.10.0
tox==2.9.1
tqdm==4.41.1
traitlets==4.3.3
twine==3.1.1
typed-ast==1.4.1
typing-extensions==3.7.4.1
urllib3==1.25.8
virtualenv==16.7.9
watchdog==0.9.0
wcwidth==0.1.8
webencodings==0.5.1