Comments (2)
goodboy
commented on May 24, 2024
This may have something to do with a race when using async generators wrapped by a portal where the far end actor errors and closes the connection, the channel get's de-allocated by the current actor, and the generator is just left hanging waiting on an already closed connection to recv data because the queue / consumer side hasn't been alerted of the break. I'm not sure why there isn't an error getting raised in the msg loop but it's probably something stupid.
from tractor.
goodboy
commented on May 24, 2024
Well here's the second issue in more detail with logging from piker:
ov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:345 Received msg {'functype': 'asyncgen', 'cid': 'f6fbcdaa-f3de-11e8-bed6-a402b9cc051a'} from ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (data_feed: MainThread) [INFO] piker.broker.data data.py:236 <Channel fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35866), raddr=('127.0.0.1', 1616)> changed symbol subscription to ()
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:305 Getting result queue for ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a') cid f6fbcdaa-f3de-11e8-bed6-a402b9cc051a
Nov 29 08:59:18 (data_feed: MainThread) [INFO] piker.broker.data data.py:275 Subscribing with existing `questrade` daemon
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:296 Delivering {'functype': 'asyncgen', 'cid': 'f6fbcdaa-f3de-11e8-bed6-a402b9cc051a'} from ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a') to caller f6fbcdaa-f3de-11e8-bed6-a402b9cc051a
Nov 29 08:59:18 (data_feed: MainThread) [INFO] tractor _actor.py:394 RPC func is <function modify_quote_stream at 0x7f8db024a9d8>
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:351 Waiting on next msg for <Channel fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35866)> from ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (data_feed: MainThread) [DEBUG] tractor _actor.py:399 Waiting on next msg for <Channel fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35866), raddr=('127.0.0.1', 1616)> from ('arbiter', 'f5c43170-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _trionics.py:279 Cancelling nursery
Nov 29 08:59:18 (data_feed: MainThread) [DEBUG] tractor _actor.py:416 Exiting msg loop for <Channel fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35866), raddr=('127.0.0.1', 1616)> from ('arbiter', 'f5c43170-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [WARNING] tractor _portal.py:206 Sending cancel request to ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a') on <Channel fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35866)>
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:305 Getting result queue for ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a') cid f6fc39d4-f3de-11e8-bed6-a402b9cc051a
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:319 Sending cmd to ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a'): self.cancel({})
Nov 29 08:59:18 (data_feed: MainThread) [INFO] piker.broker.data data.py:373 No more subscriptions for questrade
Nov 29 08:59:18 (data_feed: MainThread) [DEBUG] piker.broker-config config.py:33 Writing config file /home/tyler/.config/piker/brokers.ini
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:345 Received msg {'error': {'tb_str': 'Traceback (most recent call last):\n File "/home/tyler/repos/tractor/tractor/_actor.py", line 500, in _async_main\n log.debug("Waiting on root nursery to complete")\n File "/home/tyler/.local/share/virtualenvs/piker-PaB62peT/lib/python3.7/site-packages/trio/_core/_run.py", line 397, in __aexit__\n raise combined_error_from_nursery\n File "/home/tyler/repos/tractor/tractor/_actor.py", line 140, in _invoke\n tasks.remove((cs, func))\nValueError: list.remove(x): x not in list\n', 'type_str': 'ValueError'}} from ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [ERROR] tractor _actor.py:409 Actor errored:
Traceback (most recent call last):
File "/home/tyler/repos/tractor/tractor/_actor.py", line 356, in _process_messages
ns, funcname, kwargs, actorid, cid = msg['cmd']
KeyError: 'cmd'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tyler/repos/tractor/tractor/_actor.py", line 366, in _process_messages
f"{chan.uid}\n" + tb_str,
TypeError: can only concatenate str (not "NoneType") to str
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:416 Exiting msg loop for <Channel fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35866)> from ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:272 Releasing channel <Channel fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35866)> from ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:276 No more channels for ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:279 Peers is defaultdict(<class 'list'>, {})
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:283 Signalling no more peer channels
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:287 Disconnecting channel <Channel fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35866)>
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor.ipc _ipc.py:136 Closing <Channel fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35866)>
Nov 29 08:59:18 (MainProcess: MainThread) [INFO] tractor _actor.py:238 New connection to us <Channel fd=22, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35876)>
Nov 29 08:59:18 (data_feed: MainThread) [INFO] tractor _portal.py:45 Handshake with actor ('arbiter', 'f5c43170-f3de-11e8-bed6-a402b9cc051a')@('127.0.0.1', 1616) complete
Nov 29 08:59:18 (MainProcess: MainThread) [INFO] tractor _portal.py:45 Handshake with actor ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')@('127.0.0.1', 35876) complete
Nov 29 08:59:18 (data_feed: MainThread) [DEBUG] tractor _actor.py:305 Getting result queue for ('arbiter', 'f5c43170-f3de-11e8-bed6-a402b9cc051a') cid f6fe5cdc-f3de-11e8-a717-a402b9cc051a
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:262 Registered <Channel fd=22, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35876)> for ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:332 Entering msg loop for <Channel fd=22, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35876)> from ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (data_feed: MainThread) [DEBUG] tractor _actor.py:319 Sending cmd to ('arbiter', 'f5c43170-f3de-11e8-bed6-a402b9cc051a'): self.unregister_actor({'uid': ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')})
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:416 Exiting msg loop for <Channel fd=22, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35876)> from ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (data_feed: MainThread) [DEBUG] tractor _actor.py:332 Entering msg loop for <Channel fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35876), raddr=('127.0.0.1', 1616)> from ('arbiter', 'f5c43170-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:272 Releasing channel <Channel fd=22, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35876)> from ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:276 No more channels for ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:279 Peers is defaultdict(<class 'list'>, {})
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:283 Signalling no more peer channels
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:287 Disconnecting channel <Channel fd=22, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 1616), raddr=('127.0.0.1', 35876)>
Nov 29 08:59:18 (data_feed: MainThread) [ERROR] tractor.ipc _ipc.py:34 Stream connection ('127.0.0.1', 1616) broke
Nov 29 08:59:18 (data_feed: MainThread) [DEBUG] tractor.ipc _ipc.py:136 Closing <Channel fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35876)>
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:537 All peer channels are complete
Nov 29 08:59:18 (data_feed: MainThread) [DEBUG] tractor _actor.py:402 <Channel [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6> from ('arbiter', 'f5c43170-f3de-11e8-bed6-a402b9cc051a') disconnected
Nov 29 08:59:18 (data_feed: MainThread) [DEBUG] tractor _actor.py:416 Exiting msg loop for <Channel [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6> from ('arbiter', 'f5c43170-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:619 Shutting down channel server
Nov 29 08:59:18 (MainProcess: MainThread) [WARNING] tractor _portal.py:219 May have failed to cancel ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a')
Nov 29 08:59:18 (MainProcess: MainThread) [ERROR] tractor _trionics.py:326 Nursery for ('arbiter', 'f5c43170-f3de-11e8-bed6-a402b9cc051a') errored with <class 'trio.MultiError'>,
Traceback (most recent call last):
File "/home/tyler/repos/tractor/tractor/_trionics.py", line 357, in open_nursery
yield nursery
File "/home/tyler/repos/piker/tests/test_questrade.py", line 241, in test_option_streaming
await nursery.cancel()
File "/home/tyler/repos/tractor/tractor/_trionics.py", line 304, in cancel
n.start_soon(portal.cancel_actor)
File "/home/tyler/.local/share/virtualenvs/piker-PaB62peT/lib/python3.7/site-packages/trio/_core/_run.py", line 397, in __aexit__
raise combined_error_from_nursery
trio.MultiError: Cancelled(), Cancelled()
Details of embedded exception 1:
Traceback (most recent call last):
File "/home/tyler/.local/share/virtualenvs/piker-PaB62peT/lib/python3.7/site-packages/trio/_core/_run.py", line 632, in raise_cancel
raise exc
trio.Cancelled
Details of embedded exception 2:
Traceback (most recent call last):
File "/home/tyler/.local/share/virtualenvs/piker-PaB62peT/lib/python3.7/site-packages/trio/_core/_run.py", line 632, in raise_cancel
raise exc
trio.Cancelled
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _trionics.py:279 Cancelling nursery
Nov 29 08:59:18 (MainProcess: MainThread) [WARNING] tractor _portal.py:206 Sending cancel request to ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a') on <Channel [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:305 Getting result queue for ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a') cid f70cd776-f3de-11e8-bed6-a402b9cc051a
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _actor.py:319 Sending cmd to ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a'): self.cancel({})
Nov 29 08:59:18 (MainProcess: MainThread) [WARNING] tractor _portal.py:216 <Channel [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6> for ('data_feed', 'f5c4a6a0-f3de-11e8-bed6-a402b9cc051a') was already closed?
Nov 29 08:59:18 (MainProcess: MainThread) [DEBUG] tractor _trionics.py:224 Waiting on all subactors to complete
``
from tractor.
Related Issues (20)
- We might not need `ActorNursery._join_procs: trio.Event` in the long run
- Use of `pydantic` and `click` in sub-actor causes SIGINT ignoring, hangs...
- Macos error while trying to initialize HOT 2
- Need to handle a `KeyError` in `_debug._hijack_stdin_for_child` HOT 2
- Drop `msgpack` dependency, go full `msgspec`
- Handling >= 1 depth (nested) actor trees in the debugger
- `pdbpp` borks our `0.1.0a5` on `pypi` 😂
- Recursion error on `BroadcastReceiver.receive()`?
- `trio.Process` deprecations
- `trio.MultiError` is deprecated -> move to `exceptiongroup.ExceptionGroup` HOT 1
- Share memory (array) sub-system API and optional tight integration with `numpy`
- OS X: shared memory file name length limitation causing error
- `OSError: [Errno 9] Bad file descriptor` in cluster test
- `rust` tasks
- `QUIC` transport research & planning
- `infected_asyncio` task hang (`SIGINT` ignored!) on debug mode breakpointing?
- An actor-runtime equivalent of `trio`'s "guest mode"
- Hot-code-reload: can we do it `git` driven?
- `ray` interop?
- Switching to pdbpp is making it impossible to install with pypy and git HOT 13
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tractor.