Hi, thanks for a very cool looking library. I'm eager to give it a shot.
I'm trying to expose a SWIG-generated object (of type SwigPyObjectType)... and running into issues during api.expose when pyro tries to assign various attributes on the members of the class:

File "~/miniconda3/envs/napdev/lib/python3.9/site-packages/Pyro5/server.py", line 102, in expose
  thing._pyroExposed = True
AttributeError: 'method_descriptor' object has no attribute '_pyroExposed'

I assume this is somewhat specific to SWIG-generated objects, but I assume this "has no attribute" challenge has probably come up in other contexts. I'm wondering if you have any suggestions for a workaround?

If you want to try the exact object I'm trying to expose: pip install pymmcore

import Pyro5.api
from pymmcore import CMMCore as _CMMCore
CMMCore = Pyro5.api.expose(_CMMCore)

thanks!

biggest part of the docs is still based / copied from the pyro4 version

Hi Irmen,

I have a diff/patch (not that big) in my hand that allows me to use pyro5 exactly as I am using pyro4. That is totally transparently from client(s) to server(s) side (and vice-versa). Also allowing any python object to be passed along the wire (thx to pickle and yes I know about its security impact, this is for controlled/restricted servers&clients).

Would you be open to such a change in Pyro5 ?

The diff/patch runs ok the current entire test suite. Both of Pyro5 and of one of my project which is using Pyro4 actually ; but I'd like to eventually update to Pyro5 thus. Hence this issue..

Basically I've reintroduced the PickleSerializer and had to make a few changes here and there to make all the test suites to pass..

I'm trying to use PyKeePass remotely to keep a password database open in a background process for a few minutes.

Here are my server and client:

import Pyro5.api
from pykeepass import PyKeePass as PyKeePassRemote

PyKeePass = Pyro5.api.expose(PyKeePassRemote)

@Pyro5.api.expose
class Factory(object):
    def PyKeePass(self, filename, password=None, keyfile=None):
        kp = PyKeePass(filename, password=password, keyfile=keyfile)
        self._pyroDaemon.register(kp)
        return kp

daemon = Pyro5.api.Daemon.serveSimple(
    {
        Factory: 'Factory'
    },
    host='localhost',
    port=9999,
    ns=False,
)

import Pyro5.api
from lxml import etree

Factory = Pyro5.api.Proxy('PYRO:Factory@localhost:9999')

kp = Factory.PyKeePass('test3.kdbx', 'password', 'test3.key')
kp.entries

Under the hood, kp.entries builds a list of Entry, Group, or Attachment objects.

However, I get TypeError: don't know how to serialize class <class 'lxml.etree._Element'> because Pyro tries to serialize these objects which contain some lxml _Element and _ElementTree objects.

In general, is there a way to tell Pyro5 to automatically pass objects of a certain type as remote objects rather than building adapters around every function in PyKeePass?

Hi, what is the chance that in the future PyRO might follow versioning that is decoupled from the name? I am currently upgrading to Pyro5 and I have to rename numerous module imports, docstring Pyro4.Proxy class descriptions, and others while in practice upgrading even to a major version could be as simple as handling simply the dependencies without having to touch code.

We have a program that starts multiple Pyro daemons and we would like to have different configurations for each daemon. This was not possible on Pyro4 and I was hoping it would be a thing for Pyro5 (seems not).

The way I would envision this is to have a new configargument to the Daemon constructor which would default to a configuration.Configuration instance. It seems that in Pyro5 there is not longer a Configuration class like there was in Pyro4 so not sure how to go about doing that.

Not sure how to go about this and if this could be fitted in the vision for Pyro5.

Is it possible to connect to server via jump host by sshtunnel?

Server sockets are created with noinherit, should client sockets have this too?
Does this avoid proxy connection issues with forking/ multiprocessing?

Hey, I'm wondering if I there is any plan to support exposing async coroutines?
so that I can do something like this:

@expose
class Thingy(object):
   async def message(self, arg):
        print("Message received:", arg)
        return "Roger!"

I think it would fit nicely with Pyro5 being Python 3.5+ and all.

When a oneway call is followed immediately by _pyroRelease, the server method never gets called. This only occurs with communication over a unix domain socket and seems to be a regression from v5.8 to v5.9. This is an issue for instance when using the Proxy as a context manager when the last call is a oneway call before leaving the context.

An example to reproduce this, adapted from examples/oneway/server2:

Server:

import time
import threading
from Pyro5.api import expose, oneway, behavior, serve, Daemon


@expose
@behavior("single")
class Server(object):
    def __init__(self):
        self.counter = 0

    @oneway
    def increment_oneway(self):
        print("oneway call executing in thread", threading.get_ident())
        time.sleep(0.5)
        self.counter += 1

    def increment(self):
        time.sleep(0.5)
        self.counter += 1

    def getcount(self):
        return self.counter


with Daemon(unixsocket='/Users/samschott/Desktop/example.sock') as daemon:
    daemon.register(Server, "example")
    daemon.requestLoop()

Client:

This won't work:

from Pyro5.api import Proxy

with Proxy("PYRO:example@./u:/Users/samschott/Desktop/example.sock") as serv:
    serv.increment_oneway()

But giving it some time does work:

import time
from Pyro5.api import Proxy

with Proxy("PYRO:example@./u:/Users/samschott/Desktop/example.sock") as serv:
    serv.increment_oneway()
    time.sleep(0.1)

Pyro4 page states: "Pyro5: https://pyro5.readthedocs.io New code should strongly consider using Pyro5 unless a feature of Pyro4 is strictly required" but I can't seem to find a list of the differences or advantages.

It's non-trivial to upgrade (at the very least in terms of project planning and developer time) so we need some way to evaluate whether that is worth the upgrade at this point. Our code is not complete, so we need to do new development, but we do have quite a bit existing with Pyro4 already.

I meet quite often the need to register a non-permanent object in a daemon, and have to remember to unregister it later at some point. This cannot be done in the destructor, as the daemon holding the object will prevent its reference count from dropping to zero, thus a separate bookkeeping has to be done.

There is the weakref module just for that:

A primary use for weak references is to implement caches or mappings holding large objects, where it’s desired that a large object not be kept alive solely because it appears in a cache or mapping.

I can imagine Daemon.register(...,weak=False) where weak=True would call something like weakref.finalize(obj,self,unregister(obj)) after having registered the weakref.ref(obj), plus an extra typecheck in request handling (check the weakref is not stale and use the object referred).

Do you see this as something potentially useful?

In rare cases, an unexpected error is raised when a Proxy instance gets deleted / garbage collected:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/Pyro5/client.py", line 80, in __del__
AttributeError: 'NoneType' object has no attribute 'suppress'

The relevant line in Pyro4.client.Proxy is:

with contextlib.suppress(Exception):
    self._pyroRelease()

This issue has been introduced with 217af7b and seems to only occur when exiting the interpreter.

I have use serpent serializer. Datetime is correctly serialized on server side, but on client side it is string with iso datetime format. Is it possible to have it automatically deserialized?

From my pov, it's not a big deal to add a new custom object to be serialized/deserialized by Serpent. But it will be helpful.

This is a question for the community. I have implemented a client/server pair with callback in the manner suggested by Irmen. (Client is itself a server for the callbacks.) The server is experiencing occasional delays of a few tenths of seconds. The server has five threads, four in parallel and one collecting the results to be returned via the callback. All threads experience these delays, so that ms calculations take a few tenths of a second, roughly one time in ten. Could Pyro5 be causing this? Thanks.

How to replicate?
connect to a bunch of servers of different variable name, put them in a list, then reference the list by poping the element then putting them back to the list.

I suggest some modifications on pyro5 httpgateway in order to handle CORS Browsers Security Policy
Http Method "OPTIONS" should be handled with following includings in response header

('Access-Control-Allow-Origin', pyro_app.cors),
('Access-Control-Allow-Methods', 'GET, POST, OPTIONS'),
('Access-Control-Allow-Headers', 'Content-Type')

pyro_app.cors will contain a parameter we could provide on a new launcher -C option.
could be * to authorize every domain ou your specific Domain/URL

Once done, it's easy to have a Vuejs App calling to gateway to interact with pyro services proxies.

Tuple sent over the wire will be deserialized as list; this is a "feature" of msgpack. Other projects were facing this as well, e.g. dask/distributed#3716 (comment) . Their idea is to add a special tag to tuples so that they can be reconstructed properly. Could something like this be done in Pyro as well?

Hi Irmen, i want to start learning pyro5, but the docs not enough clearly. What can i learn pyro5? give me some advice.

This bit:

will not call special serializers created via Pyro5.api.register_class_to_dict. I sense there must be some reason for that. What is it?

Context: I am hitting a case where Pyro5 creates type replacement for an exposed class (autoproxying). When another instance of that class is not registered, the object itself is returned. Now, in such case, I would to use register_class_to_dict to do automatic daemon registration, but it is not getting called as ser_default_class is called directly.

Is it possible to use 2-way SSL when also using a nameserver to look up Pyro objects on other remote computers? I've been having a lot of trouble with the self-signed certificates, since setting config.SSL_CACERTS to a directory to allow for multiple acceptable client certificates doesn't seem to work for me. Perhaps I'm not storing the certificates correctly in that directory, I'm saving them as <HASH-VALUE>.0, but Pyro doesn't seem to like using directories for CA_CERTS?

One of the biggest problems is that I use a certificate for the nameserver, a certificate for the object server, and a certificate for the client, but the object server has to authenticate the certificate for both the nameserver (when it's trying to register the Pyro object) using locate_ns() as well as the client when it tries to connect (and they're different certificates). And it doesn't seem right to use just the same single certificate between all three computers.

Is this a scenario you have encountered before, and can Pyro5 be configured to handle it?

Hello,

Installing Pyro5 from via pip for python3.8 seems to be broken. Here's what's happening:

Installing pyro:

$ pip3 install Pyro5
Collecting Pyro5
  Using cached Pyro5-5.11-py3-none-any.whl (77 kB)
Requirement already satisfied: serpent>=1.27 in ./.local/lib/python3.8/site-packages (from Pyro5) (1.30.2)
Installing collected packages: Pyro5
Successfully installed Pyro5-5.11

Launching a python shell and importing the server module:

$ python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import Pyro5.server
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bla/.local/lib/python3.8/site-packages/Pyro5/server.py", line 20, in <module>
    from . import config, core, errors, serializers, socketutil, protocol, client
  File "/home/bla/.local/lib/python3.8/site-packages/Pyro5/core.py", line 15, in <module>
    from . import config, errors, socketutil, serializers, nameserver
  File "/home/bla/.local/lib/python3.8/site-packages/Pyro5/nameserver.py", line 269, in <module>
    @server.expose
AttributeError: partially initialized module 'Pyro5.server' has no attribute 'expose' (most likely due to a circular import)
>>> 

The docs state that a pip install should just work, so I am not sure how to continue here. The OS is Ubuntu20.04.

I would be grateful for any ideas. It could very well be that the system is configured incorrectly somehow, and Pyro5 is not at fault.

Is it possible to see in logs every function call - method name and parameters?

Pyro5.errors.SerializeError: don't know how to serialize class <class 'datetime.datetime'> using serializer MarshalSerializer. Give it vars() or an appropriate getstate

(occurs in the timezones example)

• add unit test about proxy thread ownership transfer (test_server.py)
• what else?
• docs?

perhaps it is useful to have multi-daemon multiplexing on a single server socket.

This makes it easier to NAT forward a single port to the multiplexer that than fans out to multiple daemons.

• separate multiplexer that just deals with low level socket data? Or solve this on the Pyro level inside a daemon, to which we may attach other daemons?
• attached daemons could register their set of objects with the main daemon? They should forward register/remove calls as well

Is it possible to use the SerializedBlob (or some other construct) to suppress serialization on a certain return statement? We would like to send a plain bytestring which is efficiently pre-serialized in a custom format directly via return from an RPC to a daemon.

also: is there a simple way to perform a reflexive rpc call? Example:

#(Proxy side)
def do_something_onproxy():
    ...
    return b'proxy says hello'

server_msg=server.do_something()
print(server_msg)

>>server says hello

#(Server side)
def do_something_onserver():
    ...
    proxy_msg=proxy.do_something()
    return b'server says hello'
    
print(proxy_msg)
>>proxy says hello

I looked into the callback example but it left me a tad confused...

Thanks and congratulations for your great work on this project!

Would be great to have some points in manual, describing ways to shutdown server without interrupting running processes and preventing new calls to start running.

Thanks for this amazing library.

I have the following code on the server:

from Pyro5.server import serve, expose
import time

@expose
class Server:
    def foo(self):
        time.sleep(5)

    def ping(self):
        print('Pong')

serve(
    { Server: 'server'},
    port=1234,
    use_ns=False
)

And on the client:

from Pyro5.api import Proxy

uri = 'PYRO:[email protected]:1234'
srv = Proxy(uri)

srv.ping()
print(f'Status of connection = {srv._pyroConnection}')

try:
    srv.foo() # do ctrl-c here
except KeyboardInterrupt:
    print(f'Status of connection = {srv._pyroConnection}')

I run this, and then type ctrl-c on the call of srv.foo() (on the client). The output on the client is:

Status of connection = <Pyro5.socketutil.SocketConnection object at 0x7f294d78d160>
Status of connection = None

The question is, why does the proxy loses the connection when the KeyboardInterrupt exception happens?

travis ci has stopped working https://travis-ci.org/github/irmen/Pyro5
move it over to github actions

There is a bug in the register_class_to_dict and register_dict_to_class functionality. I don’t know if the problem is with Pyro5 or serpent. When I registered a base class, followed by the subclasses, it didn’t work properly. Only the base class dict_to_class function got called. When I registered the base class after the subclasses, it worked properly.

Bytes is basic python type but when I call pyro object with bytes parameter on server side it si dict. How to get correct deserialization atomaticaly?

Some special methods, such as __len__ or __getitem__ do not go through __getattr__ when called via their special syntax: len(a) will not look for __len__ using __getattr__ (but a.__len__ will use __getattr__), just like a[0] (vs. a.__getitem__) .

Python documentation on Special method lookup says in the last paragraph:

Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).

Pyro5 documentation says:

You can expose a ‘dunder’ method with double underscore for example __len__.

which however does not work as expected — the special syntax such as len(a) will not work.

The solution would be to define those few special methods on the Proxy class itself, and have them raise an exception if they are not really exposed by the proxy.

A case in the point is here which uses the workaround of explicit __getitem__:

self.assertEqual(root.__getitem__(0).getMolecules().__getitem__(0).getAtoms().__getitem__(0).getIdentity().getElement(),'Q')

I'm on windows/py3.7/Pyro5 v5.10

The filetransfer server fails with

File "server.py", line 79, in __init__
host, _ = self.transportServer.sock.getsockname()
ValueError: too many values to unpack (expected 2)

It appears the socket is AF_INET6, so getsockname returns more stuff. Changing line 79 to host = self.transportServer.sock.getsockname()[0], the server runs. The client now fails with

thread Thread-1 getting a blob using regular Pyro call...
thread Thread-2 getting a blob using regular Pyro call...
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\pcreinhold\Miniconda3\lib\site-packages\Pyro5\client.py", line 282, in connect_and_handshake
    sslContext=sslContext)
  File "C:\Users\pcreinhold\Miniconda3\lib\site-packages\Pyro5\socketutil.py", line 283, in create_socket
    sock.connect(connect)
OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\pcreinhold\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\pcreinhold\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "client.py", line 19, in regular_pyro
    data = p.get_with_pyro(blobsize)
  File "C:\Users\pcreinhold\Miniconda3\lib\site-packages\Pyro5\client.py", line 92, in __getattr__
    self._pyroGetMetadata()
  File "C:\Users\pcreinhold\Miniconda3\lib\site-packages\Pyro5\client.py", line 358, in _pyroGetMetadata
    self.__pyroCreateConnection()
  File "C:\Users\pcreinhold\Miniconda3\lib\site-packages\Pyro5\client.py", line 342, in __pyroCreateConnection
    connect_and_handshake(conn)
  File "C:\Users\pcreinhold\Miniconda3\lib\site-packages\Pyro5\client.py", line 304, in connect_and_handshake
    raise errors.CommunicationError(err) from x
Pyro5.errors.CommunicationError: cannot connect to ('fe80::e5fa:d40b:3e2b:ff46', 61120): [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions

I'm having a hard time getting Pyro5 working between machines in a LAN.

On the machine running the nameserver, I'm starting the nameserver with a command like:

pyro5-ns --host 10.0.0.29

I then run the greeting-server.py example from the docs.

On the local machine (the one running the nameserver), I can connect to the nameserver, use ns.list() and ns.lookup() to find example.greeting and get a URI, and then use the URI with Proxy to get a proxy object with which I can access GreetingMaker.get_fortune().

From the remote machine I can also connect to the nameserver, and use ns.list() and ns.lookup() to find example.greeting and it's URI. But when I get the URI via ns.lookup() the nameserver always returns the a URI ending in localhost. Unsurprisingly, I can't access the remote object because it's not on localhost.

I've played around with the nathost/natport settings in the Daemon but that doesn't seem to help, and in fact seems to prevent access on the local machine.

I've done some basic troubleshooting to try and insure that my problems are not firewall related. I've tried with firewalls off on both machines, without any difference in behavior. To further eliminate network misconfiguration as the source of the problems, I've tried the basic ZeroMQ (via pyzmq) examples between my local and remote machine and those work fine.

In any case, I'm looking for the simplest possible example illustrating how to use the Pyro5 nameserver to get Pyro5 working across to machines in a LAN. Any guidance would be greatly appreciated.

This is an enhancement request for steamlined functioning in zero-configuration setups.

I often need to expose an object to a non-localhost network interface (which IP address is it? I use some heuristics for that) and then register it to the nameserver (discovered via broadcast), like this:

ip=guessLocalIP(ip)
daemon=Pyro5.api.Daemon(host=ip,port=0,...)
uri=daemon.register(object)
ns=Pyro5.api.locate_ns()
ns.register('objectName',uri)

This will obviously fail for more complex networking setup.

I could expose the daemon on 0.0.0.0 but then the URI is not working, although the port is technically bound and reachable.

It is reasonable to suppose that the nameserver is on the same network as where the Daemon wants to be (that's the point, after all), so extracting local IP from the nameserver proxy should be very reliable:

ns=Pyro5.api.locate_ns()
daemon=Pyro5.api.Daemon(host=ns._getLocalIP(),port=0,...)
uri=daemon.register(object)
ns.register('objectName',uri)

I am aware that I could call ns._pyroConnection.sock.getsockname()[0] but having that exposed in the API looks like a good idea to me.

Hello,

In the documentation it is stated Pyro as useful for privilege separation.
Can you provide an example of such implementation in example dir ?

Would be much appreciated.

Like many other libraries, Pyro does seem to make it easy to "call" a function running in a different process or on a different machine which is great.
But all examples I have seen so far send only a small amount of data to the worker process and the worker process is only expected to need a little time to succeed.

But can PyroX be used to distribute longer lasting processing of huge amounts of data among several workers? In this scenario a batch would be much bigger than what fits into memory and ideally k processes would, possible on l different machines, would be available to process the next piece of work from that data, as soon as they have finished the previous piece of work.

In this scenario, we want that memory is not consumed by stuffing too much data in for sending to the workers but we also do not want to allow any worker to become idle and wait for more work as long as there is still data to process.

Does Pyro5/4 provide anything to implement this?

Is it possible to implement something like an edit lock (for lack of a better term) so that only one client can invoke non-idempotent methods on a remote object? What I have in mind is something like using API keys for authorization for methods that can change the state of the system, while freely allowing other clients to use idempotent methods. This would be kind of like keeping routes that implement GET requests open to everyone but decorating routes that implement POST/PUT/DELETE requests with the '@token_auth.login_required' decorator in Flask.

My use case is that I would like to use Pyro5 to manage laboratory hardware. I have a python API for the different pieces of hardware, and an instrument manager class to wrangle them all. I'm planning to use Pyro5 and a name-server to make it easy for multiple processes (e.g. a GUI, measurement software, etc.) to access the instruments. The server will use instance_mode='single' so that there's only ever one instance of the instrument manager and instrument objects, and I'll use SERVERTYPE = 'multiplex' to serialize the incoming requests. I believe we can even use batch mode to execute blocks of calls to the instruments together. All processes will only ever be running on the local machine, so I don't have security concerns really.

My only concern now is that I want to make sure that only one process at a time can edit the state of the instruments. It would be really bad if during my measurement routine someone walked up to the computer and edited, for example, the laser output power using the GUI. I want the GUI to be polling the instruments so that I get visual feedback of what's going on, but I don't want to make any conflicts with the measurement recipe. I control all the software here, so I could in principle grey-out the buttons on the GUI when we're in measurement, but it seems like it would be a nice feature to have available to make it so that if a process that doesn't have the right to edit the state of the instruments tries to do so that you'd get an exception of some kind.

Thanks - and thanks for making such a great tool!

I've just copied example https://github.com/irmen/Pyro5/tree/master/examples/instancemode and It looks like per call instance behaves like session instance:

Showing the different instancing modes.
The number printed, is the id of the instance that handled the call.

-----PERCALL (different number possible every time) -----
139735247191632
139735247191632
139735247191632
139735247188896
139735247188896
139735247188896
139735247190720
139735247190720
139735247190720

As you can see, there are repetitions in each session.

My env:
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
pyro5 5.11 Remote object communication library, fifth major version
robotframework 3.2.2 Generic automation framework for acceptance testing and robotic process automation (RPA)
serpent 1.30.2 Serialization based on ast.literal_eval

Pyro/serpent now supports pytz and dateutil
are there other libraries that we should support? (if they don't work automatically)

msgpack is not available in anaconda, so the recipe can't build....

maybe make it an optional serializer after all?

When I use pm2 to launch the name server, nothing logs to the stdout.

Finally, I added sys.stdout.flush() before the daemon.requestLoop() and it worked.

We have a setup where multiple systems on the network run multiple Pyro4 Daemon servers. I would like to be able to do something like

$ pyro5-serve [MANY OPTIONS] PATH-TO-PYTHON-MODULE

and set that command-line into a systemd or windows service. Could this be in scope for Pyro?

If so, I'd be interested in working on that, together with some zeroconf to discover Pyro daemons on the network (the current pyro nameserver is not suitable for our uses because there's no centralized system that could fulfil that job).

add the old unit tests that are still relevant.

Can't figure out how to configure Sentry it can catch server or client side exceptions.

I am accessing a remote object via URI. In one case, I need to get a local copy of the remote object (the class is registered for serialization), as opposed to proxy. Currently, I have something like this:

@Pyro5.api.expose
class Foo(object):
    def localCopy(self): return self
foo=Foo()
daemon=Pyro5.api.Daemon()
uri=daemon.register(foo)
daemon.requestLoop()

remote:

proxy=Pyro5.api.Proxy(uri)
local=proxy.localCopy()
assert isinstance(local,Foo) # OK

As you see, I define localCopy() which effectively transfers the object. But I am not sure whether this is guaranteed, whether it can be forced (there is the autoproxy global config flag). And perhaps Proxy does have some method to get the object transferred?

When I create a new Pyro5.api.Proxy on my client it corrupt the existing connection.