sdeo / protobuf-socket-rpc Goto Github PK

Added the Service class to the protobuf package.
It takes a service stub in the constructor and then
wraps each service message so it is callable directly
from the Service class instance.

Each method can be called synchronously or asynchronously.
See the examples for details.

I´ve created test Server a test Client.

1 - Client calls Server RPC method properly.
2 - Messages arrive on the server and i get the message.
3 - By the server side, i call done.run(gBuilder.build()); to call the callback 
method at client.

The issues is: the cliente thread finished and callback method is never called.
i see that for main classes (static void main(String[] arg)), if i add 
"threadPool.shutdownNow();" after callback, SOMETIMES is runs OK, but it does 
never work into a Junit TestCase....

Please provide me some protobuf-socket-rpc document or information about "how 
can i have my callback methods being called properly".

I am using following versions:
- protobuf-java-2.4.1.jar
- protobuf-socket-rpc-2.0.jar
- JDK 6

Thank you guys

I have a really simple Python server that serves RPC requests coming from a 
Java client (all sides using the RPC implementation from this project). For 
every request processed, an exception is thrown at the very end when the 
server tries to close the connection:

server.py line 95: self.request.shutdown(socket.SHUT_RDWR)
-> error(57, 'Socket is not connected')

I'm using v1.3

What steps will reproduce the problem?

file: service.proto
---------
import "rpc.proto";

option java_package = "com.common.service";
option java_outer_classname = "ServiceProtos";

service HelloService {
  rpc say (Request) returns (Response);
}
--------- 
main method :
---------
public static void main(String[] args) {
        ServerRpcConnectionFactory rpcConnectionFactory = SocketRpcConnectionFactories.createServerRpcConnectionFactory(12345);
        RpcServer rpcServer = new RpcServer(rpcConnectionFactory, Executors.newFixedThreadPool(10), false);     
        rpcServer.registerService( ServiceProtos.HelloService.newReflectiveService(new HelloServiceImpl()));    
        rpcServer.run();
    }
---------
HelloServiceImpl:
----------
public class HelloServiceImpl implements ServiceProtos.HelloService.Interface {
    @Override
    public void say(RpcController controller, Request request, RpcCallback<Response> done) {
        int i=0;
        i=1;    
    }
}


What is the expected output? What do you see instead?
EXPECTED: A server is running on localhost and listening to the port 12345.

NOT EXPECTED: Exception raised with the method called 
rpcServer.registerService(...).
Exception in thread "main" java.lang.ExceptionInInitializerError
    at com.common.service.ServiceProtos$HelloService.getDescriptor(ServiceProtos.java:106)
    at com.common.service.ServiceProtos$HelloService.getDescriptorForType(ServiceProtos.java:110)
    at com.googlecode.protobuf.socketrpc.RpcForwarder.registerService(RpcForwarder.java:61)
    at com.googlecode.protobuf.socketrpc.RpcServer.registerService(RpcServer.java:90)
    at protocolbuf.server.MainServer.main(MainServer.java:26)
Caused by: java.lang.IllegalArgumentException: Invalid embedded descriptor for 
"service.proto".
    at com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:286)
    at com.common.service.ServiceProtos.<clinit>(ServiceProtos.java:254)
    ... 5 more
Caused by: com.google.protobuf.Descriptors$DescriptorValidationException: 
HelloService.say: ".Request" is not defined.
    at com.google.protobuf.Descriptors$DescriptorPool.lookupSymbol(Descriptors.java:1693)
    at com.google.protobuf.Descriptors$MethodDescriptor.crossLink(Descriptors.java:1469)
    at com.google.protobuf.Descriptors$MethodDescriptor.access$2200(Descriptors.java:1407)
    at com.google.protobuf.Descriptors$ServiceDescriptor.crossLink(Descriptors.java:1388)
    at com.google.protobuf.Descriptors$ServiceDescriptor.access$600(Descriptors.java:1317)
    at com.google.protobuf.Descriptors$FileDescriptor.crossLink(Descriptors.java:371)
    at com.google.protobuf.Descriptors$FileDescriptor.buildFrom(Descriptors.java:237)
    at com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:284)
    ... 6 more

What version of the product are you using? On what operating system?
latest from the SVN

Please provide any additional information below.
nothing

What steps will reproduce the problem?
1. export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
2. python setup.py develop
3. python setup.py test

What is the expected output? What do you see instead?

I expect the same output as with the python backend but get:

{{running test
running egg_info
writing src/protobuf.socketrpc.egg-info/PKG-INFO
writing top-level names to src/protobuf.socketrpc.egg-info/top_level.txt
writing dependency_links to src/protobuf.socketrpc.egg-info/dependency_links.txt
reading manifest file 'src/protobuf.socketrpc.egg-info/SOURCES.txt'
writing manifest file 'src/protobuf.socketrpc.egg-info/SOURCES.txt'
running build_ext
libprotobuf ERROR google/protobuf/descriptor.cc:2237] Invalid proto descriptor 
for file "test.proto":
libprotobuf ERROR google/protobuf/descriptor.cc:2240]   
protobuf.socketrpc.Request: "protobuf.socketrpc.Request" is already defined in 
file "rpc.proto".
libprotobuf ERROR google/protobuf/descriptor.cc:2240]   
protobuf.socketrpc.Response: "protobuf.socketrpc.Response" is already defined 
in file "rpc.proto".
libprotobuf ERROR google/protobuf/descriptor.cc:2240]   
protobuf.socketrpc.TestService.TestMethod: "protobuf.socketrpc.Request" seems 
to be defined in "rpc.proto", which is not imported by "test.proto".  To use it 
here, please add the necessary import.
libprotobuf ERROR google/protobuf/descriptor.cc:2240]   
protobuf.socketrpc.TestService.TestMethod: "protobuf.socketrpc.Response" seems 
to be defined in "rpc.proto", which is not imported by "test.proto".  To use it 
here, please add the necessary import.
Traceback (most recent call last):
  File "setup.py", line 53, in <module>
    test_suite='protobuf.socketrpc.tests',
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 137, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 117, in with_project_on_sys_path
    func()
  File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 146, in run_tests
    testLoader = loader_class()
  File "/usr/lib/python2.6/unittest.py", line 816, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.6/unittest.py", line 843, in parseArgs
    self.createTests()
  File "/usr/lib/python2.6/unittest.py", line 849, in createTests
    self.module)
  File "/usr/lib/python2.6/unittest.py", line 613, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python2.6/unittest.py", line 587, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 34, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/lib/python2.6/unittest.py", line 576, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File "/home/jan/src/debian-stuff/build-area/python-protobuf.socketrpc-1.3.2/src/protobuf/socketrpc/tests/service_test.py", line 45, in <module>
    import fake
  File "/home/jan/src/debian-stuff/build-area/python-protobuf.socketrpc-1.3.2/src/protobuf/socketrpc/tests/fake.py", line 39, in <module>
    import test_pb2
  File "/home/jan/src/debian-stuff/build-area/python-protobuf.socketrpc-1.3.2/src/protobuf/socketrpc/tests/test_pb2.py", line 15, in <module>
    serialized_pb='\n\ntest.proto\x12\x12protobuf.socketrpc\"\x1b\n\x07Request\x12\x10\n\x08str_data\x18\x01 \x02(\t\".\n\x08Response\x12\x10\n\x08str_data\x18\x01 \x02(\t\x12\x10\n\x08int_data\x18\x02 \x01(\x05\x32V\n\x0bTestService\x12G\n\nTestMethod\x12\x1b.protobuf.socketrpc.Request\x1a\x1c.protobuf.socketrpc.Response')
  File "/usr/lib/pymodules/python2.6/google/protobuf/descriptor.py", line 580, in __init__
    cpp_message.BuildFile(self.serialized_pb)
  File "/usr/lib/pymodules/python2.6/google/protobuf/internal/cpp_message.py", line 63, in BuildFile
    _net_proto2___python.BuildFile(content)
TypeError: Couldn't build proto file into descriptor pool!}}

The problem seems to be that test.proto defines types that are already defined 
in rpc.proto and the C++ backend handles this differently.

As the C++ backend will be promoted to default according to 
http://code.google.com/intl/de-DE/apis/protocolbuffers/docs/reference/python-gen
erated.html#cpp_impl this should be fixed.

Maybe it should be evaluated whether the socket-rpc implementation should be 
switched to a protoc plugin as suggested at 
http://code.google.com/intl/de-DE/apis/protocolbuffers/docs/reference/python-gen
erated.html#service

Add another registerService function that takes a BlockingService as an 
argument. This will allow 
testing of implementations of blocking calls in addition to asynchronous calls. 

Description of feature:

Some sort of authorization support.

Why do you think this feature/enhancement is needed:

Currently the server is open, i.e. Anyone who knows the host:port can
connect and execute the rpc.

Implementation suggestions (optional):

Must be generic enough. Ideally this library should not have any auth
implementation itself, all it should do is pass an auth token in the
request and server will call some configured authorizer with it and return
error if it is not authorized.

Description of feature:
Allow user to define a listening interface other than localhost

Why do you think this feature/enhancement is needed:
Users may want to allow RPC connections from other machines!

Implementation suggestions (optional):
Apply attached patch to server.py

What steps will reproduce the problem?
1. I need protobuf-socket-rpc
2. I don't know how to install it.
3.

Java, Windows XP, for purposes of communicating with a BIMserver.

What steps will reproduce the problem?
Using the python implementation...
When creating 2 or more implementations of the same service running on
separate sockets sockets, the final implementation added will over ride all
other implementations even on different sockets because serviceMap is
declared with class scope in SocketRpcServer.

Description of feature: To provide any authentication/authorization solution 
for the services

Why do you think this feature/enhancement is needed: This feature is needed to 
avoid not authorized users/clients to use the services. In addition this is a 
way to make sure that service can identify which exact user doing the service 
request.


Implementation suggestions (optional): Adding any OAuth1 and OAuth2 protocol 
implementations would great, so it would be more for "SaaS" implementation...

Thank you.

Description of feature:
Recompile the proto files using the latest Protocol Buffers 2.1.0

Why do you think this feature/enhancement is needed:
Keep up with the latest version of protocol buffers.

We're using PB 2.4.0 (not yet released at this time) for its substantially 
faster native Python protobufs.

When using the native Python protobufs (set the env var 
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp), protobuf-socket-rpc gives us the 
following against the current protobuf svn HEAD (r352):

...
  File "/home/yang/work/pod/env/lib/python2.6/site-packages/protobuf/__init__.py
", line 22, in <module>
    from service import RpcThread, RpcService
  File "/home/yang/work/pod/env/lib/python2.6/site-packages/protobuf/service.py"
, line 39, in <module>
    from protobuf.channel import SocketRpcChannel
  File "/home/yang/work/pod/env/lib/python2.6/site-packages/protobuf/channel.py", line 41, in <module>
    import rpc_pb2 as rpc_pb
  File "/home/yang/work/pod/env/lib/python2.6/site-packages/protobuf/rpc_pb2.py", line 104, in <module>
    options=None),
  File "/home/yang/work/protobuf-read-only/python/google/protobuf/descriptor.py", line 400, in __init__
    self._cdescriptor = cpp_message.GetFieldDescriptor(full_name)
  File "/home/yang/work/protobuf-read-only/python/google/protobuf/internal/cpp_message.py", line 58, in GetFieldDescriptor
    return _pool.FindFieldByName(full_field_name)
TypeError: Couldn't find field protobuf.socketrpc.Request.service_name

Not really sure how big the compatibility break is, but hopefully it requires 
only minor tweaks in protobuf-socket-rpc.

1. Mavenize the java project.

2. Upgrade to protobuf 2.3.0 for both java and python

SocketRpcChannel needs to have a constructor with an InetAddress instead of 
a hostname.

It sometimes happens that name resolutions servers are not available or 
remote machines may not have names. This is especially true when testing - I 
run the client code in an emulator that cannot resolve a name easily but can 
connect directly to a given IP.

Description of feature:

It would be nice to have a changelog (i.e. the content of 
http://code.google.com/p/protobuf-socket-rpc/wiki/ReleaseNotes) in the source 
tarballs.

Why do you think this feature/enhancement is needed:

I'm about to package your module for Debian GNU/Linux and it is nice for users 
to have a upstream changelog in /usr/share/doc/<packagename>

Implementation suggestions (optional):

put the content of 
http://code.google.com/p/protobuf-socket-rpc/wiki/ReleaseNotes in a file named 
NEWS or ChangeLog and put it into the source tarballs.

Description of feature:

Add the ability to start and stop the server since the current 
implementation blocks the current thread.


Why do you think this feature/enhancement is needed:

In the application I am working on, I wanted to give the end users to 
easily start and stop the server.


Implementation suggestions (optional):

spawn the ServerSocket on it's own tread.

Hi; I maintain protobuf-net, and I'm interested in writing a compatible 
client/server implementation. However, the wire format is not entirely clear, 
in particular for multiple messages over the same socket.

Would it be possible to clarify the message packing (in particular, 
distinguishing between successive messages in either direction, with/without 
the delimited flag set). Thanks in advance,

Marc

Description of feature:

Add support for keeping the socket open for multiple rpcs to be made
through it.

Why do you think this feature/enhancement is needed:

Better performance and efficiency of socket use if there is a high volume
of rpcs between the same client and server.

Implementation suggestions (optional):

Either add an option to SocketRpcChannel to keep persistent connection
until told explicitly to disconnect or create a new type of channel.

What steps will reproduce the problem?
1. Create a java client/server, and a python client/server, all from one .proto 
file
2. invoke java server with java client, works.
3. invoke python server with python client, works.
4. invoke java server with python client, gets an error ( (decodeError): 
Unexpected end-group tag.)
5. invoke python server with java client, client hangs.

What is the expected output? What do you see instead?
expect successful call between platforms


What version of the product are you using? On what operating system?
tried Java protobuf-socket-rpc-2.0.jar with protobuf.socketrpc-1.3.2-py2.6.egg
tried Java protobuf-socket-rpc-2.0.jar with latest python source from svn (Sep 
28 2012)

Please provide any additional information below.

Addresses issues 11 and 12.

Also cleans up old references to controller.success which no longer exists
and attempts to hide controller.error and controller.fail by renaming them
controller._error and controller._fail to force access through
controller.handleError.

The default host arg to SocketRpcServer constructor is 'localhost'
I would expect the server to bind a socket to the computers localhost
127.0.0.1 address and for that socket to not be accesible from a remote
host.  However, I would expect that I should be able to but in the value of
retrieved via

import socket
socket.gethostbyname()

into the constructor and have it bind to the hosts external facing ip
address.  However what you get is still a binding to the 127.0.0.1 address
making the socket inaccessible from a remote host.

Apparently the way to get a binding to the external facing IP is to set the
host arg to the empty string.

I would suggest changing the default host arg to be the empty string ''
which seems to give the most intuitive behavior to the constructor when
called with only a port number arg

Description of feature:

I suggest to rename the Python module from protobuf to something less generic 
like protobuf.socketrpc because protobuf is already occupied by Google's 
protocol buffers implementation.

Why do you think this feature/enhancement is needed:

Debian's Python policy suggests that a Debian package is called after the 
imported module name of the Python module [1]. The package name python-protobuf 
is already occupied by Google's protocol buffers Python module [2].

[1] 
http://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.htm
l#s-package_names
[2] http://packages.debian.org/sid/python-protobuf

Implementation suggestions (optional):

I prepared the attached changes in a local git clone of the SVN repository. The 
patch is in git's extended format to track the file moves and might not apply 
to a SVN working copy directly.

I moved all files in python/src/main/protobuf to 
python/src/main/protobuf/socket/rpc and added empty __init__.py files to 
python/src/main/protobuf and python/src/main/protobuf/socket too. Afterwards I 
changed the import statements in all Python files that required changes. Tests 
and examples are adapted to the new module path too.

I did also add some more metadata to setup.py.

What steps will reproduce the problem?

Using python client side code...
Create an rpc method whose server side code throws an exception
resulting in a call to Controller.handleError on the client side.
Make a synchronous call to the rpc method that fails.


What is the expected output? What do you see instead?

I expect to see the value of controller.error raised on the client side.
Instead, I get an error saying that the response object was not initialized

Synchronous requests don't have a proper timeout argument.

What steps will reproduce the problem?
1. use python's example/time
   (fix paths if needed)
2. change run_server.py to sleep 10 seconds just before returning
   time.sleep(10)
   done.run(response)
3. execute run_server.py
4. execute run_client.py (using 1 second timeout)

What is the expected output? What do you see instead?
we expect an exception like 
  RpcError('request timed out')
instead the request completes successfully (after ~10 seconds)

What version of the product are you using? On what operating system?
protobuf-socket-rpc 1.3.2, linux

the Java API doesn't even seem to have a timeout parameter.

Please provide any additional information below.

ideally all client APIs, python and java, will have a timeout
parameter in all synchronous calls. all requests should fail with
a timeout-exception if the request is not completed withing the 
specified timeout, for _any_ reason, including 
- connection timeout
- slow server
- suspended server
etc.

What steps will reproduce the problem?
1. install protobuf from ubuntu repositories (which installs protobuf 2.2.0 
libraries)
2. try to run an rpc server
3. you will observe

Traceback (most recent call last):
  File "./rpcserver.py", line 4, in <module>
    import protobuf.socketrpc.server as server
  File "/usr/local/lib/python2.6/dist-packages/protobuf.socketrpc-1.3.2-py2.6.egg/protobuf/socketrpc/__init__.py", line 22, in <module>
    from service import RpcThread, RpcService
  File "/usr/local/lib/python2.6/dist-packages/protobuf.socketrpc-1.3.2-py2.6.egg/protobuf/socketrpc/service.py", line 43, in <module>
    from protobuf.socketrpc.channel import SocketRpcChannel
  File "/usr/local/lib/python2.6/dist-packages/protobuf.socketrpc-1.3.2-py2.6.egg/protobuf/socketrpc/channel.py", line 43, in <module>
    import rpc_pb2 as rpc_pb
  File "/usr/local/lib/python2.6/dist-packages/protobuf.socketrpc-1.3.2-py2.6.egg/protobuf/socketrpc/rpc_pb2.py", line 10, in <module>
    DESCRIPTOR = descriptor.FileDescriptor(
AttributeError: 'module' object has no attribute 'FileDescriptor'

What is the expected output? What do you see instead?

rpc server should run. 

What version of the product are you using? On what operating system?

ubuntu. protobuf 2.2.0

Please provide any additional information below.

I think rpc_pb2.py is compiled with protobuf 2.3.0, version 2.2.0 of protobuf 
does not contain descriptor.FileDescriptor whereas 2.3.0 contains it.

Description of feature:
provide a Java example that use the hello.proto file generated Helloworld.java 

Why do you think this feature/enhancement is needed:
protobuf's most important function is to show the cross-language invokation, so 
we can use java as rpc hello client and python 

Implementation suggestions (optional):
I know that the test.proto show the same tech, but I confused with so many codes