acg / lwpb Goto Github PK

Symptoms are seen on the 'python' branch using the python bindings. There seems to be a very strange type of error where the python stack, memory or state in general is getting smashed up. Please see the following example at https://gist.github.com/11c6f1b509770b848a52 as github markup formatting couldn't easily preformat blocks of text.

Line #30 from the gist explodes. The output looks like this:
$ python test.py
1
2
Traceback (most recent call last):
File "test_perf_lwpb.py", line 13, in
for n in [1,2]:
OverflowError: could not convert to long

Platform is ubuntu lucid

dpkg -l |grep protobuf
ii libprotobuf-c0 0.14-1 Protocol Buffers C library
ii libprotobuf-c0-dev 0.14-1 Protocol Buffers C static library and header
ii libprotobuf-dev 2.4.0a-2ubuntu2 protocol buffers C++ library (development fi
ii libprotobuf-lite7 2.4.0a-2ubuntu2 protocol buffers C++ library (lite version)
rc libprotobuf5 2.2.0a-0.1ubuntu1 protocol buffers C++ library
ii libprotobuf7 2.4.0a-2ubuntu2 protocol buffers C++ library
ii protobuf-c-compiler 0.14-1 protocol buffers C compiler
ii protobuf-compiler 2.4.0a-2ubuntu2 compiler for protocol buffer definition file
ii python-protobuf 2.4.0a-2ubuntu2 Python bindings for protocol buffers

The lwpb C library doesn't enforce required fields when encoding and decoding. This is by design. It would require another recursive pass over all the fields in the message descriptor to see if any were omitted in the encoded message. It would also require tracking which fields have been seen and which haven't, which is a departure from the callback design of lwpb_decoder_decode, where caller is responsible for actually populating the decoded message.

Since validation of required missing fields is a questionable (see Required is Forever) and less frequently used feature of protobuf, and since enforcing them imposes a penalty in decoding, this should be an optional feature of the python decoder that you must explicitly request.

There's very little reason for lwpb to use autotools, it only complicates things. Replace with a simpler build. In passing, clean up the test suite build and fix the protobuf version compatibility issue there.

Hi,

I downloaded the code and built it on a mac (osx 10.11 + Python 2.7.10). But python setup.py test did not pass.

To get around build error like 'missing return', I have add -Wno-return-type option in Makefile.

======================================================================
FAIL: runTest (tests.EncoderTestCase)
Encode packed
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yichao/dev/lwpb/python/tests.py", line 35, in runTest
    self.outdata)
AssertionError: '\nP\x00\x00\x00\x00\x00\x00\x00\x00I\x92$I\x92$\xf9?I\x92$I\x92$\t@\xb7m\xdb\xb6m\xdb\x12@I\x92$I\x92$\x19@\xdb\xb6m\xdb\xb6m\x1f@\xb7m\xdb\xb6m\xdb"@\x00\x00\x00\x00\x00\x00&@I\x92$I\x92$)@\x92$I\x92$I,@"F\x80\x80\x80\x80\xf0\xf0\x1f\x81\x80\x80\x80\xf0\xf0\x1f\x90\x80\x80\x80\xf0\xf0\x1f\xd1\x80\x80\x80\xf0\xf0\x1f\x80\x82\x80\x80\xf0\xf0\x1f\xf1\x84\x80\x80\xf0\xf0\x1f\x90\x8a\x80\x80\xf0\xf0\x1f\xe1\x92\x80\x80\xf0\xf0\x1f\x80\xa0\x80\x80\xf0\xf0\x1f\xa1\xb3\x80\x80\xf0\xf0\x1f2\x10\x00\x01\x10Q\x80\x02\xf1\x04\x90\n\xe1\x12\x80 \xa13BP\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00Q\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00q\x02\x00\x00\x00\x00\x00\x00\x10\x05\x00\x00\x00\x00\x00\x00a\t\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\xa1\x19\x00\x00\x00\x00\x00\x00R(\x00\x00\x00\x00\x01\x00\x00\x00\x10\x00\x00\x00Q\x00\x00\x00\x00\x01\x00\x00q\x02\x00\x00\x10\x05\x00\x00a\t\x00\x00\x00\x10\x00\x00\xa1\x19\x00\x00' != '\nP\x00\x00\x00\x00\x00\x00\x00\x00I\x92$I\x92$\xf9?I\x92$I\x92$\t@\xb7m\xdb\xb6m\xdb\x12@I\x92$I\x92$\x19@\xdb\xb6m\xdb\xb6m\x1f@\xb7m\xdb\xb6m\xdb"@\x00\x00\x00\x00\x00\x00&@I\x92$I\x92$)@\x92$I\x92$I,@"\x10\x00\x01\x10Q\x80\x02\xf1\x04\x90\n\xe1\x12\x80 \xa132\x10\x00\x01\x10Q\x80\x02\xf1\x04\x90\n\xe1\x12\x80 \xa13BP\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00Q\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00q\x02\x00\x00\x00\x00\x00\x00\x10\x05\x00\x00\x00\x00\x00\x00a\t\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\xa1\x19\x00\x00\x00\x00\x00\x00R(\x00\x00\x00\x00\x01\x00\x00\x00\x10\x00\x00\x00Q\x00\x00\x00\x00\x01\x00\x00q\x02\x00\x00\x10\x05\x00\x00a\t\x00\x00\x00\x10\x00\x00\xa1\x19\x00\x00'

----------------------------------------------------------------------
Ran 48 tests in 0.007s

FAILED (failures=1)

Currently, the python lwpb code can't resolve nested definitions like this:

 message A {
   message B {
     enum C {
     ...
     };
   };
 };

 message D {
   optional A.B.C stuff = 1;
 };

The rules for resolving type names follow C++ scoping.

This feels like an unnecessary complication in the protocol buffers spec -- I've personally never had a need for it and simply use a flat list of definitions in my .proto files.

However, it's in the spec, so see if it can be implemented in the descriptor code.

Where can I find the version of protoc that will generate C files like the ones included in the repository?

The default make target currently descends into ./test/ and builds programs that test lwpb against Google Protocol Buffers. We say that lwpb doesn't depend on Google's protobuf, so this should be optional.

The lwpb C library doesn't apply default values for optional fields when decoding. This is by design. It would require another recursive pass over all the fields in the message descriptor to see if any were omitted in the encoded message. It would also require tracking which fields have been seen and which haven't, which is a departure from the callback design of lwpb_decoder_decode, where caller is responsible for actually populating the decoded message.

Since default values are a questionable and less frequently used feature of protobuf, and since applying them imposes a penalty in decoding, this should be an optional feature of the python decoder that you must explicitly request.

Currently there's no way to access an enumeration value by its symbolic name, which would be useful when constructing messages to encode and working with decoded messages.

ld: warning: ignoring file ../src/.libs/liblwpb.a, file was built for archive which is not the architecture being linked (i386)
ld: duplicate symbol _value in build/temp.macosx-10.6-intel-2.7/descriptor.o and build/temp.macosx-10.6-intel-2.7/decoder.o for architecture i386
collect2: ld returned 1 exit status
ld: duplicate symbol _value in build/temp.macosx-10.6-intel-2.7/descriptor.o and build/temp.macosx-10.6-intel-2.7/decoder.o for architecture x86_64
collect2: ld returned 1 exit status
lipo: can't open input file: /var/folders/r6/j1x0wz_s3wz507f660m6shmm0000gn/T//ccV3f4Ew.out (No such file or directory)
error: command 'gcc-4.2' failed with exit status 1
running clean
removing 'build/temp.macosx-10.6-intel-2.7' (and everything under it)

any hints? thanks a lot