Giter Club home page Giter Club logo

ruby-protocol-buffers's Introduction

Protocol Buffers v3

IMPORTANT This library supports the Protocol Buffers v2 specification. Google has developed official Ruby support for Protobuf v3, the gem is available at https://rubygems.org/gems/google-protobuf. Because there's now an official gem, I have no plans to implement v3 support in this library.

Ruby Protocol Buffers

Build Status

Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. Google uses Protocol Buffers for almost all of its internal RPC protocols and file formats.

This library has two components: a compiler to turn .proto definitions into Ruby modules (extension .pb.rb), and a runtime to use protocol buffers defined by these modules.

The compiler relies on Google's C++ based compiler (protoc) for much of the heavy lifting -- this has huge advantages in ensuring compatibility and correctness. If you don't need cross-language interoperability you can create Message classes directly in ruby, in which case protoc is not needed. See "Writing Message Classes Directly" below.

This library is heavily optimized for encoding and decoding speed.

Because this is a tool for generating code, the RDoc documentation is a bit unusual. See the text in the ProtocolBuffers::Message class for details on what code is generated.

Installation

$ gem install ruby-protocol-buffers

If you want to compile .proto files to ruby, you'll need protoc version >= 2.2 (the Google Protocol Buffer compiler) installed in the environment where you will be compiling them. You do not need protoc installed to use the generated .pb.rb files.

For greater performance, consider installing the varint gem as well. This optional gem builds a small C extension to make parsing protocol buffers faster. If your application uses a Gemfile, add varint to the Gemfile alongside ruby-protocol-buffers.

Example

Given the file test.proto:

package Test;

message MyMessage {
  optional string myField = 1;
}

Compile it to ruby using the command:

$ ruby-protoc test.proto

Then it can be used from ruby code:

require 'test.pb'
msg = Test::MyMessage.new(:myField => 'zomgkittenz')
open("test_msg", "wb") do |f|
  msg.serialize(f)
end
encoded = msg.serialize_to_string # or msg.to_s
Test::MyMessage.parse(encoded) == msg # true

Writing Message Classes Directly

Protocol Buffer definitions are often shared between applications written in different programming languages, and so are normally defined in .proto files and translated to ruby using the ruby-protoc binary.

However, it's quite simple to write ProtocolBuffers::Message classes directly when a .proto file isn't needed.

require 'protocol_buffers'

class User < ProtocolBuffers::Message
  required :string, :name, 1
  required :string, :email, 2
  optional :int32, :logins, 3
end

class Group < ProtocolBuffers::Message
  repeated User, :users, 1
  repeated Group, :subgroups, 2
  
  module GroupType
    include ProtocolBuffers::Enum
    Study = 1
    Play = 2
  end

  optional GroupType, :group_type, 3
end

This code is essentially equivalent to the code ruby-protoc will generate if given this .proto file:

message User {
  required string name = 1;
  required string email = 2;
  optional int32 logins = 3;
}

message Group {
  repeated User users = 1;
  repeated Group subgroups = 2;

  enum GroupType {
    Study = 1;
    Play = 2;
  }

  optional GroupType group_type = 3;
}

Using a hand-written Message subclass is the same as using a Message class generated by ruby-protoc.

group = Group.new(:group_type => Group::GroupType::Play)
group.users << User.new(:name => 'test user', :email => '[email protected]')
open("group1.test", "wb") do |f|
  group.serialize(f)
end

Features

Supported Features

  • messages, enums, field types, all basic protobuf features
  • packages
  • imports
  • nested types
  • passing on unknown fields when re-serializing a message
  • groups
  • RPC stubbing
  • formatting to and parsing from text format

Currently Unsupported Features

  • extensions
  • packed option (could be useful)
  • accessing custom options

Authors

Brian Palmer (http://github.com/codekitchen)

Source

https://github.com/codekitchen/ruby-protocol-buffers

License

See the LICENSE file included with the distribution for licensing and copyright details.

ruby-protocol-buffers's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ruby-protocol-buffers's Issues

undefined method `[]' for nil:NilClass during decode ?

Hi

I am using this proto https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto which i compiled to give me the following .rb.

#!/usr/bin/env ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!

require 'protocol_buffers'

# forward declarations
class PBDNSMessage < ::ProtocolBuffers::Message; end

class PBDNSMessage < ::ProtocolBuffers::Message
  # forward declarations
  class DNSQuestion < ::ProtocolBuffers::Message; end
  class DNSResponse < ::ProtocolBuffers::Message; end

  # enums
  module Type
    include ::ProtocolBuffers::Enum

    set_fully_qualified_name "PBDNSMessage.Type"

    DNSQueryType = 1
    DNSResponseType = 2
    DNSOutgoingQueryType = 3
    DNSIncomingResponseType = 4
  end

  module SocketFamily
    include ::ProtocolBuffers::Enum

    set_fully_qualified_name "PBDNSMessage.SocketFamily"

    INET = 1
    INET6 = 2
  end

  module SocketProtocol
    include ::ProtocolBuffers::Enum

    set_fully_qualified_name "PBDNSMessage.SocketProtocol"

    UDP = 1
    TCP = 2
  end

  set_fully_qualified_name "PBDNSMessage"

  # nested messages
  class DNSQuestion < ::ProtocolBuffers::Message
    set_fully_qualified_name "PBDNSMessage.DNSQuestion"

    optional :string, :qName, 1
    optional :uint32, :qType, 2
    optional :uint32, :qClass, 3
  end

  class DNSResponse < ::ProtocolBuffers::Message
    # forward declarations
    class DNSRR < ::ProtocolBuffers::Message; end

    set_fully_qualified_name "PBDNSMessage.DNSResponse"

    # nested messages
    class DNSRR < ::ProtocolBuffers::Message
      set_fully_qualified_name "PBDNSMessage.DNSResponse.DNSRR"

      optional :string, :name, 1
      optional :uint32, :type, 2
      optional :uint32, :class, 3
      optional :uint32, :ttl, 4
      optional :bytes, :rdata, 5
    end

    optional :uint32, :rcode, 1
    repeated ::PBDNSMessage::DNSResponse::DNSRR, :rrs, 2
    optional :string, :appliedPolicy, 3
    repeated :string, :tags, 4
    optional :uint32, :queryTimeSec, 5
    optional :uint32, :queryTimeUsec, 6
  end

  required ::PBDNSMessage::Type, :type, 1
  optional :bytes, :messageId, 2
  optional :bytes, :serverIdentity, 3
  optional ::PBDNSMessage::SocketFamily, :socketFamily, 4
  optional ::PBDNSMessage::SocketProtocol, :socketProtocol, 5
  optional :bytes, :from, 6
  optional :bytes, :to, 7
  optional :uint64, :inBytes, 8
  optional :uint32, :timeSec, 9
  optional :uint32, :timeUsec, 10
  optional :uint32, :id, 11
  optional ::PBDNSMessage::DNSQuestion, :question, 12
  optional ::PBDNSMessage::DNSResponse, :response, 13
  optional :bytes, :originalRequestorSubnet, 14
  optional :string, :requestorId, 15
  optional :bytes, :initialRequestId, 16
end

But when i receive DNSResponse events from the DNS server it get errors e.g.

/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/
field.rb:141:in `class': undefined method `[]' for nil:NilClass (NoMethodError)

This only happens when i receive an event with one or more DNSRR entries.. when there are none it works OK.

For example a message produced on the source that works is

type: DNSResponseType
messageId: "\213\344sy\314\003OX\234\014\204\220{7\r\337"
socketFamily: INET
socketProtocol: UDP
from: "\n\204\020\017"
to: "\n\204\020\017"
inBytes: 89
timeSec: 1496380819
timeUsec: 865757
id: 55841
question {
  qName: "google.com."
  qType: 6
  qClass: 1
}
response {
  rcode: 0
  queryTimeSec: 1496380819
  queryTimeUsec: 865365
}

And one that fails is

type: DNSResponseType
messageId: "\254\251v\304\223`IV\236\245#y\003\375<\214"
socketFamily: INET
socketProtocol: UDP
from: "\n\204\020\017"
to: "\n\204\020\017"
inBytes: 87
timeSec: 1496380805
timeUsec: 897818
id: 64238
question {
  qName: "google.com."
  qType: 1
  qClass: 1
}
response {
  rcode: 0
  rrs {
    name: "google.com."
    type: 1
    class: 1
    ttl: 151
    rdata: "\254\331\013."
  }
  rrs {
    name: "google.com."
    type: 1
    class: 1
    ttl: 151
    rdata: "\254\331\013."
  }
  rrs {
    name: "google.com."
    type: 1
    class: 1
    ttl: 151
    rdata: "\254\331\013."
  }
  queryTimeSec: 1496380805
  queryTimeUsec: 896443
}

Can you suggest please if this is a problem with the compiled protobuf definition or maybe a bug ?

thank you.

slow parse with rubinius 2.2.10 ?

Here's an odd one that I'd love some feedback on:

Attempting to squeeze extra performance out of an app, I moved it from MRI 2.1 to Rubinius 2.2.10. With a little tweaking, got all the tests to pass, but the ruby-protocol-buffers 'parse' seems to be about half as fast as with MRI 2.1. Does that seem right?

I'm literally just taking a proto object and doing

object.parse(encoded_data)

License missing from gemspec

Some companies will only use gems with a certain license.
The canonical and easy way to check is via the gemspec,

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Even for projects that already specify a license, including a license in your gemspec is a good practice, since it is easily
discoverable there without having to check the readme or for a license file.

For example, there is a License Finder gem to help companies ensure all gems they use
meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough
issue that even Bundler now generates gems with a default 'MIT' license.

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file),
github has created a license picker tool.

In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally
looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :).

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue and let me know. In either case, I'll follow up. Thanks!

p.s. I've written a blog post about this project

Forward usage of nested enums generates ruby files that can't be loaded

Repro: https://github.com/sergiocampama/ruby_proto_repro, just run generate_proto.sh from inside the repo.

What happens is that the MessageB gets added to the forward declaration, but its internal enums are not, so the subsequent usage of the enum type fails to be resolved.

One way to fix this particular would be to forward declare all enums, even nested ones, before messages. Then forward declare the messages and finally then extend the messages with any fields that they have defined.

Varint decoding (in Ruby) fails with "invalid byte sequence in UTF-8"

Decoding a GTFS-realtime message fails for me with a ProtocolBuffers::DecodeError, "error parsing message". Debugging using "ruby -rdebug" (see below) reveals varint.decode is failing trying to read from the input stream using io.getc (varint.rb:30).

Looks to me the code here should be trying to read a byte and not a UTF-8 character, so the error makes sense. Should this line be "io.read(1).ord" instead? That change fixes the problem for me and the message is parsed successfully. (Seems both Ruby 1.8 and 1.9 have an IO.read method, and the method is provided by the LimitedIO class as well.)


/usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/varint.rb:30: ' (NilClass) from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/message.rb:274:inparse'
from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/message.rb:280:in parse' from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/field.rb:601:indeserialize'
from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/decoder.rb:57:in decode' from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/message.rb:274:inparse'
from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/message.rb:280:in parse' from ./test.rb:6:inblock in

'
from ./test.rb:5:in open' from ./test.rb:5:in'
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/varint.rb:30: byte = io.getc.ord
(rdb:1) c
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/decoder.rb:88: invalid byte sequence in UTF-8' (ArgumentError) from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/message.rb:280:inparse'
from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/field.rb:601:in deserialize' from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/decoder.rb:57:indecode'
from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/message.rb:274:in parse' from /usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/message.rb:280:inparse'
from ./test.rb:6:in block in <main>' from ./test.rb:5:inopen'
from ./test.rb:5:in `'
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-protocol-buffers-1.3.2/lib/protocol_buffers/runtime/decoder.rb:88: raise(DecodeError, "error parsing message")

better error message if protoc is not installed, or errors from protoc

(copied from mozy#6)

$ gem install ruby-protocol-buffers
$ ruby-protoc featureful.proto
HOME_PATH/.rvm/gems/ruby-1.8.7-p334/gems/ruby-protocol-buffers-1.0.1/lib/protocol_buffers/compiler.rb:16:in compile': 127 (ProtocolBuffers::CompileError) from HOME_PATH/.rvm/gems/ruby-1.8.7-p334/gems/ruby-protocol-buffers-1.0.1/bin/ruby-protoc:38 from HOME_PATH/.rvm/gems/ruby-1.8.7-p334/bin/ruby-protoc:19:inload'
from HOME_PATH/.rvm/gems/ruby-1.8.7-p334/bin/ruby-protoc:19

The featureful.proto file is taken straight from the current source code.

cannot run ruby-protoc without requires error

just gem installed this, trying to run it, but it's failing in a require

...kernel_require.rb:55:in `require': cannot load such file -- protocol_buffers/runtime/text_parser (LoadError)

protoc --version reports "libprotoc 2.4.1"

what am I missing? is there some other library I'm supposed to install first?

Can't set nil or false as a default value

opts[:default] || some_hardcoded_default

This pattern repeats frequently for default values. Unfortunately, it doesn't work when you've explicitly asked for nil or false. Oops! Might I recommend the following instead?

opts.has_key?(:default) ? opts[:default] : some_hardcoded_default

New release of ruby-protocol-buffers.

Hey!
We're really needing the change a5de828 but it is not in 1.6.1 release yet. Could you please release new version from current master so we could just use normal gem installation with version?

Unable to properly parse proto2 string

Hello

I'm starting from that : https://github.com/google/play-work/blob/master/emm-notifications/src/main/proto/emm_pubsub.proto

No problem to generate the .pb file.

#!/usr/bin/env ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!

require 'protocol_buffers'

module EmmNotifications
  # forward declarations
  class MdmPushNotification < ::ProtocolBuffers::Message; end
  class TestPushNotification < ::ProtocolBuffers::Message; end
  class ProductApprovalEvent < ::ProtocolBuffers::Message; end
  class InstallFailureEvent < ::ProtocolBuffers::Message; end
  class AppUpdateEvent < ::ProtocolBuffers::Message; end
  class NewPermissionsEvent < ::ProtocolBuffers::Message; end
  class AppRestrictionsSchemaChangeEvent < ::ProtocolBuffers::Message; end

  class MdmPushNotification < ::ProtocolBuffers::Message
    set_fully_qualified_name "EmmNotifications.MdmPushNotification"

    optional :string, :enterprise_id, 1
    optional :int64, :event_notification_sent_timestamp_millis, 2
    repeated ::EmmNotifications::TestPushNotification, :test_push_notification, 3
    repeated ::EmmNotifications::ProductApprovalEvent, :product_approval_event, 4
    repeated ::EmmNotifications::InstallFailureEvent, :install_failure_event, 5
    repeated ::EmmNotifications::AppUpdateEvent, :app_update_event, 6
    repeated ::EmmNotifications::NewPermissionsEvent, :new_permissions_event, 7
    repeated ::EmmNotifications::AppRestrictionsSchemaChangeEvent, :app_restrictions_schema_change_event, 8
  end

  # etc ..............
end

But when I try to use it. I get

EmmNotifications::MdmPushNotification.parse(Base64.decode64("CjEKFQoIMTIzMjEzMjESCXJpZ2h0IG5vdxIWY29tLmdvb2dsZS5hbmRyb2lkLmdtcxgA"))
 => #<EmmNotifications::MdmPushNotification enterprise_id="\n\u0015\n\b12321321\u0012\tright now\u0012\u0016com.google.android.gms\u0018\u0000" event_notification_sent_timestamp_millis=<unset> test_push_notification=[] product_approval_event=[] install_failure_event=[] app_update_event=[] new_permissions_event=[] app_restrictions_schema_change_event=[]>

With the java implementation you get :

{       # (androidenterprise.pubsub.MdmPushNotification) size=59B
     enterprise_id: "C02enrg18"    # size=9
     event_notification_sent_timestamp_millis: 1459866480541       # 1.33Ti; [as milliseconds]: 2016-04-05 15:28:00.541 +0100
     product_approval_event: {     # (androidenterprise.pubsub.ProductApprovalEvent) size=39B
          product_id: "app:com.callpod.android_apps.keeper"   # size=35
          approved  : APPROVED        # integer=1 (enum androidenterprise.pubsub.ProductApprovalEvent.ApprovalStatus)
     }     # product_approval_event[0]
}

Do you have any idea, why it fails ?

How to handle multiple versions

I know this is more of a question than issue, but how would you recommend handling multiple versions of the protobuffer? When we update a proto message we need to recompile but people running older versions of our client software with older proto messages break if we try to decode them with an updated message.

Im wondering how you handle this. Thanks.

Does not work with multiple schema files

We are currently using protobuf, and written code something like
Common log schema

syntax = "proto3";
package structurelogs;
message CommonLog{
	string eventType = 1;
	string eventTime = 2;
}

Device log schema

syntax = "proto3";
package structurelogs;
message DeviceLog{
	string deviceType = 2;
	string serialNumber = 3;
	string action = 4;
	string sourceIP = 5;
	string destinationIP = 6;
}

version specific schema

syntax = "proto3";
package structurelogs;
import "CommonLog.proto";
import "DeviceLog.proto";
message VersionSpecificDeviceLog{
	CommonLog commonLog = 1;
	DeviceLog deviceLog= 2;
	//several other properties
}

in logstash using

output {
	file {
	  codec => protobuf
	  {
		 class_name => "Structurelogs::VersionSpecificDeviceLog"
                        include_path => ['/var/protobuf/CommonLog.pb.rb', '/var/protobuf/DeviceLog.pb.rb', '/var/protobuf/VersionSpecificDeviceLog.pb.rb']
	  }
	  path => "/var/parsedlogs/parsed.log"
	}
}

After starting Logstash service, I always get error like
error=>#<NoMethodError: undefined method 'eventTime='
sometimes with different variable which is part of either CommonLog or DeviceLog

I have used ruby-protoc to compile .proto files on ubuntu.

File creation of rprotoc doesn't match behavior of protoc

(copied from mozy#16)

This relates to proto files that imports other proto files. My specific use case involves having a directory hierarchy that matches the namespace hierarchy. A short solution would be for rprotoc to only output the proto file it's being asked to compile and not also generate all imports at the same time.

Initially I ran rprotoc and kept all of the files in a top level directory. Then imports fail because they are structured as 'namespace/File.pb.rb'. Copying the output into the same directory structure would give me runtime errors since both versions of the pb.rb file were being loaded.

I've found a few work arounds. Currently I'm creating the directory hierarchy and running multiple compiles, one for each file. Side effect is redundant files which I will then go and delete.

I'm currently working on Mac OS X with ruby 1.8.7, FreeBSD 8.2 with ruby 1.9 and 1.8.7. ruby_protobuf gem is 0.4.11.

Included below is the commands and output from rprotoc and protoc. Output is from FreeBSD 8.2, ruby 1.8.7, ruby_protobuf 0.4.11, protoc version 2.4.1

Related to issues #9, #10.

Directory structure.
code/gbe/pnl

$ ls -alR gbe
total 30
drwxr-xr-x  3 fuzzle  fizzle      5 Dec 28 10:58 .
drwxr-xr-x  5 fuzzle  fizzle     16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle   3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x  2 fuzzle  fizzle      3 Dec 28 10:57 pnl

gbe/pnl:
total 6
drwxr-xr-x  2 fuzzle  fizzle     3 Dec 28 10:57 .
drwxr-xr-x  3 fuzzle  fizzle     5 Dec 28 10:58 ..
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto

# run the compiler on the top level proto file
$ rprotoc -o gbe -p ".;gbe" gbe/Portfolio.proto      
gbe/Statistics.pb.rb writing...
Warning, couldn't test load proto file because of imports
gbe/Portfolio.pb.rb writing...
$ ls -alR gbe
total 31
drwxr-xr-x  3 fuzzle  fizzle      7 Dec 28 10:59 .
drwxr-xr-x  5 fuzzle  fizzle     16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle   6505 Dec 28 10:59 Portfolio.pb.rb
-rw-r--r--  1 fuzzle  fizzle   3913 Dec 28 10:50 Portfolio.proto
-rw-r--r--  1 fuzzle  fizzle   3864 Dec 28 10:59 Statistics.pb.rb
drwxr-xr-x  2 fuzzle  fizzle      3 Dec 28 10:57 pnl

gbe/pnl:
total 6
drwxr-xr-x  2 fuzzle  fizzle     3 Dec 28 10:57 .
drwxr-xr-x  3 fuzzle  fizzle     7 Dec 28 10:59 ..
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto
$ 

# but, now loads won't work because the import statement is:
import 'gbe/pnl/Statistics.proto';

$ ruby18 ./Portfolio.rb | less
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require': no such file to load -- gbe/pnl/Statistics.pb (LoadError)
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
        from ./gbe/Portfolio.pb.rb:155
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
        from ./Portfolio.rb:2

# so, now go run the compiler on the sub-directory

$ rprotoc -o gbe/pnl -p ".;gbe" gbe/pnl/Statistics.proto 
gbe/pnl/Statistics.pb.rb writing...
$ ls -alR gbe
total 41
drwxr-xr-x  3 fuzzle  fizzle      7 Dec 28 10:59 .
drwxr-xr-x  5 fuzzle  fizzle     16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle   6505 Dec 28 10:59 Portfolio.pb.rb
-rw-r--r--  1 fuzzle  fizzle   3913 Dec 28 10:50 Portfolio.proto
-rw-r--r--  1 fuzzle  fizzle   3864 Dec 28 10:59 Statistics.pb.rb
drwxr-xr-x  2 fuzzle  fizzle      4 Dec 28 11:02 pnl

gbe/pnl:
total 7
drwxr-xr-x  2 fuzzle  fizzle     4 Dec 28 11:02 .
drwxr-xr-x  3 fuzzle  fizzle     7 Dec 28 10:59 ..
-rw-r--r--  1 fuzzle  fizzle  3864 Dec 28 11:02 Statistics.pb.rb
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto


# now the program will run, if done as this example
$ ruby18 ./Portfolio.rb 
...correct output...
$

# but, now there are multiple Statistics outputs located in the build system. Some imports will fail because they find both Statistics.pb.rb files
# and the numbers collide.


##### Using the protoc compiler ######

# the initial directory layout
$ ls -alR gbe
total 30
drwxr-xr-x  3 fuzzle  fizzle      5 Dec 28 11:05 .
drwxr-xr-x  5 fuzzle  fizzle     16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle  20480 Dec 28 10:51 .Portfolio.proto.swp
-rw-r--r--  1 fuzzle  fizzle   3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x  2 fuzzle  fizzle      3 Dec 28 11:05 pnl

gbe/pnl:
total 6
drwxr-xr-x  2 fuzzle  fizzle     3 Dec 28 11:05 .
drwxr-xr-x  3 fuzzle  fizzle     5 Dec 28 11:05 ..
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto
[fuzzle@gbe-jamdev-02 ~/work/pjm/sample_code/ruby_protobuf]$ 

# running protoc
$ protoc --cpp_out=. --proto_path . --proto_path gbe  gbe/Portfolio.proto 
$ ls -alR gbe
total 31
drwxr-xr-x  3 fuzzle  fizzle       7 Dec 28 11:11 .
drwxr-xr-x  5 fuzzle  fizzle      16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle  116777 Dec 28 11:11 Portfolio.pb.cc
-rw-r--r--  1 fuzzle  fizzle   76031 Dec 28 11:11 Portfolio.pb.h
-rw-r--r--  1 fuzzle  fizzle    3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x  2 fuzzle  fizzle       3 Dec 28 11:05 pnl

gbe/pnl:
total 6
drwxr-xr-x  2 fuzzle  fizzle     3 Dec 28 11:05 .
drwxr-xr-x  3 fuzzle  fizzle     7 Dec 28 11:11 ..
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto

# NOTE: Above in the directory structure we were putting the output files into . and they ended up in gbe. The protoc
# compiler implicitly assumes that you'll have a directory structure similar to the namespace.
# Also, there is only the output for the file that we asked it to compile, not the file + imports.

# running the command on the statistics file.
$ protoc --cpp_out=. --proto_path . --proto_path gbe  gbe/pnl/Statistics.proto 
$ ls -alR gbe
total 220
drwxr-xr-x  3 fuzzle  fizzle       7 Dec 28 11:11 .
drwxr-xr-x  5 fuzzle  fizzle      16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle  116777 Dec 28 11:11 Portfolio.pb.cc
-rw-r--r--  1 fuzzle  fizzle   76031 Dec 28 11:11 Portfolio.pb.h
-rw-r--r--  1 fuzzle  fizzle    3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x  2 fuzzle  fizzle       5 Dec 28 11:13 pnl

gbe/pnl:
total 7
drwxr-xr-x  2 fuzzle  fizzle      5 Dec 28 11:13 .
drwxr-xr-x  3 fuzzle  fizzle      7 Dec 28 11:11 ..
-rw-r--r--  1 fuzzle  fizzle  55621 Dec 28 11:13 Statistics.pb.cc
-rw-r--r--  1 fuzzle  fizzle  37536 Dec 28 11:13 Statistics.pb.h
-rw-r--r--  1 fuzzle  fizzle   2149 Dec 28 10:48 Statistics.proto

# now the files are created and can be imported. 
# NOTE: we asked for the output to be in '.' and the Statistics files ended up in gbe/pnl which matches the namespace and our import statements.

# Similar output is obtained if you run using python_out.

Test file failed

Test file failed

$ ruby-protoc test.proto
Traceback (most recent call last):
        3: from /usr/local/bin/ruby-protoc:23:in `<main>'
        2: from /usr/local/bin/ruby-protoc:23:in `load'
        1: from /var/lib/gems/2.5.0/gems/ruby-protocol-buffers-1.6.1/bin/ruby-protoc:38:in `<top (required)>'
/var/lib/gems/2.5.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/compiler.rb:15:in `compile': 127 (ProtocolBuffers::CompileError)
$ ruby -v
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]
$ gem -v
2.7.6
$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8

Hi,

I noticed an issue when trying to serialize utf8 characters like á or any other utf8 specific characters.
The error occurs on ruby 2.0.0-p576 (the latest version of the 2.0.0 branch).

Everything seems to work fine on 2.1.1 and the previous 2.0.0 version: (2.0.0-p481)

Here is the smallest reproducible example I came up with:

require 'protocol_buffers'

class User < ::ProtocolBuffers::Message
  optional :int32, :id, 1
  optional :string, :title, 2
end

I seriously have no idea how this is related but doing
User.new(id: 1, title: 'á').to_s will work, but using an integer greater or equal than 128 for id will fail: User.new(id: 128, title: 'á').to_s will fail:

Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
        from /Users/pierre/.rbenv/versions/2.0.0-p576/lib/ruby/gems/2.0.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/encoder.rb:64:in `write'
        from /Users/pierre/.rbenv/versions/2.0.0-p576/lib/ruby/gems/2.0.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/encoder.rb:64:in `serialize_field_value'
        from /Users/pierre/.rbenv/versions/2.0.0-p576/lib/ruby/gems/2.0.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/encoder.rb:51:in `serialize_field'
        from /Users/pierre/.rbenv/versions/2.0.0-p576/lib/ruby/gems/2.0.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/encoder.rb:38:in `block in encode'
        from /Users/pierre/.rbenv/versions/2.0.0-p576/lib/ruby/gems/2.0.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/encoder.rb:15:in `each'
        from /Users/pierre/.rbenv/versions/2.0.0-p576/lib/ruby/gems/2.0.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/encoder.rb:15:in `encode'
        from /Users/pierre/.rbenv/versions/2.0.0-p576/lib/ruby/gems/2.0.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/message.rb:250:in `serialize'
        from /Users/pierre/.rbenv/versions/2.0.0-p576/lib/ruby/gems/2.0.0/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/message.rb:257:in `serialize_to_string'
        from (irb):33
        from /Users/pierre/.rbenv/versions/2.0.0-p576/bin/irb:12:in `<main>'

I gave a quick look at the changelog between 481 and 576 (https://gist.github.com/pjambet/e138dd1d4fa759746e7e) but couldn't find anything interesting.

It definitely looks like a ruby specific bug, but I thought I would at least create the issue here so we could start a discussion about this bug, and then eventually open an issue on the ruby bug tracker. It might also be worthwhile to add compatibility warning in the README.

Let me know if you need more information.

PS: I know that one way of getting rid of this issue would be to use 481, or jump to 2.1.1, but that would be really complicated for me to do that at the moment.

This gem gives different results on PC and Mac

I'm having an issue with parsing the information from a Google Protocol Buffer binary using this gem.

The scenario is:
I have two computers, a Mac and a PC.
They are both running Ruby 2.0.0p481.
They both have the ruby-protocol-buffers gem, version 1.5.1.
The protocol buffer binary file is 219 bytes on both computers, which leads me to believe that their contents are exactly the same.
I have the same ".proto" files on both computers, as well as the same pb.rb files that I am using to interpret the protocol buffer binary.
I wrote a short script that uses the auto-generated pb.rb files to parse the binary file, and have the same copy of the script on both computers.

However, when I run the script that interprets the binary file, the outputs are different on the Mac versus the PC.

When I run the script on the Mac, I get the exact information that I am expecting, which is all the information the protocol buffer contains. However, under the conditions mentioned above, when I run the same script on the Windows machine the output contains significantly less information.

I'm not sure why the output is so different, because I've made sure many of the variables (Ruby version, gem, gem version) are the same in both environments (Mac and PC). I also got the binary file individually on both the Mac and the PC from the same source at almost exactly the same time. I made sure to retrieve the binary file separately on both computers in case this might be an issue with line endings across Mac/PC, but it does not seem to make any difference.

Has something like this been reported before? If so, how was it resolved in the past?

Performance vs JSON

I started looking at Protocol Buffers as a faster message serialization format than JSON. I put together this contrived example which has the same performance characteristics as our production objects. On my

require 'yajl'
require 'oj'
require 'protocol_buffers'
require 'benchmark/ips'
require 'stringio'

hash = {
  'string'  => "String",
  'boolean' => true,
  'int'     => 123,
  'array'   => %w[a b c d e f]
}.freeze

class TestObj < ProtocolBuffers::Message
  required :string, :string, 1
  required :bool, :boolean, 2
  required :int32, :int, 3
  repeated :string, :array, 5

  def to_hash
    {
      "string"  => string,
      "boolean" => boolean,
      "int"     => int,
      "array"   => array
    }
  end
end

def do_pb(obj)
  io = StringIO.new
  obj.serialize(io)
  io.rewind
  TestObj.parse io
end

Benchmark.ips do |x|
  x.report("yajl") { Yajl.load(Yajl.dump(hash)) }
  x.report("oj")   { Oj.load(Oj.dump(hash)) }
  x.report("pb")   { do_pb(obj) }
end

On my i7-3520M CPU @ 2.90GHz dev machine in ruby 2.0, I'm getting the following times:

Calculating -------------------------------------
                yajl      7217 i/100ms
                  oj     19613 i/100ms
                  pb      1513 i/100ms
-------------------------------------------------
                yajl    89614.8 (±3.6%) i/s -     454671 in   5.079974s
                  oj   281144.5 (±3.4%) i/s -    1412136 in   5.028722s
                  pb    15452.5 (±2.8%) i/s -      78676 in   5.095434s

(Essentially the same for ruby 1.9.3-p392).

Is there something obvious I'm missing? Or is the C-library efficiency of Oj just that much better than the pure ruby implementation of protocol-buffers?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.