The official Twirp implementation (for Golang) and spec added a new error code Malformed (status 400: Bad Request) that is returned by the framework when the request is malformed by the client, specially for manual JSON requests.

Golang PR: twitchtv/twirp#182

Ruby Twirp is currently returning error code BadRoute (status 404: Not Found) in those cases. Updating this should be easy and safe, because existing clients don't normally trigger this error. This will affect manual calls more than anything else. However it will be released as a major update because it is fundamentally backwards incompatible.

From a proto file with a package like this:

package code.justin.tv.my_cool_package;

The nested module generated in the .twirp.rb file looks like this:

module Code.justin.tv.myCoolPackage

But it should look like this:

module Code
  module Justin
    module Tv
      module MyCoolPackage

Please consider the following .proto file:

package package.rpc.asks;
import "package/actions/asks.proto";
service Ask {
  rpc GetAskSnapshot(package.actions.asks.RequestAskSnapshot) returns (package.actions.asks.AskSnapshot) { }
}

I would expect this to generate a Ruby file

module Package
  module Rpc
    module Asks
      class AskService < Twirp::Service
        package 'superlist.rpc.asks'
        service 'Ask'
        rpc :GetAskSnapshot, Package::Actions::Asks::RequestAskSnapshot, Package::Actions::Asks::AskSnapshot, :ruby_method => :get_ask_snapshot
      end

however, it drops the package name for the parameter and generates

module Package
  module Rpc
    module Asks
      class AskService < Twirp::Service
        package 'superlist.rpc.asks'
        service 'Ask'
        rpc :GetAskSnapshot, RequestAskSnapshot, AskSnapshot, :ruby_method => :get_ask_snapshot
      end

which results in an error when trying to import the service file. The RequestAskSnapshot is expected as Package::Rpc::Asks::AskService::RequestAskSnapshot which doesn't exists (as it's supposed to be Package::Actions::Asks::RequestAskSnapshot).

Am I missing something?

Hey, this is a small feature request that I'm happy to take on if y'all are good with it.

Context

I was running something like:

curl -XPOST -H "Content-type: application/json" my.application.route/twirp/namespace/RpcCall

and kept running into Invalid request body for rpc method \"RpcCall\". In my case it was because I had a slightly malformed payload, but to figure out what was wrong I had to pop open a rails console and run RpcCall.decode_json(payload) to get the error output.

Proposed Solution

Add the error message from the caught exception here: https://github.com/twitchtv/twirp-ruby/blob/master/lib/twirp/service.rb#L131

Some HTTP clients always append charset to the content-type header e.g., content-type: application/json; charset=utf8. Requests made by those clients are rejected because of the content-type mismatch.

Twirp only allows the content-type header exactly matches one of aplication/json, application/json; strict=true or application/probobuf. This seems too strict, can we relax this constraint?

Thank you.

I recently pulled the latest repo and followed along with the example tutorial. Modified one of the protos to test and started to run code gen when I got this output trying to install the Twirp plugin:

go: downloading google.golang.org/protobuf v1.28.1
go: module github.com/golang/protobuf is deprecated: Use the "google.golang.org/protobuf" module instead.
go: upgraded google.golang.org/protobuf v1.28.0 => v1.28.1

Can I get some help understanding if this is a known plugin issue or a local dependency issue on my part? Thanks.

Protobuf v3.6.0 added support for ruby_package in order to control the module of generated code. It would be great if the Twirp generator supported it too.

Add a .devcontainer/Dockerfile with the dev environment. So we can spin a local dev container and/or codespaces. With that folks trying to contribute don't need to run the environment locally.

The code generation documentation is suggesting to use:
go get github.com/cyrusaf/ruby-twirp/protoc-gen-twirp_ruby
for code generation.

That was the previous ruby client posted on https://github.com/twitchtv/twirp#implementations-in-other-languages but that has since been updated to this library.

This library also seems to include it's own code generation tool at:
https://github.com/twitchtv/twirp-ruby/blob/master/protoc-gen-twirp_ruby/main.go but not suggested.

What should we be using?

We have a Twirp service implemented in Go and are trying to use this client to make requests in our Ruby codebase.

It seems that the Ruby client (I have only tried the protobuf version so far), does not add /twirp before the service name/method when constructing the URL.

As shown in your examples:
https://github.com/twitchtv/twirp-ruby/blob/1cecfd1b7f1990c650a5209de5b93cd1b4a99b37/example/hello_world_client.rb#L6

The user has to add the /twirp path parameter when setting the URL for the client.

This seems to be at odds with the Go client, which seems to construct that path as the Go server expects.

Example:
https://github.com/twitchtv/twirp/blob/ad6d075411e462f08e971d7efc18f4a770ac8b40/example/cmd/client/main.go#L27

As a user, I would expect that both clients operate in a similar manner, and that you would only have to set the address of your server when setting the URL with the client and not have to make sure that you add the /twirp segment.

While working on #26, I wanted to run tests but noticed that rake and minitest were missing from the Gemfile. This seems like it'd be useful to establish a baseline set of dependencies, and their versions, for contributors.

Reviewing a PR to the generator made me realize that we should have end to end tests for the full flow of generating a client/server and driving requests through them. Ideally we could hook this test up to CI so that any PR can easily know if they pass the tests.

I am thinking Docker is the best option to vendor protoc and generate the twirp-ruby files to drive tests through. Ideally we can hook up the generated client/server to the existing unit tests under test/.

twirp-ruby supports faraday 0.x, 1.x and 2.x, we need to make sure we test(ci) all the versions when doing changes.

Expected behavior

Using a proto file with no package declaration should generate Ruby code with no module.

For example this proto file:

syntax = "proto3";

service Hello {
    rpc World(HelloWorldRequest) returns (HelloWorldResponse);
}

Should generate this Ruby code:

# Code generated by protoc-gen-twirp_ruby 1.0.0, DO NOT EDIT.
require 'twirp'
require_relative 'valhalla_pb.rb'

class ValhallaService < Twirp::Service
  service "Valhalla"
  rpc :HelloWorld, HelloWorldRequest, HelloWorldResponse, :ruby_method => :hello_world
end

class ValhallaClient < Twirp::Client
  client_for ValhallaService
end

Actual Behavior (bug)

The code generated has an empty module, which results in a Ruby Syntax Error:

# Code generated by protoc-gen-twirp_ruby 1.0.0, DO NOT EDIT.
require 'twirp'
require_relative 'valhalla_pb.rb'

module
  class ValhallaService < Twirp::Service
    service "Valhalla"
    rpc :HelloWorld, HelloWorldRequest, HelloWorldResponse, :ruby_method => :hello_world
  end

  class ValhallaClient < Twirp::Client
    client_for ValhallaService
  end
end

Failing code

In the generator code, the function packageToRubyModules (https://github.com/twitchtv/twirp-ruby/blob/v1.0.0/protoc-gen-twirp_ruby/main.go#L199) assumes that strings.Split(pkgName, ".") will return an empty array, therefore the modules should be empty. But strings.Split returns an array with an empty string instead ([]string{""}) which is interpreted as 1 module with an empty name, that's why the generated code does the module line. (see example in the Go Playground https://play.golang.org/p/PyfM04lM60e.

Fix

The packageToRubyModules function should check if package is an empty string and return an empty array in that case. Also add a new testcase in main_test.go to check for empty package name.

The latest version, 1.11.0 is not published on rubygems: https://rubygems.org/gems/twirp/versions/

Can you please upload it?

I've got a protobuf which returns an Empty e.g.

syntax = "proto3";
import 'google/protobuf/empty.proto';

service Subscription {
  rpc Subscribe(Subscriptions) returns (google.protobuf.Empty);
}

However, the generated code:

   class SubscriptionService < Twirp::Service
     rpc :Subscribe, Subscriptions, Empty, :ruby_method => :subscribe
   end

Whereas it should be:

   class SubscriptionService < Twirp::Service
     rpc :Subscribe, Subscriptions, Google::Protobuf::Empty, :ruby_method => :subscribe
   end

I'm calling the generator using:

grpc_tools_ruby_protoc --ruby_out=app/rpc --twirp_ruby_out=app/rpc app/proto/*.proto

root@11768bc0a972:/# grpc_tools_ruby_protoc --version
libprotoc 3.5.1

Any ideas what I'm doing wrong?

The line at

The issue can be avoided by checking first: rack_request.body.rewind if rack_request.body.respond_to?(:rewind). I note some discussion on this in the past and not sure what kind of side effects this might have. Happy to submit a PR if desired. I think dealing with it pretty important though, because the error given is very misleading, telling users they have a malformed request when actually it's a rack version issue.

There are also a bunch of updates needed for the example app to work with rack 3 - some functionality has been separated into a separate rackup gem, for example. The example app works if bundle installed from there - but copying its pattern in a newer environment will fail. Again, happy to put it in a PR but I wonder if rack 3 compat is a bigger deal than these issues.

When I generate code from two proto files

// data.proto
syntax = "proto3";
import "ok.proto";

service Data {
    rpc create(SetDataRequest) returns (ok.Response);
}

message SetDataRequest {
    string uuid = 1;
}

// ok.proto
syntax = "proto3";
package ok;

message Response { }

With one protoc execute

protoc --proto_path=. --ruby_out=. --twirp_ruby_out=. --plugin=/Users/a.zimin/go/bin/protoc-gen-twirp_ruby data.proto ok.proto

I have a code with a missing module name of Response class

# data_twirp.rb
# Code generated by protoc-gen-twirp_ruby 1.4.1, DO NOT EDIT.
# ...SKIP...
  rpc :create, SetDataRequest, Response, :ruby_method => :create
# ...SKIP...

module Ok
  Response = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("ok.Response").msgclass
end

With two protoc calls

protoc --proto_path=. --ruby_out=. --twirp_ruby_out=. --plugin=/Users/a.zimin/go/bin/protoc-gen-twirp_ruby data.proto
protoc --proto_path=. --ruby_out=. --twirp_ruby_out=. --plugin=/Users/a.zimin/go/bin/protoc-gen-twirp_ruby ok.proto

Generated code is correct

# data_twirp.rb
# ...SKIP...
  rpc :create, SetDataRequest, Ok::Response, :ruby_method => :create
# ...SKIP...

This PR #82, which was released in 1.10.0, introduced a breaking change and should have warranted a major version bump. Namely, the method signature of ClientResp#initialize was changed in a incompatible way (from positional arguments to keyword arguments).

I'm not sure what the path forward here is. Could revoke 1.10.0 (and re-release as 2.0.0), or add a warning and ask downstreams to pin to <= 1.9 ...

I appreciate it had been quite awhile since a release, and these things happen :) There may be other breaking changes, but that is one I noticed.

Great library! Love the option to lean on protobufs without the weight of grpc.

While following the examples, I thought that a config.ru might flow a bit better, though I'm no expert. I prefer that I can pass options to rackup at runtime, vs hard coding them. I've created a file that does not work, and I'm wondering if maybe you could point me in the right direction?

# config.ru

# Instantiate Service
handler = CardService::Handler.new()
service = CardServiceService.new(handler)

# Mount on webserver
path_prefix = '/twirp/' + service.full_name
server = WEBrick::HTTPServer.new
server.mount path_prefix, Rack::Handler::WEBrick, service

run server

Then loading the server with rackup -o '0.0.0.0' -p '50052' (kept the grpc port =P)

When calling the server, I get

NoMethodError: undefined method `call' for #<WEBrick::HTTPServer:0x000056158d21b828>

This repository is moving!!!

Why?

In a bid to improve security of GitHub's primary organisation https://github.com/github, GitHub is moving all repositories with outside collaborators out of the organisation. This repository is one of those.

When?

There is no specific date, but it will be before the end of May 2023.

What will happen?

The repository will be moved to the new organisation at https://github.com/arthurnn/twirp-ruby (there's nothing there right now).

All current external contributors will retain their access.

Old external contributors will lose their access, if they haven't already.

All PRs should automatically rebase to the new repository location so you shouldn't need to do anything. If they don't please open an issue and I'll try sort things out.

All issues and discussions will be moved across at the same time too.

@arthurnn are you open to others helping maintain this gem? My team has been using this gem heavily across multiple projects and would be happy to help. We have a couple PRs that have been open for many months without any movement. Happy to help share the load, with no intention of making breaking changes.

/cc @gaffneyc @darronschall

Hello! While the project has an Apache 2 license, it should have Apache 2 license headers on all source files (see twitchtv/twirp#9).

Hello, I was wondering if there were plans to expose a service hook in twirp-ruby that's the equivalent of RequestReceived in the go library: https://github.com/twitchtv/twirp/blob/ad6d075411e462f08e971d7efc18f4a770ac8b40/hooks.go#L23-L27

We're looking to hook in to calculate a signature on the encoded payload, so we're currently re-encoding in a before hook for the purposes of calculating the signature.

It seems like the twirp-ruby before hook is equivalent to the RequestRouted hook in Go, so it feels like there should be an equivalent to RequestReceived as well.

Though there are tests, there's no CI to run them. If github actions works for you guys, I"m willing to contribute!

Has anyone ever tried to distribute protoc-gen-twirp_ruby plugin in a compiled form? Maybe even through brew or just binaries?

The fact, that we need to have go locally is a bit of nuance for us.

go install github.com/twitchtv/twirp-ruby/protoc-gen-twirp_ruby

It looks like the latest release in this repository is 1.4.0, but the latest version of rubygems.org is 1.2.0. Would it be possible to publish the missing versions? Thanks.

Hi. I'm not sure if the Twitch team still maintains this package, but I wanted to suggest (and potentially help) with a couple of changes to the Code Generator plugin.

• There should be an options to generate only Services, or Clients (not both)
• There should be options for Module/class naming
• There should be options for the relative filenames that Twirp expects (from protoc --ruby_out)

Let me know if any of this is planned!

Thanks in advance

EDIT: I was also wondering if there are any plans to drop Ruby 1.9.

The Twirp canonical implementation supports both server and client interceptors. This allows us to do similar things for both clients and servers, such as emit specific metrics based on type of request. In the Ruby stack, this facility exists only for the server side. Are there any plans to introduce client interceptors as well?

As a side note, what twirp-ruby calls service hook is actually server interceptor in the canonical lingo. Any reason for this naming choice?

In this commit the protobuf library added the flag "ignore_unknown_fields" to the decode_json method. In an ideal world twirp ruby's Encoding.decode method should set this flag when decoding JSON to make it more like protobuf.

I'm happy to make this change but there's a few ways to implement it based on your preferences so wanted to get your thoughts before making a PR.

My preferred way

This change came out 2 years ago in version 3.6.1. I don't think it's a huge lift to just bump the minimum version of protobuf required given how old the change is. To do this we'd just:

• Change the code to pass the flag in Encoding: when JSON then msg_class.decode_json(bytes, ignore_unknown_fields: true
• Bump the minimum required protobuf version to >=3.6.1 which is where they pushed it.

Make it a flag of some kind when instantiating the service

This option would be fully backwards compatible and require no version bump of protobufs but we'd have to change the service some. A few options could be:

• Change service to take in an options object that has encoding flags. Pass options to the encoding object to specify behavior.
• Bigger change where we make a custom encoding class able to be passed in or provide a hook to override encoding/decoding (might be nice but possibly too much power)

Make it so the client can specify it wants the behavior

This is also backwards compatible but requires fudging the Twirp spec at least in some way. A few options:

• We use the parameter field of content types to add something like 'application/json; ignore_unknown_fields=true'. This is a valid content type according to the rfc. I have no idea if this is common practice but it is pretty neat and probably the least offensive way for us to make this client controllable.
• We allow specifying a header or URL parameter to specify it. Header might get blocked by proxies but is a pretty easy thing to support in the service. A url parameter may go against the twirp spec but generally feels sorta gross and against the spirit.
• Could violate the twirp spec and add a new content type for it 'application/x-json-whatever'. This is a bad option but here for completeness.

Let me know if you have a preference.

Also separately let me know how open you are to some other PRs or what process would be best to chat about them. We use Twirp extensively at Scratch and have a lot of libraries or some support for things I'd love to add upstream. A few of the major candidates:

1. I'd love to upstream some deadline stuff. The twirp spec is a bit light on this but we have some middleware that propagates deadlines and in an ideal world the twirp plugin could parse a default deadline as an option from the RPC service and add it to the rpcdef. This would be a bigger effort but a lot of value if you're open to it.
2. A framework for testing that allows mounting the service using rack in a unit test and making calls (this one is awesome but will require decoupling a bit from our framework). Also some rspec matchers for be_twirp_error, be_twirp_ok, etc...
3. A framework for defining handlers as individual classes, mounting multiple services, etc.. Probably hard to upstream but could add them to a contrib (if not I may just make a standalone gem for it someday)
4. Some extension methods that do things like client.mymethod!(whatever) that raises if there's a twirp error or returns resp.data directly -- I realize you reverted a PR adding a TwirpException earlier which make sense but I do like the bang version of the methods raising since it removes a lot of unit test boiler plate.

It seems like there is a typo or something with the v7.1.2 release (should it be v1.7.2?. This makes the go get github.com/twitchtv/twirp-ruby/protoc-gen-twirp_ruby command always default to the v7.1.2 release even when expecting to get v.1.8.0.

Example:

$ go get github.com/twitchtv/twirp-ruby/protoc-gen-twirp_ruby
go get: upgraded github.com/twitchtv/twirp-ruby v1.7.2 => v7.1.2+incompatible

It's unclear why this is happening as a fresh-clone of the repo (and Github inspection) does not contain a tag for v7.1.2, any insight on what we can do to address this?

Also see: https://libraries.io/go/github.com%2Ftwitchtv%2Ftwirp-ruby which lists the v7.1.2 release.

The twirp-ruby package provides great out of the box hooks (before,on_success,on_error,exception_raised) pre and post the twirp service handler invocation. For most cases, I believe this implementation contains the required functionality to augment the call for a given rpc method, but I have recently discovered a need to yield the execution of an rpc call with Twirp specific routing information.

The Twirp wire protocol uses the POST request method for every rpc call. We have database middleware components in place that translates those Twirp POST request as requiring a :writing connection to the database, which may create unnecessary strain on our DB primaries, when in most cases, only a :reading connection is required, thereby hitting the replicas and reducing pressure on the primaries.

I'm looking for a way to wrap twirp handlers in read-only DB connections utilizing ActiveRecord::Base.connected_to(role: :reading) { ... }

So far, I've managed to come up with an alternative solution by creating a module that selects the appropriate connection role given a list of rpc methods that require :writing connections. This module is then prepended to the Twirp::Service class

  module CallRpcWithDatabaseSelection
    # Override twirp-ruby's `#call_handler` with our own database selection logic.
    def call_handler(env)

        write_connection_methods = @handler.class.write_connections_list
        connection_role = if !write_connection_methods.nil? && write_connection_methods.include?(env[:ruby_method].to_sym)
          :writing
        else
          :reading
        end

        ActiveRecord::Base.connected_to(role: connection_role) do
          # call the definition of #call_handler from the ruby gem
          super
        end
    end
  end

  class Twirp::Service
    prepend CallRpcWithDatabaseSelection
  end

While this accomplishes the functionality I need, it is certainly a bit of a hack.

I was wondering if there was a better/alternative solution that would be recommended to accomplish this task more optimally, or if you think this could be a great use case for an around{ ... } hook or a similar idea that would allow yielding execution of RPC methods with Twirp related information.

This happens when creating a proto property that is optional, and trying to generate code.

<yourProto>.proto: is a proto3 file that contains optional fields, but code generator protoc-gen-twirp_ruby hasn't been updated to support optional fields in proto3. Please ask the owner of this code generator to support proto3 optional.--twirp_ruby_out: 

Do we test the protocol-gen-twirp_ruby ?

1. I see the go bin has a main_test.go file, but I don't see we call it during our action CI build
   Note: cd protoc-gen-twirp_ruby && go test ./... will execute the test
2. We could have a integration test, where from a .proto file we generate the stubs, and asserts against that, to prevent major breaking changes.
3. Idea: bring some of the example app as part of our test suite.

1 is a easy win, 2 and 3 are more like ideas, but the overall goal here is to have tests that trigger the protoc ruby bin.

I was working on a client today and had the natural inclination to raise the response error for any of the cases we weren't sure how to handle yet. Unfortunately that doesn't work very well since Twirp::Error is just an Object and can't be raised.

Would you be open to a PR to refactor Twirp::Error to be a child of StandardError so it could be raised?

The twirp-ruby library serializes JSON as camelcase (even if specified as snakecase), while the golang generator serializes JSON exactly as specified in the proto file.

For example:

message ExampleResponse {
    string example_field = 1;
    string exampleField2 = 2;
}

Golang JSON serialization:

{
    "example_field": "some value",
    "exampleField2": "some value"
}

Ruby JSON serialization:

{
    "exampleField": "some value",
    "exampleField2": "some value"
}

As a total go n00b it took me an embarrassing amount of time to get the toy example app working (regenerating the protos). I suggest two improvements:

1. Better instructions for installing the golang protobuf compiler. The example app points us at https://github.com/golang/protobuf, which may as well be in Klingon to a non-gopher. For mac, all you need to do it seems is brew install protobuf and protoc is magically present (I had it already - the "golang compiler" step was a red herring). Installing golang itself is also, obviously, needed.

2. Golang appears to have its own special path for executables, and go get installs things in ~/go/bin, and if you don't have that in your PATH then the build will fail. For me, it was as simple as adding export PATH=/Users/james/go/bin:$PATH to the bottom of my .zshrc and re-sourcing it.

For those not accustomed to golang ways, these may be rather confusing (they were for me). Thanks!

The Rack app attempts to route the request by reading in the POST body from Rack::Request#body and decoding it on this line: https://github.com/twitchtv/twirp-ruby/blob/c8520030b3e4eb584042b0a8db9ae606a3b6c6f4/lib/twirp/service.rb#L138

Underneath the hood, this is invoking IO#read on the rack.input stream, which forwards the stream to its end. When downstream middleware attempt to re-read from the input stream, they won't get back any data. For exactly this reason, Rack apps that handle the input stream should generally IO#rewind it once they're done reading.

Specifically, we saw this bug cause exceptions when the Honeycomb Rails middleware attempted to parse the HTTP parameters again. See:

• #44
• honeycombio/beeline-ruby#31
• honeycombio/beeline-ruby#39
• honeycombio/beeline-ruby#73

The current constraint is < 2 but the latest faraday as of today is v2.3.0. Is it possible to relax the constraint to < 3?

https://rubygems.org/gems/faraday

Hi!

I've hit an incompatibility between Twirp and the Ruby client for Honeycomb (https://www.honeycomb.io).

The team at Honeycomb have resolved it by ignoring the particular case that I have hit, but I'm concerned that it could be an unusual implementation inside Twirp.

Recording here for your reference, feel free to ignore - I'm recording this issue for future me.

honeycombio/beeline-ruby#39

I think failed_precondition should be renamed to precondition_failed to match the HTTP status name and Rack status name. What do you think?

👋

I recently came across some behavior I wasn't expecting where falsey values are omitted from the json. Here's an example illustrating the issue:

[55] pry(API::Rpc::Groups)> a = MyProtobuf.new(a: true, b: true); b = MyProtobuf.new(a: false, b: false)
...
[60] pry(API::Rpc::Groups)> Twirp::Encoding.encode(a, a.class, 'application/json')
=> "{\"a\":true,\"b\":true}"
[61] pry(API::Rpc::Groups)> Twirp::Encoding.encode(b, b.class, 'application/json')
=> "{}"

The underlying Google protobuf supports additional options, one of which is the emit_defaults flag and including that directly in the encoding call seems to do the trick:

[65] pry(API::Rpc::Groups)> b.class.encode_json(b, emit_defaults: true)
=> "{\"a\":false,\"b\":false}"

I'm thinking we can just support additional pass-through options here: https://github.com/twitchtv/twirp-ruby/blob/master/lib/twirp/encoding.rb#L34. From there, I'm not sure of the best place to allow the caller to configure. Maybe in the rack env?