just like .. https://github.com/horizon-games/protoc-gen-twirp_ts

lets write gen/typescript generator just for the client

Is it somehow possible to generate code compatible with Aero framework https://github.com/aerogo/aero

i.e:

I have a field called MYID. The generated ts/js code will downcase (camelcase) that field to mYID, which is unintended.

so.. I wrote an example by hand, inspired by twirp+grpc+other here: https://github.com/webrpc/webrpc-go/tree/master/_example/proto

the goal is to use this file: https://github.com/webrpc/webrpc-go/blob/master/_example/proto/example.webrpc.json

and generate .. https://github.com/webrpc/webrpc-go/blob/master/_example/proto/proto.gen.go file

by triggering the go generate command on this file, https://github.com/webrpc/webrpc-go/blob/master/_example/proto/proto.go#L1 which in turn will execute webrpc-gen with -target=go with -client and -server flags

I think the best approach is to find some templating pkg like text/template or something else, and copy the proto.gen.go (made by hand), and templatize it, and use the schema/ package I wrote to generate Go client/server for webrpc

Let's write the code-generator inside of the gen/ pkg

if anything is unclear, please let me know

Right now only basic types are supported

I’m looking for a bidirectional rpc over websockets. How tied to plain http transport is this lib?

Streaming would also be nice. But I see you already have an issue for it.

Also, did you consider using JSON-RPC standard? I am hesitant at the lock-in that comes from a custom rpc standard. Are there advantages of your protocol over this?

In the same vein as twitchtv/twirp#161

ridl:

message MyMessage
  - myField: map<uint32, map<uint32,string>>

generated TypeScript

export interface MyMessage {
  myField: {[key: uint32]: {[key: number]: string}}
}

the design of webrpc is to HTTP-POST to /rpc/<serviceName>/<rpcMethod> a structured JSON string to the webrpc server. Great.. but, as the input argument is sent as the request body, can nginx/CloudFlare/other services use that request body to generate a cache-key needed for basic cache-control rules?

the standard way is to use the URI + query string to generate a cache string -- but, right now, the URI/query string would be the same for all requests in webrpc. But certainly some method calls could be cacheable, so let's make sure we can support it as its the easiest first step to scalability. Some ideas how to make these rpc requests cachable..

1. out of the box support? maybe web servers like nginx already support a way to generate a cache-key for a request body?
2. CloudFlare workers? CF rules, and their new workers stuff looks very flexible, we could probably make it work there
3. Add a request mode in webrpc server to take the request as a query string, which would bring us back to normal http request world, and that could be okay but less ideal then option 1 or 2 above

Any advice/ideas on how to set a cookie via a webrpc handler when the generated typescript/javascript code calls an endpoint? Eg. api.Login() on success sets a browser cookie.

export interface WebRPCError extends Error {
  code: string;
  msg: string;
  status: number;
}

export const throwHTTPError = (resp: Response) => {
  return resp.json().then((err: WebRPCError) => {
    throw err;
  });
};

This is all fine and dandy, but if the response is anything that isn't json serializable, this causes an error to be thrown when calling resp.json(). I think a more suitable method of handling an error would be to first attempt json deseraialization, and then if it throws an error, throw a new error with either the response body/ status code.

Would it be useful to be able to enumerate all possible errors for an API in the RIDL file?

This way API consumers could have a single place to learn everything they need to know about the API. Right now RIDL helps amazingly with that, except that it doesn't "contractualize" errors, so they have to be documented separately.

Also, it would be nice to be able to return multiple errors from an API. An example of an API that could benefit from this is an API that handles a form submission. Instead of returning errors one by one, it could include all of them in the first response so that the frontend can prompt the user to fix all of them before submitting the form again.

Hi,
I've been evaluating this project and noticed that this kind of tag value doesn't parse anymore:

    - CreatedAt: timestamp
        + go.tag.pg = default**:**now**()**,use_zero

The code seems to be:

and gives

parse error: "unexpected token" near ":" (line: 14, col: 30)

save this one for v2, but some ideas below.

for v1, we will just use a json file like: https://github.com/webrpc/webrpc-go/blob/master/_example/proto/example.webrpc.json

implementing the schema at https://github.com/webrpc/webrpc-go/blob/master/schema/README.md

note, the data types of webrpc are inspired by protobuf's, Go, TypeScript and WASM :) intended to be serialized as JSON

• gen/golang -- make sure to generate json struct tags always, even if we don't define them in meta

• gen/golang -- add go.tag.json override, might already work for free..

• enum error message is on wrong line -- enum Blah which misses : type should error on that line .. or set a default of uint32

• bug, defining dupes of messages in RIDL, doesnt return an error?

  • ie. message Blah; - x: string; message Blah; -x: string

Google's Flutter framework is becoming very popular and A Webrpc generated Dart client will greatly help Flutter/Dart developers to build cross platform apps with Webrtc. I request you to please consider this proposal as a priority.

we like clean and simple JSON for serialization. But, how can we include a binary data field within a request or response object...? such as a small image, or something small. Would be interesting to think about chunk`ing responses too..

maybe https://github.com/PsychoLlama/bin-json has some ideas.

Twirp has introduced support for client hooks in latest release https://github.com/twitchtv/twirp/releases for logging, monitoring and authorization.

Currently, we need to use a router and middle wares to perform above mentioned tasks on the server. I think the approach used by Twirp is much better, no need to use another package to provide routing/middleware on the server and hooks are also very useful on the client.

• error codes and messages, similar to gen/golang
• formatting, hopefully some library out there to help clean up some of this formatting for generic source so it doesnt looks so bad. Or the templating system we use, Go's text/template could have some other useful notation/other to help with clean formatting easier during generation

only allow whitelisted expression

please fix: https://github.com/webrpc/webrpc-go/blob/master/schema/parser.go#L35-L41

A common pattern is to use pointers for values that might be null, say:

type Device struct {
	ID       int64
	DeviceID string `pg:",unique,notnull"`
	CreatedAt time.Time `pg:"default:now(),use_zero"`
	DeletedAt *time.Time
        ...

Could this make any sense in webrpc?
Also, in general, shuold we reuse the types that webrpc-gen generates when dealing with the database/persistence side?

I wonder if we can run goimports from webrpc-gen directly, so we dont have to run it as separate process.. something we can integrate directly since we're in Go pipeline anyways.. prob would be easy

First off, huge thanks to this library!! It looks like exactly what I was looking for after evaluating Twirp, et al. I love that it is simple, and right to the point with little dependencies and easy to grep the source. Keep up the good work.

Although RIDL is fairly self documenting (good job), has there been any thought into a doc generator with outputs eg. swagger, etc? Just interested in your thoughts/recommendations as I was thinking about potentially writing a generator to build docs we could publish for our API.

As we have a well-defined schema, we have an opportunity to generate optimized JSON encoders/decoders for each different type that performs much better than Go's std "encoding/json". The std "encoding/json" uses the reflection to marshal/unmarshal in a flexible way, and its great, but produces extra garbage and costs more computer time then if we code-generated specific encoders for each message/type. See https://github.com/pquerna/ffjson as a viable tool we could use in gen/golang and offer 2x to 3x faster serialization performance by their benchmarks.

If I have an rpc method like this...

  - Delete(id: string) => ()

... then the client always fails (after actually succeeding server-side) because the generated code looks like this:

      return buildResponse(res).then(_data => {
        return {
        }
      })

And buildResponse always tries parse the JSON body, which is empty.

An easy workaround is to just return something from the rpc call:

  - Delete(id: string) => (ok: boolean)

note: My server-side is Go and my client is TS.

An import directive could help with a few things

1. Schema organization -- split definitions up into multiple files.

2. Internal service routing and proxying -- to have a front-end API service that can communicate to internal services just from the schema design

a 404 error is confusing, and could mean the endpoint doesnt exist

allow messages to be defined in any order in the json or ridl schemas

It might be better to use known format as IDL.

Since ID is the recommended spelling for identifier fields in Go, just wanted to let you know that a RIDL file like:

message DeviceInfo
    - ID: string

encodes to:

type DeviceInfo struct {
	ID        string     `json:"iD"`

instead of id. DeviceID encodes correctly to deviceID, though.

I got this error on my program when using a struct as function parameter and then I saw the same situation in the golang-basics example.

You can reproduce the error by running the example:

cd _examples/golang-basics
go run .

And when you try to find a user, it issues a fatal error:

curl -v -X POST -H"Content-Type: application/json" -v -d '{"q":"123"}' http://localhost:4244/rpc/ExampleService/FindUser

I figured out that the error is in the serveFindUserJSON function on the generated Go file example.gen.go:

// line 350
reqContent := struct {
  Arg0 *SearchFilter `json:"s"`
}{}

// line 362
err = json.Unmarshal(reqBody, &reqContent)

It's trying to unmarshall Arg0 from the received json, that's why it's crashing.

To fix it I just added .Arg0 to reqContent:

err = json.Unmarshal(reqBody, &reqContent.Arg0)

(The crash itself is not caused by this error, it's due to the lack of proper checking on the received parameters at the FindUser function.)

gRPC returns 1 input argument and 1 output argument, and we've gone with this 0 or 1 input args and 1 output arg in webrpc first implementation. But it should be easy to support multiple input arguments. Something to do later..

Currently javascript engine which parse JSON does not support Int64 and Uint64 and it may overflow and represent a wrong value. Have you thought about this issue?

for ..

• PkgName
• Client on/off
• Server on/off

we need to implement all the flags such that it works as.......

-schema     -- webrpc schema file
-pkg        -- package name used for target language,
-target     -- target language: go, ts, ...
-client     -- flag, code-generate client
-server     -- flag, code-generate server
-out        -- Directory to output the ie. <pkg>.gen.go file, if empty will print to stdout

sample usage would be.......

webrpc-gen -schema=example.webrpc.json -target=go -pkg=proto -server -client -out=./proto

It would be great, If we can define data validation constraints in schema for each message field and then generate code for validating message fields.

I've spent a lot of time thinking about a suitable design to support streaming support for webrpc that will also work with browsers.

The approach is to use chunked transfer encoding for request or response bodies to handle streaming, https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

As for RIDL support, it could look like below:

service ApiService
  - Ping() => (status: bool)

  # streaming request body, single response
  - stream Upload(base64: string) => (status: bool)

  # single request, streaming response body
  - Download(file: string) => stream (base64: string)

  # steaming request body and response body,
  - stream Bidirection(msg: string) => stream (msg: string)

this is a slight difference from what is currently implemented, which looks like

- Download(msg: string) => (msg: stream string)

however, moving stream at the front of input or output definition is the better approach since the entire request or response payload would need to be streamed.

as for the update to the schema, the schema/service.go structure should change to..

type Service struct {
	Name    VarName   `json:"name"`
	Methods []*Method `json:"methods"`

	Schema *WebRPCSchema `json:"-"` // denormalize/back-reference
}

type Method struct {
	Name    VarName           `json:"name"`
	Stream  string            `json:"stream"` // "input", "output", "input-output"
	Inputs  []*MethodArgument `json:"inputs"`
	Outputs []*MethodArgument `json:"outputs"`

	Service *Service `json:"-"` // denormalize/back-reference
}

type MethodArgument struct {
	Name     VarName  `json:"name"`
	Type     *VarType `json:"type"`
	Optional bool     `json:"optional"`

	InputArg  bool `json:"-"` // denormalize/back-reference
	OutputArg bool `json:"-"` // denormalize/back-reference
}

notice that Stream moved from MethodArgument to Method. As well, its defined as a string now, with values of "input", "output" or "input-output" to define the pipes that support streaming for the method.

as for implementation for Go server/client, or other languages a few notes:

1. Browsers buffer the first 1kb of chunked response, see https://gist.github.com/CMCDragonkai/6bfade6431e9ffb7fe88 -- a good idea is then to send 1024 bytes of random data, and make sure client is aware of that to omit it

2. 0\r\n\r\n defines the terminal signal that the chunked body is at EOF

3. For response streaming, it's good practice to send \r\n every 10 seconds or so to keep connections alive, and clients to support this

For the Go implementation, the interface for above methods could be..

type ApiService interface {
  Ping(ctx context.Context) (bool, error)
  Upload(ctx context.Context, stream UploadReader) (bool, error)
  Download(ctx context.Context, file string) (DownloadWriter, error)
}

type UploadReader interface {
  Done() <-chan struct{}
  Read() (UploadRequestBody, error)
}

type DownloadWriter interface {
  Done()
  Write(v DownloadResponseBody, err error) error
}

type DownloadResponseBody struct {
  Base64 string `json:"base64"`
}

BTW. Thank all you guys for creating the go-chi. 😃 I really like the minimalist vibe of it.

I don't have any messages with time.Time type in my ridl file, but time is imported anyway.

The ridl file
https://gist.github.com/pongsanti/6c38c1d28bb41c70d6d639b1e05657b1#file-schema-ridl

command:
webrpc-gen -schema=schema.ridl -target=go -pkg=main -server -out=./schema.gen.go

The generated file
https://gist.github.com/pongsanti/6c38c1d28bb41c70d6d639b1e05657b1#file-schema-gen-go

The Concise Binary Object Representation (CBOR) http://cbor.io/ is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation.

CBOR is based on the wildly successful JSON data model: numbers, strings, arrays, maps (called objects in JSON), and a few values such as false, true, and null. It has support for many languages.

Please consider CBOR for faster serialization.

If there is any whitespace chars on a blank line in .ridl file, it causes webrpc-gen to just hang and churn CPU, never exiting.

You can recreate by adding a space to line 2 of https://github.com/webrpc/webrpc/blob/master/_examples/hello-webrpc/hello-api.ridl

and try to run make generate.

in case it matters. I was running on macOS 10.14.5.

A RIDL spec like the following:

message User
  - id: uint64
  - isAdmin: bool
  - createdAt: timestamp
  - email: string
  - emailVerified: bool
  - passwordHash: string
    + json = -

will generate the following Javascript, instead of ignoring the field:

export class User {
  constructor(_data) {
    this._data = {}
    if (_data) {
     ...
      this._data['emailVerified'] = _data['emailVerified']
      this._data['-'] = _data['-']
    }
  }
  ...
  get emailVerified() {
    return this._data['emailVerified']
  }
  set emailVerified(value) {
    this._data['emailVerified'] = value
  }
  get -() {
    return this._data['-']
  }
  set -(value) {
    this._data['-'] = value
  }
 ...
}

Replacing json with go.tag.json is a temporary workaround, IIUC.