Right now the generator can only create a single Go file which includes all imports. It would be good to be able to create separate packages from the imports instead of bundling. This would allow for multiple uses of the imports, and it would fix some import namespace issues.

Hi, I notice that when some client fire a rpc request to server, then after receive the response and close the connection, the server just exits the goroutine for that connection, but does not close the connection. i know the gc will collect it for latter use,but i do not think it's good. i think it is just another kind of resource leak.
so i think add the following code in go-thrift/server.go can fix it:

func (c *serverCodec) Close() error {
c.transport.Close()
return nil
}

will you accept it? thanks:)

Also i look at net/http/server.go, it close the socket when a goroutine exits.

i want to use go-thrift in my project, it will become a thrift client to access the java-based thrift server. But now i don't know how to start to write the client with go-thrift and a thrift IDL file.

thx~

cpp_include now causes syntax error

I have writen some service using the code generated by apache official tool,then use go-thrift in the client,but the client failed to call the service.Is it compatible with codes generated by the official tool ?

When I generate a idl file, this file contains include word, then generation is failed, output is like this:

<reader>:1:1 (0): no match found 

Since we have "git.apache.org/thrift.git/lib/go/thrift", this project seems like less useful now!

If it is better to tell us such infos in readme.md

Thanks.

I tried to make a client with go-thrift and communicate with C++ server, however after the client connects to the server, it just hangs there and never return. Though go-thrift is cleaner, I have to switch to thrift4go now. Hopefully go-thrift will add more compatibility tests and become better in the future.

Example:

const map<string, string> Foo = {
  "hello": "it's me"
  "hi": "how are you",
}

Returns:

$ generator example.thrift $GOPATH/src
example.thrift:1:1 (0): rule SyntaxError: parser: syntax error

It works as expected if I add a trailing comma:

const map<string, string> Foo = {
  "hello": "it's me",
  "hi": "how are you",
}

For other RPC clients that use HTTP as the codec, connection pooling is handled at the HTTP Client level. Since go-thrift uses a different codec, how should I tackle connection pooling in a concurrent environment?

A few thoughts:

• The RPC documentation says that an RPC client can be accessed from multiple goroutines (http://golang.org/pkg/net/rpc/#Client). Does this simply imply thread safety, or does it imply pooling should be built in to the codec?
• I've thought about wrapping my calls to access the client in something like this: https://gist.github.com/3504674 - but I wanted to see if there was a lower level place that could handle this for me.

What are your thoughts?

Thanks!

Blake

generate the code from a idl file(hbase thrift idl: http://wiki.apache.org/hadoop/Hbase/ThriftApi), there is a error, imported and not used: "strconv". Then I remove the "strconv" package, but not work well.

runtime: g18: leftover defer sp=0x0 pc=0x0
    defer 0xc208140000 sp=0xc20829fce8 pc=0x776a02
    defer 0xc208416040 sp=0x0 pc=0x0
fatal error: traceback has leftover defers

runtime stack:
runtime.throw(0xa0a070, 0x1d)
    /home/yanyiwu/local/go/src/runtime/panic.go:492 +0x98
runtime.gentraceback(0x776dd0, 0xc20829e570, 0x0, 0xc208092120, 0x0, 0x0, 0x7fffffff, 0xa93290, 0x7f19cde98cc0, 0x0, ...)
    /home/yanyiwu/local/go/src/runtime/traceback.go:414 +0x6dc
runtime.copystack(0xc208092120, 0x4000)
    /home/yanyiwu/local/go/src/runtime/stack1.go:553 +0x187
runtime.newstack()
    /home/yanyiwu/local/go/src/runtime/stack1.go:725 +0xaa4
runtime.morestack()
    /home/yanyiwu/local/go/src/runtime/asm_amd64.s:309 +0x7e

goroutine 18 [copystack]:
github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc208280b0c, 0x7c58e0, 0xc208280ba0, 0xd6)
    /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:51 fp=0xc20829e578 sp=0xc20829e570
github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc208280b0c, 0x7c4f20, 0xc208280b90, 0xd6)
    /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:164 +0x1521 fp=0xc20829ea28 sp=0xc20829e578
github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc20846200c, 0x7c5a00, 0xc208462028, 0xd6)
    /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:164 +0x1521 fp=0xc20829eed8 sp=0xc20829ea28
github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc20841620f, 0x7c7620, 0xc208416250, 0xd7)
    /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:212 +0x272d fp=0xc20829f388 sp=0xc20829eed8
github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc20847a00c, 0x7c59a0, 0xc20847a010, 0xd6)
    /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:164 +0x1521 fp=0xc20829f838 sp=0xc20829f388
github.com/samuel/go-thrift/thrift.(*decoder).readValue(0xc2082808c0, 0xc20847a00c, 0x8bd740, 0xc20847a010, 0xd9)
    /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:164 +0x1521 fp=0xc20829fce8 sp=0xc20829f838
github.com/samuel/go-thrift/thrift.DecodeStruct(0x7f19cf032dc0, 0xc208047680, 0x7c5760, 0xc20847a010, 0x0, 0x0)
    /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/decoder.go:43 +0x460 fp=0xc20829fda8 sp=0xc20829fce8
github.com/samuel/go-thrift/thrift.(*clientCodec).ReadResponseBody(0xc2080a21b0, 0x7c5760, 0xc20847a010, 0x0, 0x0)
    /home/yanyiwu/golang/src/github.com/samuel/go-thrift/thrift/client.go:159 +0xa5 fp=0xc20829fe00 sp=0xc20829fda8
net/rpc.(*Client).input(0xc208094120)
    /home/yanyiwu/local/go/src/net/rpc/client.go:141 +0xa62 fp=0xc20829ffd8 sp=0xc20829fe00
runtime.goexit()
    /home/yanyiwu/local/go/src/runtime/asm_amd64.s:2403 +0x1 fp=0xc20829ffe0 sp=0xc20829ffd8
created by net/rpc.NewClientWithCodec
    /home/yanyiwu/local/go/src/net/rpc/client.go:201 +0xdb

Does anyone else run into this bug?

For better compatibility with the standard parser should parse the rarer bits of the IDL.

https://thrift.apache.org/docs/idl

I've generated a few thrift clients with this lib. Working good so far. But I have an instance where I've needed to modify the generated code. It's related to the "requiredness" of fields on a struct.

My example

Using this library, I change this Thrift struct

/**
 *  Audit response status object.
 *
 *  @param success. True if audit request completed successfully, otherwise set to false
 *  @param messages. List of any error or warning messages that may have occurred
 */
struct AuditResponse {
    1: string status,
    2: list<string> messages
}

...into this Go struct.

type AuditResponse struct {
    Status   string   `thrift:"1,required" json:"status"`
    Messages []string `thrift:"2,required" json:"messages"`
}

The service I'm communicating with often omits Messages, so I get an error when I call an endpoint that returns an AuditResponse, due to (I think) Messages being interpreted as required.

If I hack the generated code by changing the requiredness, an error no longer occurs.

type AuditResponse struct {
    Status   string   `thrift:"1,required" json:"status"`
    Messages []string `thrift:"2,optional" json:"messages"` // NOTE: Modified.
}

Should there be a "default" requiredness?

According to this documentation from Apache, there is a requiredness type called "default", which is implicit if fields in Thrift IDL are not explicitly defined as "required" or "optional".

I change the generated code from "required" to "optional", because I need the Read behavior of optional, which the "default" requiredness retains.

Is there a reason this is not implemented?

I'm trying to create an RPC service and expose it via go-thrift so that it supports a Go-RPC style method which takes args and a reply and returns an error. Critically, the modified reply struct should be available at the client. I don't think go-thrift currently allows this, but in a previous discussion, you hinted it could; maybe I'm doing something wrong.

type Cave struct{}

func (self *Cave) Echo(args *echoer_thrift.EchoArgs, reply *echoer_thrift.EchoReply) (error) {
    reply.Echo = args.Message
    return nil
}

And the Thrift definition,

struct EchoArgs {
    1: required string Message
}

struct EchoReply {
    1: required string Echo
}

service EchoerThriftface {
    void Echo(
        1: EchoArgs args
        2: EchoReply reply
    )
}

In a client,

var args = echoer_thrift.EchoArgs{}
args.Message = "hello!"
var reply echoer_thrift.EchoReply

conn, _ := net.Dial("tcp", ":1234")
thriftClient := thrift.NewClient(thrift.NewFramedReadWriteCloser(conn, 1024), thrift.NewBinaryProtocol(true, true), false)
client := &echoer_thrift.EchoerThriftfaceClient{thriftClient}
error := client.Echo(&args, &reply)

fmt.Println(reply.Echo)   // reply is EMPTY !!!

Looking at the code generated by go-thrift, it makes sense to my why this occurs. This is the expected behavior right?

If I write idl like this:

struct  Example 
{
    1: int a,
    2: int b,
}

a error return when parsing.

otherwise, I write like this:

struct Example {
    1: int a,
    2: int b,
}

It will be ok.

How about support both two style ?

typedefs are parsed but not generated

I'm using this version of Hbase.thrift and ran:

$ generator -go.pointers=false Hbase.thrift src/

This apparently works, but when I try to compile my project with the generated package, I get a bunch of mismatched type errors:

# hbase
src/hbase/hbase.go:580: cannot use val (type int32) as type *int32 in assignment
src/hbase/hbase.go:591: cannot use val (type int32) as type *int32 in assignment
src/hbase/hbase.go:602: cannot use val (type int32) as type *int32 in assignment
src/hbase/hbase.go:613: cannot use val (type int32) as type *int32 in assignment
src/hbase/hbase.go:624: cannot use val (type int32) as type *int32 in assignment
src/hbase/hbase.go:635: cannot use val (type int32) as type *int32 in assignment
src/hbase/hbase.go:1826: cannot use res.Value (type *int32) as type int32 in assignment
src/hbase/hbase.go:1848: cannot use res.Value (type *int32) as type int32 in assignment
src/hbase/hbase.go:1869: cannot use res.Value (type *int32) as type int32 in assignment
src/hbase/hbase.go:1889: cannot use res.Value (type *int32) as type int32 in assignment
src/hbase/hbase.go:1889: too many errors

Currently a uint64 is used to track required fields. This limits a struct to fields with ID < 64. Ideally this limit should be removed.

It seems that the necessary metadata for github.com/samuel/go-parser has not been setup properly, for go getting.

Go packages can't be nested but you can nest namespaces in Thrift. If you have two Thrift files with the same "leaf" namespace name then when both namespaces are used together the generated Go code won't compile. The error is redeclared as imported package name ...

Eg for this thrift:

  include "Person.Address.thrift"
  include "Computer.Address.thrift"

  struct Details
  {
	1 : optional Person.Address.Primary addr,
	2 : optional Computer.Address.Primary ip,
  }

the Go code looks something like this:

  import (
	"Person/Address"
	"Computer/Address"     // Address redeclared as imported package name
	"git.apache.org/thrift.git/lib/go/thrift"
  )
  .....
  type Details struct {
	Addr             *Address.Primary
	Ip               *Address.Primary
  }

This is not a big problem but could be avoided by importing the package under a non-conflicting name like this:

  import (
	PersonAddress   "Person/Address"
	ComputerAddress "Computer/Address"
	"git.apache.org/thrift.git/lib/go/thrift"
  )
  .....
  type Details struct {
	Addr             *PersonAddress.Primary
	Ip               *ComputerAddress.Primary
  }

If you pass a filename with a dash (or other invalid package name) the generated golang file has an invalid package:

$ cat somethrift-v1.thrift
struct Result {
    1: optional binary value
}
$ generator somethrift-v1.thrift somethrift.go
$ cat somethrift.go
// This file is automatically generated. Do not modify.

package somethrift-v1

import (
    "fmt"
)

type Result struct {
    Value []byte `thrift:"1" json:"value"`
}
$ go run somethrift.go
package :
somethrift.go:3:19: expected ';', found '-'

This can be overridden, but I feel like the generator utility shouldn't generate invalid Go code.

The official Thrift compiler has a Go specific flag called package_prefix:

$ thrift --help
  go (Go):
    package_prefix= Package prefix for generated files.
    thrift_import=  Override thrift package import path (default:git.apache.org/thrift.git/lib/go/thrift)
    package=  Package name (default: inferred from thrift file name)

My problem is this. So I have two spec files:

foo.thrift

include "common.thrift"

struct Foo {
    1: string name
}

common.thrift

exception InternalServiceError {
    1: string message
}

So when I generate this using $ generator foo.thrift rpc my foo.go will contain a broken/bad import path..

// This file is automatically generated. Do not modify.

package foo

import (
    "common"    // <---- Bad import!
    "fmt"
)

var _ = fmt.Printf

type Foo struct {
    Name string `thrift:"1,required" json:"name"`
}

This import should ideally be something illustrated by this diff:

 import (
-    "common"
+    "github.com/renstrom/my_project/rpc/common"
     "fmt"
 )

Are am I using this all wrong?

According to this doc:

https://diwakergupta.github.io/thrift-missing-guide/#_types

The Thrift byte type is a signed type. My understanding is that Go's byte is an alias to unsigned uint8.

Using go-thrift, I get the following transformation:

Thrift method signature

    AcmResponse buildAcm(1: list<byte> byteList, 2: string dataType,
                       3: map<string, string> propertiesMap)
        throws(1: InvalidInputException ex1, 2: SecurityServiceException ex2),

Interface go-thrift generated

BuildAcm(byteList []byte, dataType string, propertiesMap map[string]string) (*AcmResponse, error)

The Apache go thrift library generated the following, as a reference:

BuildAcm(byteList []int8, dataType string, propertiesMap map[string]string) (r *AcmResponse, err error)

I noticed this when interoperating with a Java service via Thrift. In Java, the byte type is a signed 8 bit integer.

Example:

struct Foo {
  1: optional list<string> Bar = 
  [
    "hello",
    "world"
  ]
}

Returns:

$ generator example.thrift $GOPATH/src
example.thrift:1:1 (0): rule SyntaxError: parser: syntax error

This only happens if you're defining a list within a struct, and doesn't happen with list const.
Example:

const list<string> Bar = 
[
  "hello",
  "world"
]

Works as expected.

Currently when I try to parse a Thrift file using union, I'll get the following error messages:

2015/08/17 10:37:41 Could not parse .thrift file: <reader>:164:1 (3915): rule SyntaxError: parser: syntax error

By some search, below is a real world example Thrift file using union quite a lot. Would be nice to also add this to test files.
https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift

Prefixing method names with "Thrift." causes Scribed to get confused and unable to talk to the Go Thrift server. I modified server.go so that it removes the prefix from the response and it works now but I don't know enough about Thrift to tell whether this is a good solution.

With /tmp/example.thrift containing:

const i64 H = 0x2603;

the go-thrift parser barfs:

$ ./gen /tmp/example.thrift /tmp/example/
<reader>:1:1 (0): rule SyntaxError: parser: syntax error

This is permitted by the official thrift parser, however:

$  thrift -o /tmp/example/ --gen go /tmp/example.thrift
$  grep 'const H' /tmp/example/gen-go/example/constants.go
const H = 9731

I've had a go and fixing this, so I will have a PR open momentarily..

When using a fresh enough golang compiler

$ go version
go version go1.12.7 linux/amd64

there is the following compilation error:

vendor/github.com/samuel/go-thrift/examples/scribe/thrift.go:63:9: e declared and not used

https://github.com/samuel/go-thrift/blob/master/examples/scribe/thrift.go#L63 indeed declares the variable e

func (s *ScribeServer) Log(req *ScribeLogRequest, res *ScribeLogResponse) error {
	val, err := s.Implementation.Log(req.Messages)
	switch e := err.(type) {
	}
	res.Value = val
	return err
}

which then is never used.

Hi, tried to use this library (after some problems with Apache's one), and ran into a problem with binary codec. I was looking at the source code and this place looks suspicious to me (sorry, I don't know the binary format spec, so it may be completely off the mark):

if req != 0 {
    for i := 0; req != 0; i, req = i+1, req>>1 {
        if req&1 != 0 {
            d.error(&MissingRequiredField{
                StructName: v.Type().Name(),
                FieldName:  meta.fields[i].name,
            })
        }
    }
}

Notice how i is never modified here. You also don't seem to need the outer if, besides, req = i+1 is the same as req = 1, but since you always discard the last bit, this operation has no effect on the following req & 1 != 0 test. I think there must be some kind of a typo here, but I cannot tell where exactly.

struct ResponseSysInfo {
  10: Response  response,
  20: CpuStats  cpu,
  30: i64       total_mem,
  40: i64       available_mem,
  50: i64       uptime,
  60: string    kernel,
}

This is Thrift description of the message I was trying to parse when I encountered this error: reading body thrift: missing required field: ResponseSysInfo.Cpu.

Is this project abandoned?

There seem to be an awful lot of issues with no responses and PR's that have been open for many months now..

If namespace contains digit letter. generate will throw error. I find the regular expression about namespace is ns:[a-zA-z.]+ in grammar.peg file. Should we change it to ns:[a-z0-9A-z.]+? Do you think the namespace word contains digit letter is normal.

I don't understand why the thrift type set<T> is mapped to the Go type map[T]interface{}.

I think that map[T]struct{} is more appropriate -- this map uses 0 bytes for the value and is the most space-efficient representation of a set using Go maps.

An alternative that's a little more convenient is map[T]bool.

Hi, samuel. I'm new learner of thrift.
Can you offer the origin thrift file, So people can study more quickly. thanks.

I'm not sure if this is even possible with PEG, but Method comments are not parsed currently. The only reason I'm creating an issue is because Method has a Comment field which appears to not be used. It would be great if the parser could capture docstrings. The normal Thrift compiler includes these in generated code.

Example:

const map<string,string> M1 = {"hello": "world", "goodnight": "moon"}

The parser chokes when it gets to the {.

I would expect this to yield the following Go, I think:

var M1 = map[string]string{"hello": "world", "goodnight": "moon"}

I'm currently working on fixing this, but any pointers you could give would be great.

It would be nice to have the standard Marshal/Unmarshal functions available.

IMO, the only place that it is appropriate to use flag variables is in a main function. From that point onward they should be passed to the objects as configuration.

Is there any reason that the flag variables flagGoImportPrefix is not set in main and passed to the generator?

Right now the error shown to the user when you try to generate from an invalid thrift file only says the offset (not the filename, line, or column).

I've fixed this issue on my branches of go-thrift and go-parser. I don't have time right this minute to formulate them into a proper pull request, and I'm not sure you'd want to do it this way anyway (it requires changes to both of the repos together).

You can see what I did here:

ooyala/go-parser@9a7e0d6
ooyala@997808d

thrift idl: http://wiki.apache.org/hadoop/Hbase/ThriftApi

server is writen by java. client use go-thrift to call a remote method will be blocked. the method can't return.

I found it in Go standard library:
// Call invokes the named function, waits for it to complete, and returns its error status.
func (client *Client) Call(serviceMethod string, args interface{}, reply interface{}) error {
call := <-client.Go(serviceMethod, args, reply, make(chan *Call, 1)).Done
return call.Error
}

client is blocked here. the length of the "Done" channel is 0, so will block here. But I don't know why the "Done" is empty.

Hope resolve it...
thx~

IDL:
service Calculator extends shared.SharedService

is generate
GO:
type Calculator interface {
Shared.SharedService
Add(num1 int32, num2 int32) (int32, error)
Calculate(logid int32, w *Work) (int32, error)
Ping() error
}

this Shared.SharedService is upper so in this code is not run.

tutorial

.\tutorial.go:7: imported and not used: "shared"
.\tutorial.go:90: Shared is not a package
.\tutorial.go:90: undefined: SharedService
.\tutorial.go:97: Shared is not a package
.\tutorial.go:149: Shared is not a package