I define a struct like this.

type Server struct {
    hostName string `msgpack:"hostName"`
    hostId   uint16 `msgpack:"hostId"`
}

But I use msgpack.Marshal return one byte??
I read the code in typeinfo.go, the Fields is a empty array... why?

And, How can I use it??

Hey Team -

I have a bit of code that receives JSON (in JSONAPI format) from a public API, builds our Go Structs, and later after doing some sanitization marshals the Go structs back into JSON and stores them as a file. Later, we do the reverse, fetching the JSON files from disk, Unmarshaling them back into Structs, then convert them to JSONAPI format.

The process then looks like:

Store

HTTP REQUEST BODY -> jsonapi.UnmarshalPayload -> GO -> json.Marshal -> Disk

Retrieve

Disk -> json.Unmarshal -> GO -> jsonapi.Marshal -> HTTP RESPONSE BODY

Using JSON as a storage format has always been a temporary placeholder to something more space efficient. We'd been planning to swap the stock JSON library for MSGPACK for some time, and finally got to it last night. It took about 5 minutes to swap the code over. Very sweet!

Once completed, I ran into an unexpected problem however. For some reason, I'm getting back a different object from msgpack.Unmarshal than I previously got from json.Unmarshal. Specifically, somewhere in our (very large) object we have a map[interface {}]interface {} where previously we had a map[string]interface {}.

Am I missing something obvious? I had assumed the marshal/unmarshal process would recreate the same object.

define StringAlias as follow:
type StringAlias string

then encode []StringAlias will panic.

test code:

package main

import (
        msgpack "gopkg.in/vmihailenco/msgpack.v2"
)

type StringAlias string

func main() {
        msgpack.Marshal([]StringAlias{StringAlias("sa")})
}

i notice the following
the encoding for a nested map[string]interface{} type then decoding it
it give me a map of type map[interface{}]interface{}
which led me to be non json compatible

is that right for msgpack spec. or what?

like omitempty

Trying to run the following code:

package main

import "github.com/vmihailenco/msgpack"
import "fmt"
import "unsafe"
import "reflect"

type Test struct {
    A uint16
    B uint16
    C string
    D [4]string
    E [2]int64
}

func main() {
    test := Test{
        A: uint16(1),
        B: uint16(25565),
        C: "yolotest",
        D: [4]string{"a", "b", "c", "d"},
        E: [2]int64{int64(1348293849324), int64(4345345345)},
    }
    buf, err := msgpack.Marshal(test)
    if err != nil {
        fmt.Printf("%s\n", err)
    }
    fmt.Printf("%s\n", buf)
    fmt.Printf("%s\n", len(buf))
    fmt.Printf("%s\n", unsafe.Sizeof(test))

    newzy := Test{}
    msgpack.Unmarshal(buf, &newzy)
    fmt.Printf("%s\n", newzy)
    fmt.Printf("%s\n", unsafe.Sizeof(newzy))
}

I get the following error:

panic: reflect: call of reflect.Value.IsNil on array Value

goroutine 1 [running]:
reflect.Value.IsNil(0x46ca98, 0xf840000e18, 0x112, 0x4aa070, 0x46ca98, ...)
    /usr/lib/go/src/pkg/reflect/value.go:895 +0x130
github.com/vmihailenco/msgpack.(*Encoder).encodeSlice(0xf84002f220, 0x46ca98, 0xf840000e18, 0x112, 0x1, ...)

This happens both when trying to compact a struct, and a single slice like:

func main() {
    test := [4]string{"a", "b", "c", "d"}
    buf, err := msgpack.Marshal(test)
    fmt.Printf("err: %s\n", err)
    newzy := make([]string, 4)
    msgpack.Unmarshal(buf, &newzy)
    fmt.Printf("V: %s\n", newzy)
}

If one was to comment the invalid IsNil call, you get the following error:

panic: interface conversion: interface is [4]string, not []string

goroutine 1 [running]:
github.com/vmihailenco/msgpack.(*Encoder).encodeSlice(0xf84002f220, 0x46ca98, 0xf840000e18, 0x112, 0x1, ...)

Has there been any thought about bringing this library up to date with the latest MSGPACK spec?

I'm happy to help work towards that goal; also keen to synchronise with others' efforts on this.

I have found this errors

reproduce step:
Set an int64 or uint64 pointer in structure, and not set value for it.

Example code:

type f9 struct {
    A *int64
    B string
}
func test() {
    ff, err := msgpack.Marshal(&f9{B: "123"})
    if err != nil {
        fmt.Println("marshal error: %v", err)
    }
    fmt.Println("byte", string(bs))
    g := &f9{}
    err = msgpack.Unmarshal(ff, g)
    if err != nil {
        fmt.Println("unmarshal error: %v", err) //msgpack: invalid code c0 decoding int64
    }
    fmt.Println("unmarshal: ", g.B)
}

do you have any idea?

In my opinion it would be nicer, if an error was returned.

Instruction: https://github.com/msgpack/website/blob/master/README.md#how-to-list-up-your-project-on-msgpackorg

Hi I am unable to install msgpack due to the following report:

$ go get github.com/vmihailenco/msgpack

github.com/vmihailenco/msgpack

../../vmihailenco/msgpack/decode.go:23: undefined: bufio.NewBuffer

go version go1.2.1 darwin/amd64

Any solutions?

Is it possible to add marshall/unmarshall tags as with bson/json packages? I'm using this to send data to/from a websocket, and I had to shorten the names of the properties of my structs, but it would be preferable if I could use things like msgpack:"name,omitempty" to shorten the message size over the wire.

Example code:

package main

import (
        "errors"
        "fmt"

        "gopkg.in/vmihailenco/msgpack.v2"
)

type APIRes struct {
        Code    int
        Message string
        Error   error
}

func main() {
        a := &APIRes{Error: errors.New("hi")}
        b := &APIRes{}
        buf, _ := msgpack.Marshal(a)
        msgpack.Unmarshal(buf, b)
        fmt.Print(b)
}

error:

panic: reflect.Set: value of type map[interface {}]interface {} is not assignable to type error

goroutine 1 [running]:
panic(0xe23c0, 0xc820076340)
        /usr/local/Cellar/go/1.6.3/libexec/src/runtime/panic.go:481 +0x3e6
reflect.Value.assignTo(0xe1a40, 0xc82007a3f0, 0x15, 0x13a790, 0xb, 0x102d20, 0xc82007a258, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.6.3/libexec/src/reflect/value.go:2164 +0x3be
reflect.Value.Set(0x102d20, 0xc82007a258, 0x194, 0xe1a40, 0xc82007a3f0, 0x15)
        /usr/local/Cellar/go/1.6.3/libexec/src/reflect/value.go:1334 +0x95
gopkg.in/vmihailenco/msgpack.v2.(*Decoder).interfaceValue(0xc82007a3c0, 0x102d20, 0xc82007a258, 0x194, 0x0, 0x0)
        $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:226 +0xe3
gopkg.in/vmihailenco/msgpack.v2.decodeInterfaceValue(0xc82007a3c0, 0x102d20, 0xc82007a258, 0x194, 0x0, 0x0)
        $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/decode_value.go:157 +0x7b
gopkg.in/vmihailenco/msgpack.v2.(*field).DecodeValue(0xc8200721c0, 0xc82007a3c0, 0x112b80, 0xc82007a240, 0x199, 0x0, 0x0)
        $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/types.go:84 +0x99
gopkg.in/vmihailenco/msgpack.v2.decodeStructValue(0xc82007a3c0, 0x112b80, 0xc82007a240, 0x199, 0x0, 0x0)
        $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/decode_map.go:215 +0x2a1
gopkg.in/vmihailenco/msgpack.v2.(*Decoder).DecodeValue(0xc82007a3c0, 0x112b80, 0xc82007a240, 0x199, 0x0, 0x0)
        $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:188 +0x98
gopkg.in/vmihailenco/msgpack.v2.(*Decoder).decode(0xc82007a3c0, 0xd5560, 0xc82007a240, 0x0, 0x0)
        $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:183 +0x4f1
gopkg.in/vmihailenco/msgpack.v2.(*Decoder).Decode(0xc82007a3c0, 0xc820041ee8, 0x1, 0x1, 0x0, 0x0)
        $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:65 +0xbd
gopkg.in/vmihailenco/msgpack.v2.Unmarshal(0xc82006e094, 0x17, 0x40, 0xc820041ee8, 0x1, 0x1, 0x0, 0x0)
        $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/decode.go:39 +0x37b
main.main()
        $GOPATH/src/test/t.go:20 +0x2ac
exit status 2

We use this package for connector to in-memory database tarantool .
Tarantool uses msgpack arrays for its row types.
Pack/unpack from a struct to msgpack array currently ought to go through CustomEncoder/CustomDecoder.

Proposal is to add ability to point that struct should be packed/unpacked as an array without need for CustomEncoder/Decoder.

ugorji/codec provides this ability through tags on specific field: https://godoc.org/github.com/ugorji/go/codec#Encoder.Encode

However, struct values may encode as arrays. This happens when:

• StructToArray Encode option is set, OR
• the tag on the _struct field sets the "toarray" option

I have error when using custom type on the basis map[string]interface{}
How to use?

Error:

panic: interface conversion: interface {} is store.S, not map[string]interface {}

Ex. code:

package store

import (
    "testing"
)

type S map[string]interface{}

func (s S) Set(key string, v interface{}) {
    s[key] = v
}

func (s S) Get(key string) interface{} {
    return s[key]
}

func TestStore(t *testing.T) {
    msgpack.RegisterExt(1, S{})

    v := S{}

    v.Set("a", "a")
    assert.Equal(t, v.Get("a"), "a")

    network, err := msgpack.Marshal(v)
    assert.Equal(t, err, nil, "encode")

    // 

    v = S{}
    err = msgpack.Unmarshal(network, &v)
    assert.Equal(t, err, nil, "decode")

    assert.Equal(t, v.Get("a"), "a", "custom type")
}

Check full details on Airbrake: http://getexceptional.airbrake.io/projects/1055/groups/46/notices/912842046857805825

While trying to test your fix for #127, I'm getting byte sequences that are slightly off.
The following patch, when run, results in a panic.
Upon closer inspection, this seems to be happening:

MarshalMsgpack produces byte sequence []byte{0x7, 0x5b, 0xcd, 0x15, 0x0, 0x0, 0x0, 0x7b}, but the value that we get via Decode is []byte{0xd7, 0x0, 0x7, 0x5b, 0xcd, 0x15, 0x0, 0x0, 0x0}. The msgpack type byte and the null padding seems to be included in the byte sequence passed to UnmarshalMsgpack

diff --git a/example_registerExt_test.go b/example_registerExt_test.go
index 9a2681d..a0b26cd 100644
--- a/example_registerExt_test.go
+++ b/example_registerExt_test.go
@@ -1,6 +1,7 @@
 package msgpack_test
 
 import (
+       "bytes"
        "encoding/binary"
        "fmt"
        "time"
@@ -16,6 +17,10 @@ type EventTime struct {
 var _ msgpack.Marshaler = (*EventTime)(nil)
 var _ msgpack.Unmarshaler = (*EventTime)(nil)
 
+func init() {
+       msgpack.RegisterExt(0, &EventTime{})
+}
+
 func (tm *EventTime) MarshalMsgpack() ([]byte, error) {
        b := make([]byte, 8)
        binary.BigEndian.PutUint32(b, uint32(tm.Unix()))
@@ -33,9 +38,7 @@ func (tm *EventTime) UnmarshalMsgpack(b []byte) error {
        return nil
 }
 
-func ExampleRegisterExt() {
-       msgpack.RegisterExt(0, &EventTime{})
-
+func ExampleRegisterExtMarshalUnmarshal() {
        b, err := msgpack.Marshal(&EventTime{time.Unix(123456789, 123)})
        if err != nil {
                panic(err)
@@ -49,3 +52,19 @@ func ExampleRegisterExt() {
        fmt.Println(v.(EventTime).UTC())
        // Output: 1973-11-29 21:33:09.000000123 +0000 UTC
 }
+
+func ExampleRegisterExtEncodeDecode() {
+       var buf bytes.Buffer
+       var t1 = EventTime{time.Unix(123456789, 123)}
+       if err := msgpack.NewEncoder(&buf).Encode(&t1); err != nil {
+               panic(err)
+       }
+
+       var t2 EventTime
+       if err := msgpack.NewDecoder(&buf).Decode(&t2); err != nil {
+               panic(err)
+       }
+
+       fmt.Println(t2.UTC())
+       // Output: 1973-11-29 21:33:09.000000123 +0000 UTC
+}

$ go test -run=Register -v
=== RUN   TestRegisterExtPanic
--- PASS: TestRegisterExtPanic (0.00s)
=== RUN   ExampleRegisterExtMarshalUnmarshal
--- PASS: ExampleRegisterExtMarshalUnmarshal (0.00s)
=== RUN   ExampleRegisterExtEncodeDecode
--- FAIL: ExampleRegisterExtEncodeDecode (0.00s)
panic: invalid data length: got 9, wanted 8 [recovered]
	panic: invalid data length: got 9, wanted 8

goroutine 1 [running]:
testing.runExample.func2(0xed099305b, 0xc436c5e8ca, 0x13941e0, 0xc42000c1e8, 0xc42000c018, 0xc42006ab40, 0x1274f1c, 0x1e, 0x127bad0, 0x1277d79, ...)
	/Users/daisuke/.gvm/gos/go1.8/src/testing/example.go:117 +0x359
panic(0x122ace0, 0xc420017500)
	/Users/daisuke/.gvm/gos/go1.8/src/runtime/panic.go:489 +0x2cf
gopkg.in/vmihailenco/msgpack%2ev2_test.ExampleRegisterExtEncodeDecode()
	/Users/daisuke/dev/src/gopkg.in/vmihailenco/msgpack.v2/example_registerExt_test.go:65 +0x4c2
testing.runExample(0x1274f1c, 0x1e, 0x127bad0, 0x1277d79, 0x28, 0x0, 0x0)
	/Users/daisuke/.gvm/gos/go1.8/src/testing/example.go:122 +0x204
testing.runExamples(0xc42003bed8, 0x138f5a0, 0xb, 0xb, 0x1230101)
	/Users/daisuke/.gvm/gos/go1.8/src/testing/example.go:46 +0x167
testing.(*M).Run(0xc42003bf20, 0xc42003bf20)
	/Users/daisuke/.gvm/gos/go1.8/src/testing/testing.go:823 +0x182
main.main()
	gopkg.in/vmihailenco/msgpack.v2/_test/_testmain.go:138 +0xf7
exit status 2
FAIL	gopkg.in/vmihailenco/msgpack.v2	0.011s

I a using msgpack to encode a map ie:
newmap["rate"] = 100
marshalledMap, _ := msgpack.Marshal(newmap)
var valOut map[string]interface{}
_ = msgpack.Unmarshal(marshalledMap, &valOut)
reflect.TypeOf(valOut["rate"])

uint64

it should be packed as int or int64 and back it int or int64, not uint64.

Example where it persists with json encoding end decoding but not with msgpack.
https://play.golang.org/p/VaeYnkLPS3

Can appengine.go be moved to a subpackage (aka github.com/vmihailenco/msgpack/appengine) so that tools like govend can prune the fairly heavyweight google.golang.org/appengine/datastore dependency?

It would require users to _ (underscore) import it instead of just relying on build tags, but since most users won't need it, it'll cut down binary size and number of vendored packages.

I can send in a simple PR, but I haven't used appengine before, so I won't be able to test out that part right away.

Edit: Of course, this would be a breaking change for appengine users, so this wouldn't be easy to implement right away. Maybe I'll focus on a PR for govend that allows specific build tags to be ignored.

f.PkgPath != "" && !f.Anonymous

http://tip.golang.org/doc/go1.6#reflect

I use msgpack serialization for storing about 10 billion records. Each record has around 15 fields in a go struct. The most common use case is that of reading from this set and examining some fields to do processing. In the whole process, I realized that most of the time is being spent in msgpack deserialization to map[string]interface{}.

I noticed that I usually only read a small subset of the fields. Is there a way to improve deserialization performance by only reading those fields I need? (I tried defining a struct with only those fields but that made performance worse than using a map[string]interface{}).

Something along the lines of the following might be useful.

type SubData struct {
    Y string
}

type Record struct {
    A int
    B SubData
    C int
    D string
    E []int
    F bool
}

A := msgpack.DeserializeInt(record_bytes, "A")
B := msgpack.DeserializeString(record_bytes, "B.Y")

or

type DeserializedSubData struct {
    Y string
}

type DeserializedRecord struct {
    A int
    B DeserializedSubData
}

var d DeserializedRecord
msgpack.Deserialize(record_bytes, &d)

Please note that the second case works currently but performance isn't any better for reading only some fields.

Could you please update import path to use fully qualified import as it described here:
https://github.com/golang/appengine#2-update-import-paths

Following line breaks usage of godeps because of relative import:
https://github.com/vmihailenco/msgpack/blob/master/appengine.go#L8

To demonstrate the issue, I have updated the test from #30 in https://github.com/dim/msgpack/blob/bug/marshal-interfaces/custom_test.go, by rewriting the msgpack.Register callbacks as MarshalMsgpack/UnmarshalMsgpack, implementing both, the Marshaler and the Unmarshaler interface.

The test is now failing - it looks like nested structs are not recognised correctly as Unmarshaler interfaces.

https://github.com/dim/msgpack/tree/bug/custom-coders

--- FAIL: TestBin (0.00s)
panic: reflect: call of reflect.Value.IsNil on struct Value [recovered]
  panic: reflect: call of reflect.Value.IsNil on struct Value

goroutine 62 [running]:
testing.func·006()
  $GOROOT/src/testing/testing.go:441 +0x181
reflect.Value.IsNil(0xa0fea0, 0xc20802b070, 0xd9, 0x8abd00)
  $GOROOT/src/reflect/value.go:942 +0x100
gopkg.in/vmihailenco/msgpack%2ev2.encodeCustomValue(0xc208087e90, 0xa0fea0, 0xc20802b070, 0xd9, 0x0, 0x0)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/typeinfo.go:240 +0x4e
gopkg.in/vmihailenco/msgpack%2ev2.(*Encoder).EncodeValue(0xc208087e90, 0xa0fea0, 0xc20802b070, 0xd9, 0x0, 0x0)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/encode.go:119 +0xa1
gopkg.in/vmihailenco/msgpack%2ev2.(*Encoder).encodeArray(0xc208087e90, 0x890f40, 0xc20800b360, 0x57, 0x0, 0x0)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/slice.go:114 +0x135
gopkg.in/vmihailenco/msgpack%2ev2.(*Encoder).encodeSlice(0xc208087e90, 0x890f40, 0xc20800b360, 0x57, 0x0, 0x0)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/slice.go:105 +0xa6
gopkg.in/vmihailenco/msgpack%2ev2.encodeSliceValue(0xc208087e90, 0x890f40, 0xc20800b360, 0x57, 0x0, 0x0)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/typeinfo.go:164 +0x58
gopkg.in/vmihailenco/msgpack%2ev2.(*Encoder).EncodeValue(0xc208087e90, 0x890f40, 0xc20800b360, 0x57, 0x0, 0x0)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/encode.go:119 +0xa1
gopkg.in/vmihailenco/msgpack%2ev2.(*Encoder).encode(0xc208087e90, 0x890f40, 0xc20800b360, 0x0, 0x0)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/encode.go:114 +0x3d9
gopkg.in/vmihailenco/msgpack%2ev2.(*Encoder).Encode(0xc208087e90, 0xc208143e78, 0x1, 0x1, 0x0, 0x0)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/encode.go:67 +0xc6
gopkg.in/vmihailenco/msgpack%2ev2.Marshal(0xc208143e78, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/encode.go:45 +0x320
gopkg.in/vmihailenco/msgpack%2ev2_test.TestBin(0xc208068240)
  $GOPATH/src/gopkg.in/vmihailenco/msgpack.v2/types_test.go:132 +0x140
testing.tRunner(0xc208068240, 0xdf1fa8)
  $GOROOT/src/testing/testing.go:447 +0xbf
created by testing.RunTests
  $GOROOT/src/testing/testing.go:555 +0xa8b

I ran the following example with go test -run ExampleCustomEncoder -v and confirmed the example test passes.

package main

import (
    "fmt"

    "gopkg.in/vmihailenco/msgpack.v2"
)

type customStruct struct {
    S   string
    N   int
}

func (s *customStruct) EncodeMsgpack(enc *msgpack.Encoder) error {
    return enc.Encode(s.S, s.N)
}

func (s *customStruct) DecodeMsgpack(dec *msgpack.Decoder) error {
    return dec.Decode(&s.S, &s.N)
}

func main() {
    b, err := msgpack.Marshal(&customStruct{S: "hello", N: 42})
    if err != nil {
        panic(err)
    }

    var v customStruct
    err = msgpack.Unmarshal(b, &v)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%#v", v)
}

Are the following variables declaration in https://godoc.org/gopkg.in/vmihailenco/msgpack.v2#example-CustomEncoder really needed?

var (
    _   msgpack.CustomEncoder = &customStruct{}
    _   msgpack.CustomDecoder = &customStruct{}
)

Check full details on Airbrake: http://getexceptional.airbrake.io/projects/1055/groups/46/notices/912842046857805825

(I'm sure there's a more general problem than just net/url in here, but I haven't had time to dig down to the root cause, it's just the case that broke in our test suite.)

Consider the sample code at https://gist.github.com/nicholasknight/62fda58c3dc3c27c60209a22a88b44c1

In commit 775d4b0, that code printed:

Original struct: {Scheme:https Opaque: User:<nil> Host:github.com Path:/vmihailenco/msgpack RawPath: RawQuery: Fragment:}
msgpack bytes: ??Scheme?https?Opaque??User??Host?github.com?Path?/vmihailenco/msgpack?RawPath??RawQuery??Fragment?
Deserialized struct: {Scheme:https Opaque: User: Host:github.com Path:/vmihailenco/msgpack RawPath: RawQuery: Fragment:}

But as of commit 2f94887 (pull #72), it prints:

Original struct: {Scheme:https Opaque: User:<nil> Host:github.com Path:/vmihailenco/msgpack RawPath: RawQuery: Fragment:}
msgpack bytes: ??Scheme?https?Opaque??User??Host?github.com?Path?/vmihailenco/msgpack?RawPath??RawQuery??Fragment?
Deserialized struct: {Scheme:https Opaque: User:<nil> Host: Path: RawPath: RawQuery: Fragment:}

Note how only Scheme is filled in, not Host or Path.

I'm assuming it stopped parsing on a nil value but I haven't hunted down the exact cause. If I have time I will, but it might be a couple weeks.

(Edited to make things clearer. I should not file bugs after being up this long.)

Hello!

The test fail on armhf:

go test -v gopkg.in/vmihailenco/msgpack.v2 gopkg.in/vmihailenco/msgpack.v2/codes
# gopkg.in/vmihailenco/msgpack.v2_test
src/gopkg.in/vmihailenco/msgpack.v2/msgpack_test.go:66: constant 4294967296 overflows uint
src/gopkg.in/vmihailenco/msgpack.v2/msgpack_test.go:67: constant 18446744073709551613 overflows uint
src/gopkg.in/vmihailenco/msgpack.v2/msgpack_test.go:68: constant 18446744073709551614 overflows uint
src/gopkg.in/vmihailenco/msgpack.v2/msgpack_test.go:69: constant 18446744073709551615 overflows uint
src/gopkg.in/vmihailenco/msgpack.v2/msgpack_test.go:103: constant -9223372036854775808 overflows int

This was found by the Debian CI

Hi,
Sorry to bother you on this repository.
I'm trying to compile https://github.com/shirou/mqforward which finally depends on the non-existing github.com/vmihailenco/bufio repository. I assume, that your code made it into the official go-language repository.

go get fails with:

 go get
# cd .; git clone https://github.com/vmihailenco/bufio /home/xy/opt/go/src/github.com/vmihailenco/bufio
Klone nach '/home/xy/opt/go/src/github.com/vmihailenco/bufio' ...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
package github.com/vmihailenco/bufio: exit status 128

I 'm new to go and can't really figure out which package refers to the non-existing repository. Do you have a recommendations on how I could break up this dependency?

Type monitor.Telemetry is defined as

type TelemetryData interface{}
type Telemetry map[string]TelemetryData

panic: reflect.Value.Convert: value of type monitor.Telemetry cannot be converted to type map[string]interface {}

goroutine 1 [running]:
reflect.Value.Convert(0x4196960, 0xc4200fa1b0, 0x15, 0x428f320, 0x41939e0, 0x8, 0xc420074750, 0x42a05a0)
        /usr/local/Cellar/go/1.8/libexec/src/reflect/value.go:2187 +0x202
gopkg.in/vmihailenco/msgpack%2ev2.encodeMapStringInterfaceValue(0xc4200fa570, 0x4196960, 0xc4200fa1b0, 0x15, 0x4196960, 0xc4200fa1b0)
        /Users/vadzim/Documents/gopath/src/gopkg.in/vmihailenco/msgpack.v2/encode_map.go:66 +0x103
gopkg.in/vmihailenco/msgpack%2ev2.(*Encoder).EncodeValue(0xc4200fa570, 0x4196960, 0xc4200fa1b0, 0x15, 0x15, 0x0)
        /Users/vadzim/Documents/gopath/src/gopkg.in/vmihailenco/msgpack.v2/encode.go:124 +0x8c
gopkg.in/vmihailenco/msgpack%2ev2.(*Encoder).encode(0xc4200fa570, 0x4196960, 0xc4200fa1b0, 0x30, 0x41a9fa0)
        /Users/vadzim/Documents/gopath/src/gopkg.in/vmihailenco/msgpack.v2/encode.go:119 +0x21c
gopkg.in/vmihailenco/msgpack%2ev2.(*Encoder).Encode(0xc4200fa570, 0xc420051c30, 0x1, 0x1, 0x9, 0x9)
        /Users/vadzim/Documents/gopath/src/gopkg.in/vmihailenco/msgpack.v2/encode.go:74 +0x5b
gopkg.in/vmihailenco/msgpack%2ev2.Marshal(0xc420051c30, 0x1, 0x1, 0xc4200f88a0, 0x40, 0x0, 0x0, 0x0)
        /Users/vadzim/Documents/gopath/src/gopkg.in/vmihailenco/msgpack.v2/encode.go:34 +0x15a

Briefly, this implementation of msgpack encodes signed ints always as negatives:
msgpack.Marshal(int(128)) => "d1 00 80"
Correct result is "cc 80"

I tested it with C, Rust msgpack implementations and they return "cc 80" for signed int 128.

Specification does not distinguish signed and unsigned int.
What they refer as uint8,uint16,uint32,uint64 is actualy "positive int value" with specified width
What they refer as int8,int16,int32,int64 is actualy "negative int value" with specified width

Original issue: tarantool/go-tarantool#9

Hello !

We've been using your library for internal development and we're very happy about it.

The other day, one of the guy updated his server to 2.1.3 and it started to break many services that were interacting with it as they were trying to decode some MsgPack 2.1.2 and were getting some 2.1.3 encoded.

As it's Embedded structure, we thought it might be due to one of your last commits and reverted back to 2.1.2 all the servers.

Was it intentional or did you not know there was compatibility issues between both ?

Thanks.

Check full details on Airbrake: http://getexceptional.airbrake.io/projects/1055/groups/44/notices/912836946550587395

Your package depends on github.com/ugorji/go-msgpack which is deprecated and in turn depends on a package launchpad.net/mgo/bson which has moved to https://gopkg.in/mgo.v2, so the dependency chain is now broken. I've created issue https://github.com/ugorji/go-msgpack/issues/16 but maybe this package should considering moving to ugorji/go-msgpack's successor instead?

Is it possible to encode/decode map[string]interface{}? I know you can technically do it, but it comes back as a map[interface{}]interface{}, and was wondering if the spec allowed it to be map[string]interface{}.

The following code causes an io.EOF error if the amount of Objects is larger than 1.

package main

import "gopkg.in/vmihailenco/msgpack.v2"

type innerObj struct {
	Objects []*obj
}

type obj struct {
	inner *innerObj
}

func main() {
	o1 := obj{
		// works:
		// inner: &innerObj{[]*obj{&obj{}}},

		// doesn't work:
		inner: &innerObj{[]*obj{&obj{}, &obj{}}},
	}

	bytes, err := msgpack.Marshal(&o1)
	if err != nil {
		panic(err)
	}

	o2 := obj{}
	if err = msgpack.Unmarshal(bytes, &o2); err != nil {
		panic(err)
	}
}

func (o *obj) MarshalMsgpack() ([]byte, error) {
	return msgpack.Marshal(o.inner)
}

func (o *obj) UnmarshalMsgpack(data []byte) error {
	inner := new(innerObj)
	if err := msgpack.Unmarshal(data, inner); err != nil {
		return err
	}

	o.inner = inner
	return nil
}

For the record, a similar construction does work with Go's xml package.

Check full details on Airbrake: http://getexceptional.airbrake.io/projects/1055/groups/45/notices/912837904781279233

I was trying to embed one struct into another and the marshall/unmarshall. But there seems to be something wrong. The variable name of the embedded struct is added and everything in it is put into a object.

type A struct {
        ValOne uint8 `msgpack:"valone"`
        ValTwo string `msgpack:"valtwo"`
}

type B struct {
        A *A
        ValThree string `msgpack:"valthree"`
}

got

{
  "A": {
    "valone": 1,
    "valtwo": "two!"
  },
  "valthree": "three!"
}

expected

{
  "valone": 1,
  "valtwo": "two!",
  "valthree": "three!"
}

This is different behaviour from the "encoding/json" package or I'm using it wrong.

What do you think?

Hello,

Debian is getting failing tests on i386.

src/gopkg.in/vmihailenco/msgpack.v2/bench_test.go:81: constant 2147483648 overflows in

Full log: https://tests.reproducible-builds.org/debian/rb-pkg/testing/i386/golang-gopkg-vmihailenco-msgpack.v2.html

Hi,
This project seems to be missing a LICENSE file,
please include a LICENSE file or something to that effect.

Best regards,

We have embedded times in our code, like this:

type CustomTime struct {
    time.Time
}

These have encoded ok in the past, but when I updated msgpack they will always encode to nil.
See this example code: https://gist.github.com/fredr/be78ebe33b894ef4c833

Run it on commit 42dfffd and it will work fine, output:

i: main.Thing{T1:time.Time{sec:63557272447, nsec:0x207cedd1, loc:(*time.Location)(0x1a2940)}, T2:main.EmbTime{Time:time.Time{sec:63557272447, nsec:0x207cedd1, loc:(*time.Location)(0x1a2940)}}}

[130 162 84 49 146 210 84 189 12 127 210 32 124 237 209 162 84 50 129 164 84 105 109 101 146 210 84 189 12 127 210 32 124 237 209]

out: map[interface {}]interface {}{"T1":[]interface {}{1421675647, 545058257}, "T2":map[interface {}]interface {}{"Time":[]interface {}{1421675647, 545058257}}}

Run it on commit ab83284 and T2 will be empty after encode/decode:

i: main.Thing{T1:time.Time{sec:63557272432, nsec:0xf7b18ad, loc:(*time.Location)(0x19d920)}, T2:main.EmbTime{Time:time.Time{sec:63557272432, nsec:0xf7b18ad, loc:(*time.Location)(0x19d920)}}}

[130 162 84 49 146 210 84 189 12 112 210 15 123 24 173 162 84 50 128]

out: map[interface {}]interface {}{"T1":[]interface {}{1421675632, 259725485}, "T2":map[interface {}]interface {}{}}

This is my Struct:

package articles

import "time"

type ArticleObj struct {
	ID         int64
	Title      string
	Excerpt    string
	Html       string
	Content    []ContentObj
	Url        string
	Images     []*ImageObj
	PubishTime time.Time
	Categories []int
	SourceID   int
}

type ContentObj interface {
	Type() string
}
type ImageObj struct {
	URL    string
	Mime   string
	Width  int
	Height int
}
type ImageContentObj struct {
	ImageObj
	TypeContent string
}

func (i ImageContentObj) Type() string {
	return i.TypeContent
}

type ParagraphObj struct {
	Text        string
	TypeContent string
}

func (p ParagraphObj) Type() string {
	return p.TypeContent
}

This is example of Article's object:

var (
	contentObjs = []articles.ContentObj{
		&articles.ParagraphObj{"I will stand over the world", "text"},
		&articles.ImageContentObj{
			articles.ImageObj{
				URL:    "http://bxcode.com/images/gambar.png",
				Mime:   "png",
				Width:  200,
				Height: 200,
			},
			"image",
		},
	}
	images = []*articles.ImageObj{
		&articles.ImageObj{
			URL:    "http://bxcode.com/images/winner.png",
			Mime:   "png",
			Width:  200,
			Height: 200,
		},
	}
	now         = time.Now()
	mockArticle = &articles.ArticleObj{
		ID:         int64(63),
		Title:      "I'm On Fire",
		Excerpt:    "We will won, whatever happen for us",
		Html:       "<h3> I'm On Fire </h3> <br> <p> We will Won, Whatever Happer for us </p>",
		Content:    contentObjs,
		Url:        "http://bxcode.com/article/23",
		Images:     images,
		PubishTime: now,
		Categories: []int{2, 3, 4},
		SourceID:   23,
	}
)

If this object serialized to JSON , be like this :

{
	"ID": 63,
	"Title": "I'm On Fire",
	"Excerpt": "We will won, whatever happen for us",
	"Html": "<h3> I'm On Fire </h3> <br> <p> We will Won, Whatever Happer for us </p>",
	"Content": [
		{
			"Text": "I will stand over the world",
			"TypeContent": "text"
		},
		{
			"URL": "http://bxcode.com/images/gambar.png",
			"Mime": "png",
			"Width": 200,
			"Height": 200,
			"TypeContent": "image"
		}
	],
	"Url": "http://bxcode.com/article/23",
	"Images": [
		{
			"URL": "http://bxcode.com/images/winner.png",
			"Mime": "png",
			"Width": 200,
			"Height": 200
		}
	],
	"PubishTime": "2017-06-08T12:04:01.968832678+07:00",
	"Categories": [
		2,
		3,
		4
	],
	"SourceID": 23
}

When i trying to serialize it using your library
like this :

b, err := msgpack.Marshal(mockArticle)
	assert.NoError(t, err)

	var a articles.ArticleObj
	err = msgpack.Unmarshal(b, a)
	assert.NoError(t, err)
	assert.Equal(t, mockArticle.Categories, a.Categories)

I got the error response :

Error: Received unexpected error msgpack: Decode(nonsettable articles.ArticleObj)

Any Idea about this, i couldn't find any documentations about this.

Can this msgpack support for nested struct in golang ?

I'm using msgpack format to store data in redis.

After switching to your (fast!) lib I have this weird issue: on my local machine one of many test are failing inexplicably. But the same test passes when I am running the same test suite in a docker environment. Switching back to the old msgpack lib the tests are fine in both environments.

Do you have any hints for me on where should I be looking to fix this?

I've previously been using github.com/ugorji/go/codec and recently moved over to yours.

A normal float32 of value 0 gets encoded into [202 0 0 0 0] in both yours and ugorjis
But when inside a struct

type TestStruct struct {
    MyFloat float32
}

When this gets encoded ( with MyFloat being 0), the value becomes [129 163 78 117 109 203 0 0 0 0 0 0 0 0]

The same on ugorji is [129 163 78 117 109 202 0 0 0 0]

There are two differences

1. extra 4 zeroes ( I'm not facing any issue with this but shouldn't it be 4 bytes instead of 8 ? )
2. The code 203 is being used instead of 202.

I'm having an issue with the second difference since decoding ugorji encodings with your decoder is causing it to panic

failed to decode ugorji encoding...msgpack: invalid code ca decoding float64
    panic: msgpack: invalid code ca decoding float64

Here is a detailed comparison to recreate the same.

Check full details on Airbrake: https://getexceptional.airbrake.io/projects/1055/groups/56308429/notices/901291446437417014

I'm trying to encode/decode an extension, namely https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1#eventtime-ext-format

If my understanding is correct, to implement this you need to the ability to write raw bytes to the underlying io.Writer instead of using EncodeInt8, EncodeInt32, etc. But currently the capability to do this is in Encoder.w, which is no exposed to end users.

Should this be exposed? or is there a better way to do this?

Is it possible to export EncodeBytesLen() in the encoder APIs? My use case is that I need to encode a large number of items, each of which contains an ID, which in turn contains a prefix, a body, and a suffix, all of which are byte slices, e.g.,

type ID struct {
  Prefix []byte
  Body []byte
  Suffix []byte
}

What I'd like to achieve is to encode the ID object as if it were a single []byte so that I can avoid unnecessary byte copies during decoding. With the existing API, I'd have to concatenate the prefix/body/suffix and create a new byte slice to use the EncodeBytes method. However if we export EncodeBytesLen(), I can call EncodeBytesLen() to encode the number of bytes first, then write the raw bytes.

If that sounds reasonable, I'd be happy to put up a PR.

Is there any reason why a uint32 value (when stored as interface{}) is encoded as uint64 although the msgpack spec supports uint32? For reference link to the spec.

To be more specific:

n := uint32(10)
var v interface{} = n

Now if v is Marshaled it will be get encoded as uint64