mitchellh / mapstructure Goto Github PK

Go library for decoding generic map values into native Go structures and vice versa.

Home Page: https://gist.github.com/mitchellh/90029601268e59a29e64e55bab1c5bdc

License: MIT License

Go 100.00%

mapstructure's Introduction

mapstructure

mapstructure is a Go library for decoding generic map values to structures and vice versa, while providing helpful error handling.

This library is most useful when decoding values from some data stream (JSON, Gob, etc.) where you don't quite know the structure of the underlying data until you read a part of it. You can therefore read a map[string]interface{} and use this library to decode it into the proper underlying native Go structure.

Installation

Standard go get:

$ go get github.com/mitchellh/mapstructure

Usage & Example

For usage and examples see the Godoc.

The Decode function has examples associated with it there.

But Why?!

Go offers fantastic standard libraries for decoding formats such as JSON. The standard method is to have a struct pre-created, and populate that struct from the bytes of the encoded format. This is great, but the problem is if you have configuration or an encoding that changes slightly depending on specific fields. For example, consider this JSON:

{
  "type": "person",
  "name": "Mitchell"
}

Perhaps we can't populate a specific structure without first reading the "type" field from the JSON. We could always do two passes over the decoding of the JSON (reading the "type" first, and the rest later). However, it is much simpler to just decode this into a map[string]interface{} structure, read the "type" key, then use something like this library to decode it into the proper structure.

mapstructure's People

Contributors

Stargazers

Watchers

Forkers

gyim vlebedev goinggo ryanuber sprucehealth nobu-k ottemo mtabini ronlobo ludei dancannon yyzi zihuxinyu godeep hashrabbit rhinoman gsh45 pib beihl shutej leowmjw flosch phinze andibrunner pombredanne liujianping dvln cbroglie benma hectorj mcos hydrologic apackeer h2object comdex varver jmptrader akutz jen20 iwanbk 17twenty sbward dmarkham arthurpro abdel-raouf henvic zj8487 camronlevanger pschlump segmentio easeway patricktoca cocap10 youngpm cloudxtreme mr8 junxie6 daria-lewandowska jefferai archsaber arhitiron jim3ma phenixsh cjellick linearregression dnephin mattrgreen countfenring nmiyake gernoteger richardknop is-kh skipor elliotmr stgerrard008 sitano calvinchengx seosgithub grubernaut ncodes narita-takeru amrfaissal qq7987590 lootek itachi123456789 gavbaa willhug aleksi hori-ryota hunterchen beornf zhaojkun iketutg yext pranj proximaio zenithar zyukin ef2k ericxiushu

mapstructure's Issues

support for embed anonymous struct

I found out that Decoder in this package cannot decode the input data into a embed structure field.

	type HeroName struct {
		Name string `json:"name"`
	}
	type Hero struct {
		HeroName
		Title string `json:"title"`
	}

	input := map[string]interface{}{
		"name":  "DK",
		"title": "Dead Knight",
	}

	var hero Hero

	config := &mapstructure.DecoderConfig{
		Metadata:         nil,
		Result:           &hero,
		TagName:          "json",
		WeaklyTypedInput: true,
		ZeroFields:       false,
	}

	decoder, _ := mapstructure.NewDecoder(config)
	decoder.Decode(input)
	fmt.Println(hero)

the output is:
{{} Dead Knight}
what I expected is:
{{DK} Dead Knight}

the decoder just ignore the embed field "Name"

Custom decoders?

The json package providers a Unmarshaler interface that allows types to define how they should be decoded:

e.g: https://github.com/xushiwei/mgo/blob/master/src/labix.org/v2/mgo/bson/bson.go#L232

I'm looking to do something similar with mapstructure, what would be the best approach?

Decode fails on key name with an underscore

See attached file for test case with the problem that I am seeing.
test.go.txt

Basically I want to convert part of a JSON string to a struct. In the example this is the 'key' section. To do the conversion I unmarshal the JSON to an unknown map, extract part of the map I am interested in, and then use Decode to convert that map to a struct.

The example has 2 (hopefully) identical functions processing slightly different JSON. In the first function a nested JSON key has an underscore in it, and in the second function the same nested key does not have an underscore in it.

The output of the first function shows that Decode loses the value of that key with the underscore. The output of the second function shows that Decode does not lose the value of that key. I have tried to create the example so that the only meaningful difference between these 2 functions is the underscore in the key name.

Apologies for not finding the actual bug (assuming there is one), hopefully you can easily find it and fix it.

Add support for field name remapping hook

Currently, if I'm not mistaken, the only way to reconcile input data field names and destination struct field names is to use annotations on destination struct's fields.

It will be convenient, if it was possible to have a hook function which will return possible alternative names for any given source/destination field name (either one appears to be enough). There exist many common cases for such functionality, examples being underscore delimited to camel case translation, unicode/punicode encoding handling, various escaped strings and so on.

Default value

Hi,

is there a way to provide default values for an field?

Sth. like

...
    Field string "mapstructure:"field" mapstructure_default:"A default value"`
...

Nil pointers

I have a struct that looks like this:

type Subtype struct{
  SomeField string
}

type Mytype struct{
  MyField *Subtype
}

If I try to decode code the following map, it produces a zero value struct:

testMap := map[string]interface{}{
  "MyField": nil,
}

// Creates
result :=Mytype{
  MyField: nil,
}

It would be nice to have pointer fields set to nil if a the input is a nil. This allows us to truly distinguish whether a field is nil (because there is no data) or not nil (has data). Like so:

result :=Mytype{
  MyField: &Subtype{
    SomeField: "",
  },
}

Squash embedded structs by default

To be consistent with "encoding/json", an embedded struct like

type Outer struct {
    Inner
}

should behave as though tagged with mapstructure:",squash", unless explicitly given a name like mapstructure:"Inner".

Example code demonstrating the inconsistency with "encoding/json":

package main

import (
    "encoding/json"
    "fmt"
    "github.com/mitchellh/mapstructure"
)

type Inner struct {
    Name string
}

type Outer struct {
    Inner // works if you add `mapstructure:",squash"`
}

func main() {
    input := Outer{Inner: Inner{Name: "mitchellh"}}
    fmt.Printf("input: %#v\n", input)

    bytes, err := json.Marshal(input)
    if err != nil { panic(err) }
    fmt.Printf("json: %v\n", string(bytes))

    var outputGood Outer
    err = json.Unmarshal(bytes, &outputGood)
    if err != nil { panic(err) }
    fmt.Printf("outputGood: %#v\n", outputGood)

    var tmp interface{}
    err = json.Unmarshal(bytes, &tmp)
    if err != nil { panic(err) }
    fmt.Printf("tmp: %#v\n", tmp)

    var outputBad Outer
    err = mapstructure.Decode(tmp, &outputBad)
    if err != nil { panic(err) }
    fmt.Printf("outputBad: %#v\n", outputBad)
}

The output is:

input: main.Outer{Inner:main.Inner{Name:"mitchellh"}}
json: {"Name":"mitchellh"}
outputGood: main.Outer{Inner:main.Inner{Name:"mitchellh"}}
tmp: map[string]interface {}{"Name":"mitchellh"}
outputBad: main.Outer{Inner:main.Inner{Name:""}}

The correct output would have:

outputBad: main.Outer{Inner:main.Inner{Name:"mitchellh"}}

Add config to error if field is NOT set

Provide more info to the hook funcs

When you want to convert from map[string]interface{} to map[string]interface{} you don't have a lot of type information to identify a sensible type conversion.

I want to serialize some maps as JSON to disk and read them back, preserving as much of the type info as possible. This works fine when the target is typed (I can create hooks to convert from/to), not so much for map[string]interface{}. I believe this is a common enough requirement.

I can provide a patch to support this if you look positive on this (I already started a patch), but I think we need to discuss this a little first.

As I see it, we're missing some context here:

The field name
Whether it is a key or a value

In my quick test, testing with a map with a *big.Rat, I logged these values:

key: name: Data[myRat] , t1:  string , t2: string value: myRat
value: name: Data[myRat] , t1:  string , t2: interface value: 1/300

I know that Data[myRat] is a *big.Rat when I serialize it, so I can create a type mapping that I can use when I read it back, If I had a way to identify it.

Panic on decode squashed in squashed struct

Run of such test panics.

func TestNextSquashMapstructure(t *testing.T) {
	data := &struct {
		Level1 struct {
			Level2 struct {
				Foo string
			} `mapstructure:",squash"`
		} `mapstructure:",squash"`
	}{}
	err := mapstructure.Decode(map[interface{}]interface{}{"foo": "baz"}, &data)
	require.NoError(t, err)
	assert.Equal(t, "baz", data.Level1.Level2.Foo)
}

panic: reflect: call of reflect.Value.Type on zero Value

panic(0x2f5d80, 0xc4202d4ae0)
	1.7.3/libexec/src/runtime/panic.go:500 +0x1a1
testing.tRunner.func1(0xc4200a0240)
	1.7.3/libexec/src/testing/testing.go:579 +0x25d
panic(0x2f5d80, 0xc4202d4ae0)
	1.7.3/libexec/src/runtime/panic.go:458 +0x243
reflect.Value.Type(0x0, 0x0, 0x0, 0x2c53b1, 0x6)
	1.7.3/libexec/src/reflect/value.go:1670 +0x224
mitchellh/mapstructure.(*Decoder).decodeStruct(0xc420084268, 0x0, 0x0, 0x2f9080, 0xc420247d10, 0x300d00, 0xc4202ad110, 0x199, 0x0, 0x0)
	mitchellh/mapstructure/mapstructure.go:672 +0x625
mitchellh/mapstructure.(*Decoder).decode(0xc420084268, 0x0, 0x0, 0x2f9080, 0xc420247d10, 0x300d00, 0xc4202ad110, 0x199, 0xc41fffaa0c, 0x0)
	mitchellh/mapstructure/mapstructure.go:225 +0x6e8
mitchellh/mapstructure.(*Decoder).decodePtr(0xc420084268, 0x0, 0x0, 0x2f9080, 0xc420247d10, 0x2d67c0, 0xc420084260, 0x196, 0xc42008cde0, 0x1122e)
	mitchellh/mapstructure/mapstructure.go:557 +0x15b
mitchellh/mapstructure.(*Decoder).decode(0xc420084268, 0x0, 0x0, 0x2f9080, 0xc420247d10, 0x2d67c0, 0xc420084260, 0x196, 0xc4202ad101, 0xc420284fc0)
	mitchellh/mapstructure/mapstructure.go:229 +0x617
mitchellh/mapstructure.(*Decoder).Decode(0xc420084268, 0x2f9080, 0xc420247d10, 0x0, 0xc4202ad130)
	mitchellh/mapstructure/mapstructure.go:180 +0xb3
mitchellh/mapstructure.Decode(0x2f9080, 0xc420247d10, 0x2cb000, 0xc420084260, 0xc4202ad130, 0x0)
	mitchellh/mapstructure/mapstructure.go:122 +0xb6
TestNextSquashMapstructure(0xc4200a0240)

int8 not surport?

XML and JSON tag support

It'd be nice if there were a way to tell the codec to use the stock JSON or XML tags for writing or reading it's data. We find a lot of times moving from JSON where the names are mapped, we have to duplicate the JSON tags to mapstructure tags to get the desired results. A nice configuration option would be to tell the coded the tag type to honor.

Handle time.Time?

I have a structure like this:

type TestEntry struct { Id stringjson:"id" mapstructure:"id" MemberId string Present bool Date time.Time Type string }

And a map[string]interface{} like this that I am trying to map to the structure:

map[string]interface {}{"Present":true, "Type":"Other", "id":"342358db-4e66-4e39-a452-75859097bc39", "Date":time.Time{sec:63566170074, nsec:619999885, loc:(*time.Location)(0xc208030840)}, "MemberId":"81b3d896-4314-4078-aa00-a27782bc00b4"}

When I try to decode, I get the following error:

'Date' expected a map, got 'struct'

Any suggestions?

Document mapstructure tag and how to use it.

Godoc hints at its existence, and I can find references to it in the source code, but gives no examples of how to use it.

Decoding url.Values

I've had a dig around the godocs, but can't see a clear way to decode url.Values (which is map[string][]string) into a struct with string fields (i.e. string[0] into string).

Is this currently possible with mapstructure? If not, is it possible to look at doing so and setting a default case (i.e. use the first slice value, or the last, or allow configuration of this via NewDecoder). gorilla/schema uses the first slice value.

(I'm asking this as I like the 'configurability' of this package over schema, outside of this use-case!)

No encoding ?

mapstructure is a Go library for decoding generic map values to structures and vice versa

At first glance, there is no way to encode a struct to a map with this library ..?

Accept string to time.Time as weak input

It would be nice if the import can handle strings with a timestamp or a date time format as string.
Something like that: "2015-09-30T01:18:56.096224525+02:00"

Add to awesome-go list?

This looks like a useful high-quality Go package that serves a specific need.

It's not listed in the awesome-go list right now, perhaps you should add it there so more people can discover it?

Would you be open to support array types?

I have added in a case to handle array type because I need to deal with uuid.UUID type

	var err error
	dataKind := getKind(val)
	fmt.Println(name, val, dataKind)
	switch dataKind {
	case reflect.Bool:
		err = d.decodeBool(name, data, val)
	case reflect.Interface:
		err = d.decodeBasic(name, data, val)
	case reflect.String:
		err = d.decodeString(name, data, val)
	case reflect.Int:
		err = d.decodeInt(name, data, val)
	case reflect.Uint:
		err = d.decodeUint(name, data, val)
	case reflect.Float32:
		err = d.decodeFloat(name, data, val)
	case reflect.Struct:
		err = d.decodeStruct(name, data, val)
	case reflect.Map:
		err = d.decodeMap(name, data, val)
	case reflect.Ptr:
		err = d.decodePtr(name, data, val)
	case reflect.Slice:
		err = d.decodeSlice(name, data, val)
	case reflect.Array: // required to handle uuid.UUID type
		err = d.decodeArray(name, data, val)
	default:
		// If we reached this point then we weren't able to decode it
		return fmt.Errorf("%s: unsupported type: %s", name, dataKind)
	}

Would you accept this new feature?

Squash tags not taken into account when decoding from struct to map.

Squash tags not taken into account when decoding from struct to map. This means that decoding from a map to a struct and back to a map will result in different maps when the struct has squashed members.

Example:

package mapstructure

import (
	"fmt"
)

func ExampleDecode_reversibleSquash() {
	// Squashing multiple embedded structs is allowed using the squash tag.
	// This is demonstrated by creating a composite struct of multiple types
	// and decoding into it. In this case, a person can carry with it both
	// a Family and a Location, as well as their own FirstName.
	type Family struct {
		LastName string
	}
	type Location struct {
		City string
	}
	type Person struct {
		Family    `mapstructure:",squash"`
		Location  `mapstructure:",squash"`
		FirstName string
	}

	input := map[string]interface{}{
		"FirstName": "Mitchell",
		"LastName":  "Hashimoto",
		"City":      "San Francisco",
	}

	var result Person
	err := Decode(input, &result)
	if err != nil {
		panic(err)
	}

	// Squashing should also work when decoding from a struct back into a map
	// so that we can recover our input map from our previous result.
	output := make(map[string]interface{}, 3)
	err = Decode(result, &output)
	if err != nil {
		panic(err)
	}

	for key, value := range output {
		fmt.Println(key, value)
	}
	// Unordered output:
	// FirstName Mitchell
	// LastName Hashimoto
	// City San Francisco
}

Output:

=== RUN   ExampleDecode_reversibleSquash
--- FAIL: ExampleDecode_reversibleSquash (0.00s)
got:
Family map[LastName:Hashimoto]
Location map[City:San Francisco]
FirstName Mitchell

want (unordered):
FirstName Mitchell
LastName Hashimoto
City San Francisco

squashed embedded pointer panics

I was trying to wrap my head around the expected behavior of squashing by writing some new tests and ran across this. The simplest way to quickly demonstrate this is by changing the embedded type in the EmbeddedSquash type to a pointer. It causes a panic emanating from mapstructure.go:510.

type EmbeddedSquash struct {
    *Basic  `mapstructure:",squash"`
    Vunique string
}

--- FAIL: TestDecode_EmbeddedSquash (0.00 seconds)
panic: reflect: NumField of non-struct type [recovered]

json string of time.Time Decode() error: * 'Data.Created' expected a map, got 'string'

Below is my go code:

var js string = `{"Status":"OK", "Message":"", "Data":{"id":"e9b6ef297150c2fe2ddd73684b5042770eaebd5c466ce950de69e355ccb4ed5f","parent":"fdb0acf64df9338f5b8a42575a5c18b51c41bd07678ab892e65971fedda7e365","created":"2014-05-22T08:27:29.991940319Z","container":"ff8baa555f67024f54515c72e2676c83a82b415050f870cdd81bf94acd0783db","container_config":{"Hostname":"ff8baa555f67","Domainname":"","User":"","Memory":0,"MemorySwap":0,"CpuShares":0,"AttachStdin":false,"AttachStdout":true,"AttachStderr":true,"PortSpecs":null,"ExposedPorts":null,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":["yum","install","-y","xm-tengine"],"Image":"centos/nc","Volumes":null,"WorkingDir":"","Entrypoint":null,"NetworkDisabled":false,"OnBuild":null},"docker_version":"0.11.1","config":{"Hostname":"","Domainname":"","User":"","Memory":0,"MemorySwap":0,"CpuShares":0,"AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"PortSpecs":null,"ExposedPorts":null,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":["yum","install","-y","xm-tengine"],"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"NetworkDisabled":false,"OnBuild":null},"architecture":"amd64","os":"linux","Size":31407280}}`

    var ob interface{}
    err := json.Unmarshal([]byte(js), &ob)
    if err != nil {
        fmt.Println(err)
    }

    oc := struct {
        Status  string
        Message string
        Data    dockerlib.Container
    }{}

        oo, _ := ob.(map[string]interface{})
    err = mapstructure.Decode(oo, &oc)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%#v", ob)

But it can be json.Umarshal directly.
        od := struct {
        Status  string
        Message string
        Data    dockerlib.Container
    }{}
        err := json.Unmarshal([]byte(js), &od)
    if err != nil {
        fmt.Println(err)
    }
       fmt.Printf("%#v", od)
ouput:
   struct { Status string; Message string; Data docker.Container }{Status:"OK", Message:"", Data:docker.Container{ID:"e9b6ef297150c2fe2ddd73684b5042770eaebd5c466ce950de69e355ccb4ed5f", Created:time.Time{sec:63536344049, nsec:0x3b1fcedf, loc:(*time.Location)(0x4e6d40)}, Path:"", Args:[]string(nil), Config:(*docker.Config)(0xc2100516c0), State:docker.State{RWMutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, Running:false, Pid:0, ExitCode:0, StartedAt:time.Time{sec:0, nsec:0x0, loc:(*time.Location)(nil)}, FinishedAt:time.Time{sec:0, nsec:0x0, loc:(*time.Location)(nil)}, Ghost:false}, Image:"", NetworkSettings:(*docker.NetworkSettings)(nil), SysInitPath:"", ResolvConfPath:"", HostnamePath:"", HostsPath:"", Name:"", Driver:"", Volumes:map[string]string(nil), VolumesRW:map[string]bool(nil), HostConfig:(*docker.HostConfig)(nil)}}

Add a Tag for dep

dep is a dependency manager for go. It has been started with the intention of being merged into the official toolchain and while I cannot really guess whether that's achievable, it is the tool that I and I believe the go community as a whole likes most for open-source package management.

It likes to operate on Releases/git tags, locking them in as constraints. If at all possible, I would like for this library to create a tag at a sensible commit and label it something sensible (like 1.0.0 if such versioning makes sense) to prevent users of mapstructure to either have to absolutely lock in a commit hash (as such preventing one from getting possible security updates) or a branch (prone to API breakage).

Thank you all for your time.

Use something other then "mapstructure" in tags

I am using gorethink and in order for update/insert to work i have to use it like this:

CentralLocation bool gorethink:"central_location" - now this is working when doing insert/update
but when doing read it is not when i use your mapstructure.

However when i change
CentralLocation bool mapstructure:"central_location" read is working but again update/insert is not.

So is thre any way that i can get this working but with some other tag?

Zero pointer to struct fields, even if ZeroFields:false

Hello!
I have found unexpected behaviour of Decode to struct with pointer to struct fields.

import (
    "testing"

    "github.com/mitchellh/mapstructure"
    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"
)

type Level1 struct {
    Rewrited string
    Default  string
}

type Level2 struct {
    StructPtr  *Level1
    OtherValue string
}

type M map[string]interface{}

func TestNestedStructDecode(t *testing.T) {
    const iniVal = "initialValue"
    const newVal = "writedValue"
    result := Level2{
        StructPtr:  &Level1{iniVal, iniVal},
        OtherValue: iniVal,
    }

    decoder, err := mapstructure.NewDecoder(
        &mapstructure.DecoderConfig{
            ZeroFields: false,
            Result:     &result,
        })
    require.NoError(t, err)

    err = decoder.Decode(M{
        "StructPtr": M{"Rewrited": newVal},
    })
    require.NoError(t, err)
    assert.Equal(t, newVal, result.StructPtr.Rewrited) //OK
    assert.Equal(t, iniVal, result.OtherValue)         //OK
    assert.Equal(t, iniVal, result.StructPtr.Default)  //Failed
}

Last assert fails with error:

     Error:          Not equal: "initialValue" (expected)
                                != "" (actual)

However, if make StructPtr field have type Layer1, but *Layer1, test passed and all is working as expected.
As for me, result should not changed, after changing type to pointer and reverse.
Thanks!

Nested map[string]interface{} not decoded

The example below prints:

map[V1:v1 M2:map[V2:v2]]
V1
v1
M2
map[V2:v2]
[VS1_1 VS1_2]
VS1_1
VS1_2
[map[VMS:vms]]
map[VMS:vms]

The first map[string]interface{} is traversed, but the last (map[V2:v2]) seems to be just set as-is without any decoding of its elements. The same is the case for maps inside slices.

package main

import (
	"fmt"
	"log"

	"reflect"

	"github.com/mitchellh/mapstructure"
)

type T struct {
	M1 map[string]interface{}
	S1 []interface{}
	S2 []interface{}
}

func main() {
	dest := &T{}

	hook := func(t1 reflect.Type, t2 reflect.Type, v interface{}) (interface{}, error) {
		fmt.Printf("%v\n", v)
		return v, nil
	}

	input := map[string]interface{}{
		"M1": map[string]interface{}{
			"V1": "v1",
			"M2": map[string]interface{}{
				"V2": "v2",
			},
		},
		"S1": []interface{}{"VS1_1", "VS1_2"},
		"S2": []interface{}{map[string]interface{}{
			"VMS": "vms",
		}},
	}

	conf := &mapstructure.DecoderConfig{
		DecodeHook:       hook,
		ErrorUnused:      false,
		WeaklyTypedInput: false,
		Metadata:         nil,
		Result:           dest,
		ZeroFields:       true,
	}

	dec, err := mapstructure.NewDecoder(conf)
	if err != nil {
		log.Fatal(err)
	}
	err = dec.Decode(input)
	if err != nil {
		log.Fatal(err)
	}

}

Is performance a goal of this package or not?

Hi, a question about the objective of this package.

I understand it deals primarily with converting input of type map[string]interface{} (which you typically get as a result of unmarshalling JSON into interface{} when you don't know its concrete type in advance) to native Go structs.

However, beyond that, is performance a priority? Or is performance secondary to convenience of use (e.g., to make the user code simpler/shorter) or flexibility of input/output (e.g., mapstructure can unmarshal input of type map[interface{}]interface{})?

The reason I ask is because it doesn't seem very fast compared to a very dumb brute-force alternative implementation: to marshal the input as some format (JSON, gob, XML, etc.) and decode it into target Go struct. For example:

// DecodeViaJSON takes the map data and passes it through encoding/json to convert it into the
// given Go native structure pointed to by v. v must be a pointer to a struct.
func DecodeViaJSON(data interface{}, v interface{}) error {
    // Perform the task by simply marshalling the input into JSON, then unmarshalling
    // it into target native Go struct.
    b, err := json.Marshal(data)
    if err != nil {
        return err
    }
    return json.Unmarshal(b, v)
}

I've written a benchmark to compare it against mapstructure.Decode:

// Original benchmark that uses mapstructure.Decode.
func Benchmark_Decode(b *testing.B) {
    type Person struct {
        Name   string
        Age    int
        Emails []string
        Extra  map[string]string
    }

    input := map[string]interface{}{
        "name":   "Mitchell",
        "age":    91,
        "emails": []string{"one", "two", "three"},
        "extra": map[string]string{
            "twitter": "mitchellh",
        },
    }

    var result Person
    for i := 0; i < b.N; i++ {
        Decode(input, &result)
    }
}

// My benchmark that uses DecodeViaJSON with same input as above.
func Benchmark_DecodeViaJSON(b *testing.B) {
    type Person struct {
        Name   string
        Age    int
        Emails []string
        Extra  map[string]string
    }

    input := map[string]interface{}{
        "name":   "Mitchell",
        "age":    91,
        "emails": []string{"one", "two", "three"},
        "extra": map[string]string{
            "twitter": "mitchellh",
        },
    }

    var result Person
    for i := 0; i < b.N; i++ {
        DecodeViaJSON(input, &result)
    }
}

Here are the results:

Benchmark_Decode          100000         18507 ns/op        2773 B/op         73 allocs/op
Benchmark_DecodeViaJSON   100000         14292 ns/op        1466 B/op         43 allocs/op

Counterintuitively, the seemingly dumb approach of just marshalling input into JSON bytes and decoding it from those bytes is actually both 29.5% faster and causes less memory allocations. /cc @gbbr

Do you have thoughts on this? Is this finding expected or unexpected?

(I understand this DecodeViaJSON will not work on all inputs that mapstructure.Decode currently supports, but it will work for all basic use cases.)

Minor issue in docs/signature of Decode.

There is an issue in mapstructure.go#L108-L109. The documentation states:

val must be a pointer to a struct.

However, the signature has no parameter val:

func Decode(m interface{}, rawVal interface{}) error {

I am guessing val should be changed to rawVal in documentation, or rawVal should be changed to val in the func signature. I'm not sure which direction is better, but the discrepancy should be resolved.

Method to handle URL.Query()

It would be awesome if the library supported a straight forward mapping from URL.Query() from the standard library.

Query() returns a map like:
map[a:[aaaa] b:[bbbb] c:[cccc]]

And as expected I get type error: A' expected type 'string', got unconvertible type '[]string'

The struct looks like:

type Measure struct {
	A    string
	B    string
	C   string
}

*I've no idea why URL.Query returns a map with array elements inside tough. (well.. I know why but a GET is not likely to have duplicated params)

Map with a nil value panics

/cc @armon

Incorrect decoding (Silent data corruption)

Decoding large integer values causes silent changes to the data, leading to incorrect values.

package main

import (
	"encoding/json"
	"fmt"

	"github.com/mitchellh/mapstructure"
)

type example struct {
	Field int64 `json:"field"`
}

func main() {
	// Create a high original value and marshal it to bytes.
	original := &example{
		Field: 5577006791947779410,
	}
	buff, _ := json.Marshal(original)

	interim := make(map[string]interface{})
	json.Unmarshal(buff, &interim)

	target := &example{}
	mapstructure.Decode(interim, target)

	fmt.Printf("Original: %v\n", original.Field)
	fmt.Printf("Target: %v\n", target.Field)
	fmt.Printf("Variance: %v\n", original.Field-target.Field)
}

The output I get:

Original: 5577006791947779410
Target: 5577006791947779072
Variance: 338

Had been having random data variance issues for a while and finally pinned this down. It's caused one hell of a headache. @mitchellh - Am I doing something wrong here?

nil value not set

A nil value in the input should set a pointer in the destination struct to nil.

Example program:

package main

import (
	"fmt"
	mstruct "github.com/mitchellh/mapstructure"
)

type T struct {
	A *int
}

func main() {
	x := 5
	dest := &T{A: &x}

	valueMap := map[string]interface{}{
		"A": nil,
	}

	conf := &mstruct.DecoderConfig{
		ErrorUnused:      false,
		WeaklyTypedInput: false,
		Metadata:         nil,
		Result:           dest,
		ZeroFields:       true,
	}
	dec, err := mstruct.NewDecoder(conf)
	if err != nil {
		panic(err.Error())
	}
	err = dec.Decode(valueMap)
	if err != nil {
		panic(err.Error())
	}

	fmt.Printf("%#v\n", dest.A)
}

Prints: (*int)(0xc42000e2d8) not (*int)(nil). T.A is not modified.

No support for anonymous structs

Given a struct ("outer") that hosts an anonymous inner struct ("inner")...

type inner struct {
  I1 string
}

type outer struct {
  inner
  O1 int
}

I expected mapstructure.Decode to take the following map:

map[string]interface{}{
  "I1": "foo",
  "O1": 2
}

And populate outer.I1. I expected this mostly because I think of mapstructure.Decode and json.Unmarshal as analogous. json.Unmarshal does populate anonymous struct fields like this.

Was it intentional not to support anonymous structs?

why decodeString do'not suppport float64

Add support for decoding into slices defined by struct tags

I have a struct defined as follows:

type ConfigDefinition struct {
    Addresses []string    `mapstructure:"--addresses"`
}

var config ConfigDefinition

// Parse through CLI options using github.com/docopt/docopt-go
// The result is arguments will contain a map of strings (type: map[string]interface{})
arguments, _ := docopt.Parse(get_usage(), nil, true, "Test App", false)

// Now decode the arguments map into config
mapstructure.Decode(arguments, &config)
fmt.Printf("%#v\n", config)

What I'd expect:
config.Addresses would contain an array/slice of size 1 containing the string I specified via the CLI option --addresses

What I observe
config.Addresses contains an array of 14 items all with empty strings

why not support this interface?

func Decode(m interface{}, rawVal interface{}, tagName string) error {
	config := &DecoderConfig{
		Metadata: nil,
		Result:   rawVal,
                TagName: tagName,
	}

	decoder, err := NewDecoder(config)
	if err != nil {
		return err
	}

	return decoder.Decode(m)
}

Implement JSON like unmarshaling interface

Hey folks ;)

I think it would be nice if mapstructure would support an unmarshaling interface like for the json module, so custom marshalers can be defined on a per-type basis easily:

type MyType struct {
    number int
    numberTimesTwo int
}

// For example we call the interface function for mapstructure UnmarshalMap
func (t *MyType) UnmarshalMap(value interface{}) error {
    // Although not a real-world usecase, just multiply our read number by 2.
    intValue, ok := value.(int)

    if !ok {
        return errors.New("Can't convert value to int")
    }

    t.number = value
    t.numberTimesTwo = value * 2

    return nil
}

Specifically I tried to do custom marshaling inside viper, which internally uses mapstructure to unmarshal from its internal map-representation of config-key-values to my config-struct. But there are no possibilities there to "override" marshaling behaviour for my types and I have to write quite some boilerplate to circumvent that^^

Create a tag

Can a tag be created of this so that dependent packages can pull a specific version rather than having to use current sources? That would be much appreciated!

While this shows a 0.1, I made that version up. You can see in the ebuild it pulls the master.zip file rather than a tagged version.

https://github.com/Obsidian-StudiosInc/os-xtoo/blob/master/dev-go/mapstructure/mapstructure-0.1.ebuild

Decoding to pointers with nil value

Mapstructure is very useful and great library, thank you!

But many gophers use pointer type for know whether value is absent, or whether not.
The problem that mapstructure could not properly decode map values into struct field that is pointer.

Example:
We have target struct

type Result struct {
	T *time.Time
}

and have a decode hook, which try to parse string to *time.Time

func (f reflect.Type, t reflect.Type, data interface{}) (interface{}, error) {
	if t == reflect.TypeOf(&time.Time{}) && f.Kind() == reflect.String && data == "" {
		return nil, nil
	}
        
        ...

	return data, nil
}

After decoding
I would like to expect

type Result struct {
	T *time.Time // nil
}

But I got

type Result struct {
	T *time.Time // 0001-01-01 00:00:00 +0000 UTC
}

The reason is that decodePtr function creates zero value for valElemType

if realVal.IsNil() || d.config.ZeroFields {
	realVal = reflect.New(valElemType)
}

Question
Maybe we wont create zero value if data is nil and ZeroFields is false?
Or we can create specific tag for fields that could have nil value after decoding.

int64 support?

I just tried to convert a json file with unix timestamps, these are int64 in go. There seems to be no support for this data type, is this planned?

Decode fails for anonymous structures

I tried to decode from map to a struct which has anonymous struct inside. This works on simple json.Marshal -> json.Unmarshal but it fails in mapstructure.

package deserializer

import (
	"encoding/json"
	"testing"
	"github.com/mitchellh/mapstructure"
	"github.com/stretchr/testify/assert"
)

type A struct {
	Field1 int
}

type B struct {
	A
	Field2 int
}

func TestAnonymous(t *testing.T) {
	b := B{}
	b.Field1 = 1
	b.Field2 = 2

	outB1 := &B{}
	serialized, _ := json.Marshal(b)
	json.Unmarshal(serialized, outB1)

	assert.Equal(t, b.Field1, outB1.Field1)
	assert.Equal(t, b.Field2, outB1.Field2)

	rawMap := make(map[string]interface{})
	json.Unmarshal(serialized, &rawMap)

	outB2 := &B{}
	mapstructure.Decode(rawMap, outB2)

	assert.Equal(t, b.Field1, outB2.Field1)
	assert.Equal(t, b.Field2, outB2.Field2)
}

This line fails assert.Equal(t, b.Field1, outB2.Field1)

How to retrieve struct tag in DecodeHook?

I want to decode file path to file content, since I can't rely on type to decide if I should decode or not, it seems I should check struct tag instead.

But from the source code there doesn't seems to be a way to do that in DecodeHook, which only receives a field value?

mapstructure.go:667: undefined: reflect.ArrayOf

Hi,

I'm getting an issue when I use go get to install mapstructure

go get github.com/mitchellh/mapstructure

github.com/mitchellh/mapstructure

src/github.com/mitchellh/mapstructure/mapstructure.go:667: undefined: reflect.ArrayOf

Decode does not zero the destination variable before decoding the value into it.

I realized today that Decode does not zero the destination variable before decoding the value into it. Is this expected; or is this a bug?

Example:


	var rec r.ChangeResponse
	var updatedDec UserDecision

      	go func() {
		for cur.Next(&rec) {
			newValInterface := rec.NewValue
			fmt.Printf("\nNewvalue: %+v\n", newValInterface)
			fmt.Printf("\nupdatedDec: %+v\n", updatedDec) 
			mapstructure.Decode(newValInterface, &updatedDec) 
			fmt.Printf("\nDecision update: %+v\n", updatedDec)
			decUpdateCh <- updatedDec
		}
		log.Println("Exiting goroutine listening on decUpdate changes.")
	}()

In the above example updatedDec is not zeroed across iterations. updatedDec contains an array. If that array has more elements than the same array in the new value, then some of the earlier values remain in there after mapstrucure.Decode.

Panic on too-short slice decode

I don't know if this is "expected behavior" or not, but after #61 was merged, Decoding a slice of structs will panic if given a pointer to a non-nil slice which is not long enough to hold the input data.

func TestNestedStructInSlice(t *testing.T) {
	t.Parallel()

	input := []map[string]interface{}{
		map[string]interface{}{
			"vstring": "foo",
			"vint":    42,
			"vbool":   true,
		},
		map[string]interface{}{
			"vstring": "bar",
			"vint":    52,
			"vbool":   false,
		},
	}

	result := []Basic{}
	err := Decode(input, &result)
	if err != nil {
		t.Fatalf("got an err: %s", err.Error())
	}
}

The issue is with the loop in at mapstructure.go:609.

for i := 0; i < dataVal.Len(); i++ {
	currentData := dataVal.Index(i).Interface()
	currentField := valSlice.Index(i)

	fieldName := fmt.Sprintf("%s[%d]", name, i)
	if err := d.decode(fieldName, currentData, currentField); err != nil {
		errors = appendErrors(errors, err)
	}
}

If valSlice is shorter than dataVal we get a panic.

The question is: Should this throw an error, or automatically lengthen the input slice? I would guess the later would be appreciated (and that was the behavior before the pull request).

Allow anonymous non struct fields

Hi!
I'm facing an issue on traefik traefik/traefik#743 that prevents us from using an anonymous slice field in a struct:

type Users []string

type Basic struct {
    Users `mapstructure:","` // explicitly don't squash 
}

Calling Decode() on Basic will produce the error Users: unsupported type: slice.
This behavior is due to the test

mapstructure/mapstructure.go

Line 654 in a6ef2f0

if fieldType.Anonymous {

which I'm not sure is needed as few lines later you have

mapstructure/mapstructure.go

Line 673 in a6ef2f0

if fieldKind != reflect.Struct {

Commenting the first test fixes my issue.
What are your thoughts on this?

Pre-decode hook?

Is there a way to set a transformation function for before decoding?

DecodeHook: func(from reflect.Kind, to reflect.Kind, v interface{}) (interface{}, error) {
  fmt.Println(from, to)
  if from == reflect.String && to == reflect.Map {
    return map[string]interface{}{
      "id": v,
    }, nil
  }

  return v, nil
},

I wanted to do something like that before decoding / before mapstructure panics.

Support paths in the mapstructure name tag

When dealing with convoluted JSON hierarchies, I'd love to be able to write something like

type MyType struct {
 Field1 string `mapstructure:"/path/to/some/data/i/want"`
 Field2 int    `mapstructure:"/path/to/other/data/i/care/about"`
 Field3 []struct{
  Field4 float64 `mapstructure:"/support/for/nested/constructions"`
  ...
 } `mapstructure:"/would/be/nice/too"`
}

rather than typecast my way through all those map[string]interfaces{}, or define a bazillion intermediate types.

I think this can be implemented in a backwards-compatible way as well, since you can just check if the first character of the mapstructure tag is a legal Go identifier character, and if not, interpret the tag as a path (up to the first comma, to support attributes like squash). Said first character can then be used as the component separator, giving the design some extra flexibility. (Of course, I'm making the assumption here that most end-users of the package have mapstructure names that don't begin with special characters - those that do need to change their code so that the tag begins with a unique special character, to simulate a single-element path.)

unknown configuration key: "InternetAccessible.Zone"

Hi,

I'm a Go and mapstructure newbie, and I appreciate your help.
Why does the following code give unknown configuration key "InternetAccessible.Zone", and how do I fix it?

type (
	ABCInternetAccessible struct {
		Data string
	}
	ABCPlacement struct {
		Zone string `mapstructure:"Zone"`
	}

	Config struct {

		// Fields from config file
		InternetAccessible ABCInternetAccessible `mapstructure:"InternetAccessible"`
		Placement          ABCPlacement          `mapstructure:"Placement"`
		Location           string
		Ctx                interpolate.Context
	}
)

func main() {
	rawData := map[string]interface{}{
		"InternetAccessible": map[string]interface{}{"Zone": "Arh..."},
		"Location":           "Here",
	}
	var c Config
	err := config.Decode(&c, &config.DecodeOpts{
		Interpolate:        true,
		InterpolateContext: &c.Ctx,
		InterpolateFilter: &interpolate.RenderFilter{
			Exclude: []string{},
		},
	}, rawData)
	log.Printf("Err: %+v", err)
}

Decode into actual type pointed using interface{}

Here's the example:

type StructX struct {
    Field1 string
    Field2 ...
}

type Toplevel struct {
    Data interface{}
}

data := map[string]interface{}{
    "Data": map[string]interface{}{
        "Field1": "value1",
        ...
    },
}

out := &Toplevel{Data: &StructX{}}
decoder, _ := mapstructure.NewDecoder(mapstructure.DecoderConfig{Result: out})
decoder.Decode(data)

The expected result is out.Data is still pointing to the original structure and everything is filled out in the structure. However, the actual result is out.Data is overwritten and pointed to a map.

The fix should be simple, in decodeBasic:

if val.IsValid() {
   return d.decode(name, data, reflect.ValueOf(val.Interface())
}
...