iancoleman / orderedmap Goto Github PK

orderedmap is a golang map where the keys keep the order that they're added. It can be de/serialized from/to JSON. It's based closely on the python collections.OrderedDict.

License: MIT License

Go 100.00%

orderedmap's Introduction

orderedmap

A golang data type equivalent to python's collections.OrderedDict

Retains order of keys in maps

Can be JSON serialized / deserialized

Usage

package main

import (
    "encoding/json"
    "github.com/iancoleman/orderedmap"
)

func main() {

    // use New() instead of o := map[string]interface{}{}
    o := orderedmap.New()

    // use SetEscapeHTML() to whether escape problematic HTML characters or not, defaults is true
    o.SetEscapeHTML(false)

    // use Set instead of o["a"] = 1
    o.Set("a", 1)

    // add some value with special characters
    o.Set("b", "\\.<>[]{}_-")

    // use Get instead of i, ok := o["a"]
    val, ok := o.Get("a")

    // use Keys instead of for k, v := range o
    keys := o.Keys()
    for _, k := range keys {
        v, _ := o.Get(k)
    }

    // use o.Delete instead of delete(o, key)
    o.Delete("a")

    // serialize to a json string using encoding/json
    bytes, err := json.Marshal(o)
    prettyBytes, err := json.MarshalIndent(o, "", "  ")

    // deserialize a json string using encoding/json
    // all maps (including nested maps) will be parsed as orderedmaps
    s := `{"a": 1}`
    err := json.Unmarshal([]byte(s), &o)

    // sort the keys
    o.SortKeys(sort.Strings)

    // sort by Pair
    o.Sort(func(a *orderedmap.Pair, b *orderedmap.Pair) bool {
        return a.Value().(float64) < b.Value().(float64)
    })
}

Caveats

OrderedMap only takes strings for the key, as per the JSON spec.

Tests

go test

Alternatives

None of the alternatives offer JSON serialization.

orderedmap's People

Contributors

Stargazers

Watchers

orderedmap's Issues

Open or close brace in key or value breaks unmarshal

Trying to unmarshal a json string where there are unmatched braces in keys or values will break the unmarshalling.

In my experience it leads to keys being excluded or to panics.

The bad lines are https://github.com/iancoleman/orderedmap/blob/master/orderedmap.go#L92-L93

syntax error in the latest merge

the latest merge has a syntax error

`$ go get github.com/chrusty/protoc-gen-jsonschema/cmd/protoc-gen-jsonschema

github.com/iancoleman/orderedmap

../../go/src/github.com/iancoleman/orderedmap/orderedmap.go:7:2: imported and not used: "strings"
`

Use json.Number to prevent precision loss

I tried to unmarshal my json string into an orderedmap, this is my string:
"{\n \"IDs\": [\n 7236290603911250220\n ] \n}"

But I got the result:

I check the source code, and find out that the func UnmarshalJSON of OrderedMap does not use json.Number when decoding string into values, which leads to the precision loss in this issue.

I think it's necessary to use json.Number like this:
dec := json.NewDecoder(bytes.NewReader(b)) dec.UseNumber() err := dec.Decode(&o.values) if err != nil { return err }

Improve Unmarshal efficiency

Unmarshalling maps into orderedmaps is really inefficient because it does a lot of string allocation and failing json.Unmarshal calls.

Maybe try to develop something modelled on encoding/json:decode.go, especially looking at the objectInterface.

It's also a bit unreliable how it is since it has to detect nested objects and doing that with string manipulation is just not that great.

TestOrderedMap has a redundant for loop

orderedmap/orderedmap_test.go

Line 68 in 7c914fa

for i, key := range expectedKeys {

	// Keys method
	keys := o.Keys()
	expectedKeys := []string{
		"number",
		"string",
		"strings",
		"mixed",
	}
	for i, key := range keys {
		if key != expectedKeys[i] {
			t.Error("Keys method", key, "!=", expectedKeys[i])
		}
	}
	for i, key := range expectedKeys {
		if key != expectedKeys[i] {
			t.Error("Keys method", key, "!=", expectedKeys[i])
		}
	}

The second for loop is not meaningful. I'm assuming that it was meant for a values method or to test values. It's possible it was also copy-pasta.

Cannot decode deep 3 map in slices

Test json (added [{"inner": "map"}]) :

`{
  "number": 4,
  "string": "x",
  "z": 1,
  "a": "should not break with unclosed { character in value",
  "b": 3,
  "slice": [
    "1",
    1
  ],
  "orderedmap": {
    "e": 1,
    "a { nested key with brace": "with a }}}} }} {{{ brace value",
	"after": {
		"link": "test {{{ with even deeper nested braces }"
	}
  },
  "test\"ing": 9,
  "after": 1,
  "multitype_array": [
    "test",
	1,
	{ "map": "obj", "it" : 5, ":colon in key": "colon: in value" },
	[{"inner": "map"}]
  ],
  "should not break with { character in key": 1
}`

Failing part:

vislice, _ := o.Get("multitype_array")
vslice := vislice.([]interface{})
expectedKeys = []string{"inner"}
vinnerslice := vslice[3].([]interface{})
vinnermap := vinnerslice[0].(OrderedMap)
k = vinnermap.Keys()
for i := range k {
	if k[i] != expectedKeys[i] {
		t.Error("Key order for nested map 3 deep", i, k[i], "!=", expectedKeys[i])
	}
}

Failing message:

panic: interface conversion: interface {} is map[string]interface {}, not orderedmap.OrderedMap [recovered]
	panic: interface conversion: interface {} is map[string]interface {}, not orderedmap.OrderedMap

Reason: The decoding logic doesn't do recursive parsing on [array of elements]

Please provide a Values() method that returns the underlying Go map

Hi,

First of all, many thanks for this code. It would be helpful if you can add a Values() method similar to the Keys() method to return the underlying Go map. Right now, I have to iterate over keys and get their value to construct a similar Go map.

I need the raw map, for example, to pass it to the mapstructure package to convert the map into a struct.

I'd be happy to create a PR.

Thanks.

Empty array [] is decoded to nil

It seems that [] in value is decoded to []interface{}(nil) but I expect []interface{}{}

Sample code

package main

import (
	"encoding/json"
	"fmt"

	"github.com/iancoleman/orderedmap"
)

func main() {
	src := []byte(`{"x":[]}`)

	var m interface{}
	json.Unmarshal(src, &m)
	fmt.Printf("%#v\n", m)
	bs, _ := json.Marshal(m)
	fmt.Printf("%s\n", string(bs))

	om := orderedmap.New()
	json.Unmarshal(src, om)
	fmt.Printf("%#v\n", om)
	bs, _ = json.Marshal(om)
	fmt.Printf("%s\n", string(bs))
}

Actual behavior

map[string]interface {}{"x":[]interface {}{}}
{"x":[]}
&orderedmap.OrderedMap{keys:[]string{"x"}, values:map[string]interface {}{"x":[]interface {}(nil)}}
{"x":null}

Expected behavior

map[string]interface {}{"x":[]interface {}{}}
{"x":[]}
&orderedmap.OrderedMap{keys:[]string{"x"}, values:map[string]interface {}{"x":[]interface {}{}}
{"x":[]}

how to unmarshal json array

s := `[{"a":1, "b":2},{"a":0, "b":3},]`
o := orderedmap.New()
err := json.Unmarshal([]byte(s), &o)

however, it does not work

How do i use it in template?

{{range $i, $n := .columns.Get "ID"}}{{end}}
Doesn't work then get error:

can't call method/function "Get" with 2 results

Add benchmarks/performance notes

I'd love to see some benchmarks to compare this to using a regular map or just using a slice containing a Key and Value.

I'd also love a little section in the README describing the big-O computational complexity of each operation.

This would really differentiate this little library from the others.

Thanks!

Cannot parse escaped character in object key

Description

This library ignores object keys which contains escape characters.

Example

package main

import (
	"encoding/json"
	"fmt"
	"log"

	"github.com/iancoleman/orderedmap"
)

func main() {
	src := `{ "\u0041": 1 }`

	o := orderedmap.New()
	if err := json.Unmarshal([]byte(src), &o); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%#v\n", o)

	fmt.Printf("%#v\n", o.Keys())
	for _, k := range o.Keys() {
		v, _ := o.Get(k)
		fmt.Printf("%q %#v\n", k, v)
	}
	bs, err := json.Marshal(o)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%s\n", bs)

	fmt.Println("= = = = = = = = = =")
	var v map[string]interface{}
	if err := json.Unmarshal([]byte(src), &v); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%#v\n", v)
	for k, v := range v {
		fmt.Printf("%q %#v\n", k, v)
	}
	bs, err = json.Marshal(v)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%s\n", bs)
}

When I run the above code, I get

&orderedmap.OrderedMap{keys:[]string{}, values:map[string]interface {}{"A":1}, escapeHTML:true}
[]string{}
{}
= = = = = = = = = =
map[string]interface {}{"A":1}
"A" 1
{"A":1}

but I expect

&orderedmap.OrderedMap{keys:[]string{"A"}, values:map[string]interface {}{"A":1}, escapeHTML:true}
[]string{"A"}
"A" 1
{"A":1}
= = = = = = = = = =
map[string]interface {}{"A":1}
"A" 1
{"A":1}

Per spec

Due to the JSON spec, key of an object is a string, which can contain some escaped characters. Hence { "\u0041": 1 }, { "\n": 1 }, { "\\": 1 } are all valid JSON while this library does not support.

Environment

Go 1.15.6, darwin/amd64
github.com/iancoleman/orderedmap v0.1.0

JSON decoder.UseNumber not working

func TestOrderedMap_UseNumber(t *testing.T) {
	srcStr := `{"x":1}`
	om := New()
	d := json.NewDecoder(strings.NewReader(srcStr))
	d.UseNumber()
	d.Decode(om)
	if v, ok := om.Get("x"); ok {
		if _, ok := v.(json.Number); !ok {
			t.Errorf("Expect type is json.Number, got %v", reflect.TypeOf(v))
		}
	}
}

Shows:
Expect type is json.Number, got float64

MarshalJSON outputs too many newlines

Current implementation:

		// add key
		if err := encoder.Encode(k); err != nil {
			return nil, err
		}
		buf.WriteByte(':')
		// add value
		if err := encoder.Encode(o.values[k]); err != nil {
			return nil, err
		}

encoder.Encode will always add a newline:

	// Terminate each value with a newline.
	// This makes the output look a little nicer
	// when debugging, and some kind of space
	// is required if the encoded value was a number,
	// so that the reader knows there aren't more
	// digits coming.
	e.WriteByte('\n')

So we always get a newline after the key and after the value:

{"id"
:"123456"
,"status"
:"active"

This could be avoided by using json.Marshal(), but json.Marshal doesn't support turning off HTML encoding. So in order to preserve BC, we need to have a helper that will strip the last byte from the encoded output.

Consider erroring on JSON duplicate keys instead of "last value wins"

For an implementation with this rule applied, see https://github.com/Cyphrme/Coze/blob/76f019b67712c0dc16cdde9895cff5448342d63f/orderedmap.go#L31

Duplicate fields are a problem in JSON that wasn't addressed by the original spec.

After JSON was widely adopted, Douglas Crockford, JSON's inventor, tried to fix this but it was decided it
was too late.

Although Douglas Crockford couldn't change the spec forcing all implementations
to error on duplicate, his Java JSON implementation errors on duplicate names.
Others use last-value-wins, support duplicate keys, or other non-standard
behavior. The JSON
RFC states that
implementations should not allow duplicate keys, notes the varying behavior
of existing implementations, and states that when names are not unique, "the
behavior of software that receives such an object is unpredictable." Also note
that Javascript objects (ES6) and Go structs already require unique names.

Duplicate fields are a security issue, a source of bugs, and a surprising
behavior to users. See the article, "An Exploration of JSON Interoperability
Vulnerabilities"

Disallowing duplicates conforms to the small I-JSON RFC. The author of I-JSON,
Tim Bray, is also the author of current JSON specification (RFC 8259). See also
json5/json5-spec#38.