Giter Club home page Giter Club logo

go-yaml's Introduction

YAML support for the Go language

PkgGoDev Go codecov Go Report Card

Why a new library?

As of this writing, there already exists a de facto standard library for YAML processing for Go: https://github.com/go-yaml/yaml. However we feel that some features are lacking, namely:

  • Pretty format for error notifications
  • Direct manipulation of YAML abstract syntax tree
  • Support for Anchor and Alias when marshaling
  • Allow referencing elements declared in another file via anchors

Features

  • Pretty format for error notifications
  • Supports Scanner or Lexer or Parser as public API
  • Supports Anchor and Alias to Marshaler
  • Allow referencing elements declared in another file via anchors
  • Extract value or AST by YAMLPath ( YAMLPath is like a JSONPath )

Installation

go get -u github.com/goccy/go-yaml

Synopsis

1. Simple Encode/Decode

Has an interface like go-yaml/yaml using reflect

var v struct {
	A int
	B string
}
v.A = 1
v.B = "hello"
bytes, err := yaml.Marshal(v)
if err != nil {
	//...
}
fmt.Println(string(bytes)) // "a: 1\nb: hello\n"
	yml := `
%YAML 1.2
---
a: 1
b: c
`
var v struct {
	A int
	B string
}
if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
	//...
}

To control marshal/unmarshal behavior, you can use the yaml tag.

	yml := `---
foo: 1
bar: c
`
var v struct {
	A int    `yaml:"foo"`
	B string `yaml:"bar"`
}
if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
	//...
}

For convenience, we also accept the json tag. Note that not all options from the json tag will have significance when parsing YAML documents. If both tags exist, yaml tag will take precedence.

	yml := `---
foo: 1
bar: c
`
var v struct {
	A int    `json:"foo"`
	B string `json:"bar"`
}
if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
	//...
}

For custom marshal/unmarshaling, implement either Bytes or Interface variant of marshaler/unmarshaler. The difference is that while BytesMarshaler/BytesUnmarshaler behaves like encoding/json and InterfaceMarshaler/InterfaceUnmarshaler behaves like gopkg.in/yaml.v2.

Semantically both are the same, but they differ in performance. Because indentation matters in YAML, you cannot simply accept a valid YAML fragment from a Marshaler, and expect it to work when it is attached to the parent container's serialized form. Therefore when we receive use the BytesMarshaler, which returns []byte, we must decode it once to figure out how to make it work in the given context. If you use the InterfaceMarshaler, we can skip the decoding.

If you are repeatedly marshaling complex objects, the latter is always better performance wise. But if you are, for example, just providing a choice between a config file format that is read only once, the former is probably easier to code.

2. Reference elements declared in another file

testdata directory contains anchor.yml file:

├── testdata
   └── anchor.yml

And anchor.yml is defined as follows:

a: &a
  b: 1
  c: hello

Then, if yaml.ReferenceDirs("testdata") option is passed to yaml.Decoder, Decoder tries to find the anchor definition from YAML files the under testdata directory.

buf := bytes.NewBufferString("a: *a\n")
dec := yaml.NewDecoder(buf, yaml.ReferenceDirs("testdata"))
var v struct {
	A struct {
		B int
		C string
	}
}
if err := dec.Decode(&v); err != nil {
	//...
}
fmt.Printf("%+v\n", v) // {A:{B:1 C:hello}}

3. Encode with Anchor and Alias

3.1. Explicitly declared Anchor name and Alias name

If you want to use anchor or alias, you can define it as a struct tag.

type T struct {
  A int
  B string
}
var v struct {
  C *T `yaml:"c,anchor=x"`
  D *T `yaml:"d,alias=x"`
}
v.C = &T{A: 1, B: "hello"}
v.D = v.C
bytes, err := yaml.Marshal(v)
if err != nil {
  panic(err)
}
fmt.Println(string(bytes))
/*
c: &x
  a: 1
  b: hello
d: *x
*/

3.2. Implicitly declared Anchor and Alias names

If you do not explicitly declare the anchor name, the default behavior is to use the equivalent of strings.ToLower($FieldName) as the name of the anchor.

If you do not explicitly declare the alias name AND the value is a pointer to another element, we look up the anchor name by finding out which anchor field the value is assigned to by looking up its pointer address.

type T struct {
	I int
	S string
}
var v struct {
	A *T `yaml:"a,anchor"`
	B *T `yaml:"b,anchor"`
	C *T `yaml:"c,alias"`
	D *T `yaml:"d,alias"`
}
v.A = &T{I: 1, S: "hello"}
v.B = &T{I: 2, S: "world"}
v.C = v.A // C has same pointer address to A
v.D = v.B // D has same pointer address to B
bytes, err := yaml.Marshal(v)
if err != nil {
	//...
}
fmt.Println(string(bytes)) 
/*
a: &a
  i: 1
  s: hello
b: &b
  i: 2
  s: world
c: *a
d: *b
*/

3.3 MergeKey and Alias

Merge key and alias ( <<: *alias ) can be used by embedding a structure with the inline,alias tag.

type Person struct {
	*Person `yaml:",omitempty,inline,alias"` // embed Person type for default value
	Name    string `yaml:",omitempty"`
	Age     int    `yaml:",omitempty"`
}
defaultPerson := &Person{
	Name: "John Smith",
	Age:  20,
}
people := []*Person{
	{
		Person: defaultPerson, // assign default value
		Name:   "Ken",         // override Name property
		Age:    10,            // override Age property
	},
	{
		Person: defaultPerson, // assign default value only
	},
}
var doc struct {
	Default *Person   `yaml:"default,anchor"`
	People  []*Person `yaml:"people"`
}
doc.Default = defaultPerson
doc.People = people
bytes, err := yaml.Marshal(doc)
if err != nil {
	//...
}
fmt.Println(string(bytes))
/*
default: &default
  name: John Smith
  age: 20
people:
- <<: *default
  name: Ken
  age: 10
- <<: *default
*/

4. Pretty Formatted Errors

Error values produced during parsing have two extra features over regular error values.

First, by default, they contain extra information on the location of the error from the source YAML document, to make it easier to find the error location.

Second, the error messages can optionally be colorized.

If you would like to control exactly how the output looks like, consider using yaml.FormatError, which accepts two boolean values to control turning these features on or off.

5. Use YAMLPath

yml := `
store:
  book:
    - author: john
      price: 10
    - author: ken
      price: 12
  bicycle:
    color: red
    price: 19.95
`
path, err := yaml.PathString("$.store.book[*].author")
if err != nil {
  //...
}
var authors []string
if err := path.Read(strings.NewReader(yml), &authors); err != nil {
  //...
}
fmt.Println(authors)
// [john ken]

5.1 Print customized error with YAML source code

package main

import (
  "fmt"

  "github.com/goccy/go-yaml"
)

func main() {
  yml := `
a: 1
b: "hello"
`
  var v struct {
    A int
    B string
  }
  if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
    panic(err)
  }
  if v.A != 2 {
    // output error with YAML source
    path, err := yaml.PathString("$.a")
    if err != nil {
      panic(err)
    }
    source, err := path.AnnotateSource([]byte(yml), true)
    if err != nil {
      panic(err)
    }
    fmt.Printf("a value expected 2 but actual %d:\n%s\n", v.A, string(source))
  }
}

output result is the following:

Tools

ycat

print yaml file with color

ycat

Installation

go install github.com/goccy/go-yaml/cmd/ycat@latest

Looking for Sponsors

I'm looking for sponsors this library. This library is being developed as a personal project in my spare time. If you want a quick response or problem resolution when using this library in your project, please register as a sponsor. I will cooperate as much as possible. Of course, this library is developed as an MIT license, so you can use it freely for free.

License

MIT

go-yaml's People

Contributors

al2klimov avatar braydonk avatar chanxuehong avatar cmaster11 avatar goccy avatar hasheddan avatar itchyny avatar jeffalder avatar johejo avatar k1low avatar kitagry avatar kyoh86 avatar lestrrat avatar martin-sucha avatar mumoshu avatar nakabonne avatar nasa9084 avatar navteniev avatar nervo avatar ondrej-fabry avatar ozraru avatar quasilyte avatar quentinmit avatar rspruyt-google avatar shogo82148 avatar songmu avatar testwill avatar vikstrous2 avatar wass3rw3rk avatar zoncoen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-yaml's Issues

yaml.BytesUnmarshaler receives quoted string

package yaml

import (
	"encoding/json"
	"testing"
)

type unmarshalableStringValue string

func (v *unmarshalableStringValue) UnmarshalYAML(raw []byte) error {
	*v = unmarshalableStringValue(string(raw))
	return nil
}

type unmarshalableStringContainer struct {
	V unmarshalableStringValue `yaml:"value" json:"value"`
}

func TestUnmarshalableString(t *testing.T) {
	var container unmarshalableStringContainer
	t.Run("empty string", func(t *testing.T) {
		if err := Unmarshal([]byte(`value: ""`), &container); err != nil {
			t.Fatalf("failed to unmarshal %v", err)
		}
		if container.V != "" {
			t.Fatalf("expected empty string, but %q is set", container.V)
		}
	})
	t.Run("filled string", func(t *testing.T) {
		if err := Unmarshal([]byte(`value: "aaa"`), &container); err != nil {
			t.Fatalf("failed to unmarshal %v", err)
		}
		if container.V != "aaa" {
			t.Fatalf("expected empty string, but %q is set", container.V)
		}
	})
	t.Run("(json) empty string", func(t *testing.T) {
		if err := json.Unmarshal([]byte(`{"value": ""}`), &container); err != nil {
			t.Fatalf("failed to unmarshal %v", err)
		}
		if container.V != "" {
			t.Fatalf("expected empty string, but %q is set", container.V)
		}
	})
	t.Run("(json) filled string", func(t *testing.T) {
		if err := json.Unmarshal([]byte(`{"value": "aaa"}`), &container); err != nil {
			t.Fatalf("failed to unmarshal %v", err)
		}
		if container.V != "aaa" {
			t.Fatalf("expected empty string, but %q is set", container.V)
		}
	})
}

This test will be failed.

source: https://github.com/kyoh86/go-yaml/blob/empty-string/decode_unmarshalable_string_test.go

Distinguish between empty slice and nil slice OR Encode both empty and nil slices into `[]`

Hi @goccy !!

Thank you for your GREAT library !!

I want to encode "empty slices" and "nil slices" as follows

  • Plan 1: Distinguish between empty slice and nil slice like encoding/json
  • Plan 2: Encode both empty and nil slices into [] like go-yaml/yaml
go-yaml/yaml encoding/json goccy/go-yaml.Flow(false) ( default ) goccy/go-yaml.Flow(true) goccy/go-yaml.Flow(Plan 1) goccy/go-yaml.Flow(Plan 2)
empty slice ( []string{} ) a: [] {"a": []} a: a: [] a: [] a: []
nil slice ( []string(nil) ) a: [] {"a": null} a: a: [] a: a: []
play.golang.org OR commit here here here here commit commit

What do you think about handling this empty slice and nil slice ?

Single '{' at the beginning of strings w/o matching closing brace causes panic

I have the following code snippet:

package main

import (
  "fmt"
  "github.com/goccy/go-yaml"
)

func main() {
  var result string
  err := yaml.Unmarshal([]byte("{this is invalid"), &result)
  if err != nil {
    fmt.Printf("Error: %v", err)
  }else{
    fmt.Printf("Got: %s", result)
  }
}

Playground: https://play.golang.org/p/9nZNkWEopY6

Obviously, I'm trying to unmarshal an invalid Yaml string (unmatched, leading curly brace).

Expectation: A non-nil error is returned.
Actual Behaviour: A panic (nil-pointer dereference) is raised (see log at the end of the post).

Interestingly, the string "{{this is invalid}" returns a non-nil error, as well as the string "this is{ invalid". The panic only seems to occur when the curly brace is at the beginning of the string and there are no other curly braces anywhere.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x4e40ad]

goroutine 1 [running]:
github.com/goccy/go-yaml/parser.(*parser).createNullToken(...)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/parser/parser.go:132
github.com/goccy/go-yaml/parser.(*parser).parseMapValue(0xc000068c70, 0xc000060450, 0x556c60, 0xc00000c0a0, 0x0, 0x0, 0x10, 0x10, 0x513040)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/parser/parser.go:140 +0x42d
github.com/goccy/go-yaml/parser.(*parser).parseMappingValue(0xc000068c70, 0xc000060450, 0x0, 0x0, 0x0, 0x0)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/parser/parser.go:208 +0x1d4
github.com/goccy/go-yaml/parser.(*parser).parseMapping(0xc000068c70, 0xc000060450, 0xc00005c050, 0x0, 0x0, 0x0)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/parser/parser.go:30 +0x1e5
github.com/goccy/go-yaml/parser.(*parser).parseToken(0xc000068c70, 0xc000060450, 0xc00005c050, 0x0, 0xc000060450, 0xc000010280, 0x2)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/parser/parser.go:526 +0x1b5
github.com/goccy/go-yaml/parser.(*parser).parse(0xc000068c70, 0xc000010280, 0x2, 0x2, 0x0, 0x0, 0x1, 0x3)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/parser/parser.go:549 +0x105
github.com/goccy/go-yaml/parser.Parse(0xc000010280, 0x2, 0x2, 0x0, 0x2, 0xc000068cf8, 0x4ba637)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/parser/parser.go:585 +0x57
github.com/goccy/go-yaml/parser.ParseBytes(0xc000100000, 0x10, 0x40, 0x0, 0xc000060420, 0x10, 0x0)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/parser/parser.go:575 +0x8d
github.com/goccy/go-yaml.(*Decoder).parse(0xc00007e000, 0xc000100000, 0x10, 0x40, 0x0, 0x0, 0x0)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/decode.go:1291 +0x55
github.com/goccy/go-yaml.(*Decoder).decodeInit(0xc00007e000, 0x16, 0x16)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/decode.go:1319 +0xe9
github.com/goccy/go-yaml.(*Decoder).Decode(0xc00007e000, 0x5056e0, 0xc000010270, 0xc000068ee8, 0x4c6af2)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/decode.go:1361 +0x239
github.com/goccy/go-yaml.UnmarshalWithOptions(0xc00002c040, 0x10, 0x10, 0x5056e0, 0xc000010270, 0x0, 0x0, 0x0, 0x405d65, 0xc00005e058)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/yaml.go:161 +0x1a6
github.com/goccy/go-yaml.Unmarshal(...)
	/tmp/gopath386768941/pkg/mod/github.com/goccy/[email protected]/yaml.go:154
main.main()
	/tmp/sandbox860830854/prog.go:10 +0xad


Windows line endings not supported

When yaml files are read, that have windows line endings (CRLF), the application fails, as it does not recognize the YAML file correctly.

I created a small example repository that showcases the issue. You can check it out at TobsCore/yaml-crlf-error.

The error that occurs is as follows:

panic: Cannot parse yaml [2:1] unexpected key name
       1 | ---
    >  2 | name: TobsCore
       3 | age: 40
          ^

goroutine 1 [running]:
main.main()
	/tmp/yaml-crlf-error/main.go:26 +0x240
exit status 2

Panic when parsing empty mappings

Parsing this doc:

  • foo: { }

causes panic in yaml_ast:283 (parseMappingNode), b/c Values slice is empty.

A work-around that I've found is to explicitly check for empty, and use node.GetToken() in such case - but perhaps it would make sense to revise node.GetToken() so that it could be used regardless of whether the mapping is empty. (I am not sure what node.GetToken(), resolving to node.Start, is meant to indicate, but it doesn't point to the first token of the mapping, which I'd expect from the name).

Allow partial Unmarshal to retain anchors and aliases without resolving them

Taking a look at an issue over in Helm where we need to be able to merge several yaml and various other value formats into a single file but we'd like to retain the anchors and possibly aliases.

My initial thought here is we'd probably prefer to to a partial unmarshal that doesn't resolve anchors and alias so we can detect when we are dealing with a literal vs an anchor/alias.

Use case is here:
helm/helm#6699

Two levels of inline depth results in reflection error

Hey @goccy,

I've added the following test:

func TestDecoder_InlineDoubleBase(t *testing.T) {
	type Base struct {
		A int
		B string
	}
	yml := `---
a: 1
b: hello
`
	type Base2 struct {
		Base *Base `yaml:",inline"`
	}
	var v struct {
		Base2 *Base2 `yaml:",inline"`
	}
	if err := yaml.NewDecoder(strings.NewReader(yml), yaml.Strict()).Decode(&v); err != nil {
		t.Fatalf("%+v", err)
	}
	if v.Base2.Base.A != 1 {
		t.Fatal("failed to decode with inline key")
	}
	if v.Base2.Base.B != "hello" {
		t.Fatal("failed to decode with inline key")
	}
}

the test resulted in the following reflect panic

--- FAIL: TestDecoder_InlineDoubleTest (0.00s)
panic: reflect: call of reflect.Value.FieldByName on ptr Value [recovered]
        panic: reflect: call of reflect.Value.FieldByName on ptr Value

goroutine 27 [running]:
testing.tRunner.func1.1(0x12099c0, 0xc00032c920)
        /usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:941 +0x3d0
testing.tRunner.func1(0xc0003306c0)
        /usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:944 +0x3f9
panic(0x12099c0, 0xc00032c920)
        /usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:967 +0x15d
reflect.flag.mustBe(...)
        /usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:208
reflect.Value.FieldByName(0x11efee0, 0xc0002c5bc8, 0x196, 0x11e87a4, 0x4, 0x11eff01, 0x11e87aa, 0xe)
        /usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:880 +0x1ed
github.com/goccy/go-yaml.(*Decoder).deleteStructKeys(0xc0003034a0, 0x11efee0, 0xc0002c5bc8, 0x196, 0xc0003343c0, 0x12153e0, 0xc0002c5c20)
        /Users/user/work/go-yaml/decode.go:386 +0x238
github.com/goccy/go-yaml.(*Decoder).decodeStruct(0xc0003034a0, 0x1211da0, 0xc0002c5bc8, 0x199, 0x12ae720, 0xc000320d00, 0x28, 0x121d6c0)
        /Users/user/work/go-yaml/decode.go:788 +0xde7
github.com/goccy/go-yaml.(*Decoder).decodeValue(0xc0003034a0, 0x1211da0, 0xc0002c5bc8, 0x199, 0x12ae720, 0xc000320d00, 0x0, 0x12)
        /Users/user/work/go-yaml/decode.go:474 +0x1b0a
github.com/goccy/go-yaml.(*Decoder).decode(0xc0003034a0, 0x11f5460, 0xc0002c5bc8, 0x16, 0x11f5460, 0x1415800)
        /Users/user/work/go-yaml/decode.go:1244 +0xd2
github.com/goccy/go-yaml.(*Decoder).Decode(0xc0003034a0, 0x11f5460, 0xc0002c5bc8, 0x10565f9, 0x7380299ca20cb)
        /Users/user/work/go-yaml/decode.go:1273 +0x272
github.com/goccy/go-yaml_test.TestDecoder_Inline(0xc0003306c0)
        /Users/user/work/go-yaml/decode_test.go:1315 +0x1da
testing.tRunner(0xc0003306c0, 0x125f6f0)
        /usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:992 +0xdc
created by testing.(*T).Run
        /usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:1043 +0x357
FAIL    github.com/goccy/go-yaml        0.519s
FAIL

DisallowDuplicateKey doesn't work when decoding into struct

Hey @goccy,

I've added the following test:

func TestDecoder_DisallowDuplicateKeyOnStruct(t *testing.T) {  
        yml := `                                                                                             
a: b              
a: c                                
`     
        expected := `  
[3:1] duplicate key "a"           
   2 | a: b                  
>  3 | a: c                    
       ^                            
`           
        var c struct {                          
                A string `yaml:"a"`                                                                                                        
        }                                                                                       
        err := yaml.NewDecoder(strings.NewReader(yml), yaml.DisallowDuplicateKey()).Decode(&c)                     
        if err == nil {                                                                                                
                t.Fatalf("decoding should fail")                                                          
        }                                                                                                         
        actual := "\n" + err.Error()                                                                              
        if expected != actual {                                                                                   
                t.Fatalf("expected:[%s] actual:[%s]", expected, actual)                                           
        }                                                                                                         
}                                                                                                                                                                                                                                     

For some reason, Decode doesn't yield and error although I've specified DisallowDuplicateKey

--- FAIL: TestDecoder_DisallowDuplicateKeyOnStruct (0.00s)
    decode_test.go:1518: decoding should fail
FAIL
FAIL    github.com/goccy/go-yaml        0.396s
FAIL

go.mod out of date?

just running go test against b6878e0 causes go.mod to change.

finch% git diff
finch% go test . 
ok  	github.com/goccy/go-yaml	0.035s
finch% git diff
diff --git a/go.mod b/go.mod
index 4de68a0..f55929a 100644
--- a/go.mod
+++ b/go.mod
@@ -4,8 +4,11 @@ go 1.12
 
 require (
        github.com/fatih/color v1.7.0
+       github.com/go-playground/universal-translator v0.17.0 // indirect
+       github.com/leodido/go-urn v1.2.0 // indirect
        github.com/mattn/go-colorable v0.1.4
        github.com/mattn/go-isatty v0.0.10 // indirect
        golang.org/x/sys v0.0.0-20191010194322-b09406accb47 // indirect
        golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898
+       gopkg.in/go-playground/validator.v9 v9.30.0
 )
diff --git a/go.sum b/go.sum
index 8b1c262..b2f7875 100644
--- a/go.sum
+++ b/go.sum
@@ -1,15 +1,31 @@
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
+github.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8ceBS/t7Q=
+github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
+github.com/go-playground/universal-translator v0.17.0 h1:icxd5fm+REJzpZx7ZfpaD876Lmtgy7VtROAbHHXk8no=
+github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
+github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
+github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
 github.com/mattn/go-colorable v0.1.4 h1:snbPLB8fVfU9iwbbo30TPtbLRzwWu6aJS6Xh4eaaviA=
 github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
 github.com/mattn/go-isatty v0.0.8 h1:HLtExJ+uU2HOZ+wI0Tt5DtUDrx8yhUqDcp7fYERX4CE=
 github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
 github.com/mattn/go-isatty v0.0.10 h1:qxFzApOv4WsAL965uUPIsXzAKCZxN2p9UqdhFS4ZW10=
 github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223 h1:DH4skfRX4EBpamg7iV4ZlCpblAHI6s6TDM39bFZumv8=
 golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191010194322-b09406accb47 h1:/XfQ9z7ib8eEJX2hdgFTZJ/ntt0swNk5oYBziWeTCvY=
 golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/go-playground/validator.v9 v9.30.0 h1:Wk0Z37oBmKj9/n+tPyBHZmeL19LaCoK3Qq48VwYENss=
+gopkg.in/go-playground/validator.v9 v9.30.0/go.mod h1:+c9/zcJMFNgbLvly1L1V+PpxWdVbfP1avr/N00E2vyQ=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
finch% 

API to change error behavior

ドキュメントを読んでたら以下を見たのですが

errors.ColoredErr = false

yamlパッケージで色々処理を書いているのにerrorsのグローバル変数をいじってエラー表示方法を変えるのがちょっと微妙だなー、と思います。

  1. ユーザー的には errorsがpublicである必要性がない
  2. 必要ならerrorsじゃなくてyamlパッケージに対して処理を依頼するほうがユーザー的には自然
  3. そもそも、エラーを表示する方法を変えるだけなので、yaml側で変更するより、ユーザー側のコードで制御するほうが自然なのでは

と感じました。

今のAPIだとこうですが、

import (
   "github.com/goccy/go-yaml"
   "github.com/goccy/go-yaml/errors" // errors.ColoredErrのためだけに必要
)

if err := yaml.Unmarshal(...); err != nil {
   errors.ColoredErr = false
   fmt.Println(err)
}

こんな感じにしてしまえばよいかと

import (
   "github.com/goccy/go-yaml"
    // go-yaml/errorsは go-yaml/internal/errors に隠蔽してしまう
)

if err := yaml.Unmarshal(...); err != nil {
   yaml.PrettyPrintError(err) // or yaml.PrettyPrintError(err, boolColoredErr, boolIncludeSource)
}

とかにするとユーザー側としては

  1. そもそもerrorsパッケージを外部から利用する必要もなければ、internalだから利用もできない
  2. errorsではなくyamlパッケージにPrettyPrintを依頼する形になり自然
  3. 明示的に yaml.PrettyPrintError()を呼ぶことによって、ユーザー側で出力を変更できるし、グローバル変数にも依存しない

という形にできるかと思います。

よければPR書きます

[Question] tokens: Next/Prev and reserved keywords

  1. 現在、Tokens = []*Token で、添え字アクセスしつつ、Next/Prevを使うことをしてますが、これってNext/Prevだけでlinked listを作ればよくないですか?

  2. reserved tokensは一回作って使い回していますが、これって複数のgoroutineでアクセスするときとか、filteredTokensとかを作る時にNext/Prevとかの値が上書きされておかしなことになりませんか?reservedも動的に生成するようにしていいですか?

これら修正してよいなら、#48 の流れでついでにやりたいと思います。

bug on inline array reading

https://play.golang.org/p/l3cZy3n9T62

package main

import (
	"fmt"
	"github.com/goccy/go-yaml"
	y "gopkg.in/yaml.v2"
)

func main() {
	yml := `
- a: [2 , 2] 			
  b: [2 , 2] 			
  c: [2 , 2]`
	var d interface{}
	yaml.Unmarshal([]byte(yml), &d)
	fmt.Printf("%v\n%v\n", yml, d)
	y.Unmarshal([]byte(yml), &d)
	fmt.Printf("%v\n%v\n", yml, d)
}
- a: [2 , 2] 			
  b: [2 , 2] 			
  c: [2 , 2]
[map[a:[2 2]]]

- a: [2 , 2] 			
  b: [2 , 2] 			
  c: [2 , 2]
[map[a:[2 2] b:[2 2] c:[2 2]]]

Parsing YAML files with anchors does not keep pointers to anchors when unmarshalling

Thanks @goccy for your work on this package :)

Similar to #69, I have a configuration file that looks like this:

hosts:
  - host: &host1
      hostname: queue.example.com
      username: queue1
      password: queue1
      port: 5672

  - host: &host2
      hostname: queue2.example.com
      username: queue2
      password: queue2
      port: 5672

queues:
  - name: queue
    host: *host1
  - name: queue2
    host: *host1
  - name: queue3
    host: *host2

I have a set of structs in a separate package, describing the YAML file

package configuration

type HostList struct {
	Host *Host  `yaml:",inline,anchor"`
	Name string `yaml:","`
}

type Host struct {
	*Host    `yaml:",omitempty"`
	Hostname string
	Username string
	Password string
}

type Queue struct {
	Name  string `yaml:","`
	*Host `yaml:",alias"`
}

type ConfigFile struct {
	HostList []*HostList `yaml:"hosts"`
	Queues   []*Queue    `yaml:"queues"`
}

I am reading my configuration file like this

func Read(configFile string) (error, ConfigFile) {
	var cf ConfigFile
	yamlFile, err := ioutil.ReadFile(configFile)
	if err != nil {
		return err, cf
	}

	err = yaml.Unmarshal(yamlFile, &cf)
	if err != nil {
		// clear the config file that might have been partially parsed
		var cf ConfigFile
		return err, cf
	}

	return nil, cf
}

and inside my main.go, I have the following code:

	err, cf = configuration.Read(configFile)
	if err != nil {
		log.Fatal(err)
		return err
	}

	fmt.Println(Configuration.Queues[0].Host.Hostname)
	fmt.Println(Configuration.Queues[1].Host.Hostname)

	Configuration.Queues[0].Host.Hostname = "updated_hostname.example.org"
	
	fmt.Println(Configuration.Queues[0].Host.Hostname)
	fmt.Println(Configuration.Queues[1].Host.Hostname)

Expected output:

queue.example.com
queue.example.com
updated_hostname.example.org
updated_hostname.example.org

Actual output:

queue.example.com
queue.example.com
updated_hostname.example.org
queue.example.com

It seems that the anchor in the created document to host1 is not being respected when unmarshalling the document.

I would expect that both Configuration.Queues[0].Host and Configuration.Queues[1].Host would be a pointer to Configuration.Hosts[0].Host, reflecting the document's intent.

[Question] Tags support, explicit ast decoding?

I've noticed that the ast package has a TagNode. Are local tags (e.g. !Foo bar) supported by the higher level encoder/decoder?

I'm toying with the idea of implementing a YAML templating language, that is context aware, and local tags might be one way to use the YAML language in order to inject a custom type that can be replaced during decoding.

Another interesting option would be to have a public method in the Decoder that would take a direct reference to an ast.Node, allowing the AST to be manipulated previously to being decoded.

Overall, this library is looking great. Good work!

UPDATE: I just realized that explicit AST decoding wouldn't be useful unless the AST can be manipulated, which currently it's not possible. It's an interesting idea but definitely more work than I first though when I wrote the initial question.

%!v(PANIC=Error method: runtime error: slice bounds out of range [2:1])

Above error occurs when I write code below.
I found this occurs at printer.goL250. But, I don't understand how to fix it correctly. Sorry.

go-yaml/printer/printer.go

Lines 249 to 251 in 3032e43

if tk != nil {
tk.Origin = org[len(trimmed)+1:] + tk.Origin
}

type C struct {
	B string `yaml:"b"`
}
                           
var v struct {
	A C `yaml:"complicated"`
}

const src = `---
complecated:
  b: string
`

err := yaml.NewDecoder(strings.NewReader(src), yaml.DisallowUnknownField()).Decode(&v)
fmt.Printf("%v\n", err)

// OUTPUT:
// [2:1] unknown field "complecated"
//        1 | ---
//     >  2 | complecated:
//           ^
//        3 |   b: string

[Question] Is there an option doesn't allow different type?

Currently, github.com/goccy/go-yaml is executed as follows.
Is there an option that doesn't allow this?

type st struct {
	A int `json:"a"`
}

func main() {
	d := yaml.NewDecoder(strings.NewReader("a: string"))  // different type
	var a st
	err := d.Decode(&a)
	fmt.Println(a.A)    // --> 0
	fmt.Println(err)    // --> nil (But I want error)
}

Consider removing go.sum

I was just working on a separate PR, and go.sum just keeps changing.

diff --git a/go.sum b/go.sum
index 8b1c262..b2f7875 100644
--- a/go.sum
+++ b/go.sum
@@ -1,15 +1,31 @@
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
+github.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8ceBS/t7Q=
+github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
+github.com/go-playground/universal-translator v0.17.0 h1:icxd5fm+REJzpZx7ZfpaD876Lmtgy7VtROAbHHXk8no=
+github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
+github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
+github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
 github.com/mattn/go-colorable v0.1.4 h1:snbPLB8fVfU9iwbbo30TPtbLRzwWu6aJS6Xh4eaaviA=
 github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
 github.com/mattn/go-isatty v0.0.8 h1:HLtExJ+uU2HOZ+wI0Tt5DtUDrx8yhUqDcp7fYERX4CE=
 github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
 github.com/mattn/go-isatty v0.0.10 h1:qxFzApOv4WsAL965uUPIsXzAKCZxN2p9UqdhFS4ZW10=
 github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223 h1:DH4skfRX4EBpamg7iV4ZlCpblAHI6s6TDM39bFZumv8=
 golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191010194322-b09406accb47 h1:/XfQ9z7ib8eEJX2hdgFTZJ/ntt0swNk5oYBziWeTCvY=
 golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/go-playground/validator.v9 v9.30.0 h1:Wk0Z37oBmKj9/n+tPyBHZmeL19LaCoK3Qq48VwYENss=
+gopkg.in/go-playground/validator.v9 v9.30.0/go.mod h1:+c9/zcJMFNgbLvly1L1V+PpxWdVbfP1avr/N00E2vyQ=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

This would also be true if I happen to have a different setting for GOPROXY than you (@goccy), and referenced different versions somehow. Unless you are creating an application, I don't think keeping a go.sum in the repository of a library is good thing to do, as it's error prone when you make commits

fixing README

READMEの英語を直したいのですが、いくつかわからないところがあるので教えてください

If omitted anchor name, assigned default rendering name ( strings.ToLower(FieldName) ) as anchor name. If omitted alias name and it's field type is pointer type, assigned anchor name automatically from same pointer address.

よくわからなかったので日本語でどういうことなのか教えてくれるとうれしいです!

Marshal/Encode result of a float value without decimal part is not same with go-yaml/yaml

Marshal result of a float value without decimal part is not same with go-yaml/yaml.

package main

import (
	goccyyaml "github.com/goccy/go-yaml"
	goyaml "gopkg.in/yaml.v3"
	"os"
)

func main() {
	goyaml.NewEncoder(os.Stdout).Encode(1.0)
	goccyyaml.NewEncoder(os.Stdout).Encode(1.0)
}
1
1.0

This behavior is not also same with JSON marshalers and not comfortable when roundtrip.

Support dynamic anchor names, depending on value inside struct?

This is going to be a little tricky to describe, but hopefully with the attached code you should be able to see what I am trying to do.

I want to make a YAML document that looks something like:

hosts:
- host: &host1
    hostname: host1.example.com
    username: queue1
    password: queue1
- host: &host2
    hostname: host2.example.com
    username: queue1
    password: queue1
queues:
- name: queue
  host: *host1

This is the code that I have written at the moment:

import (
	"fmt"
	"github.com/goccy/go-yaml"
)

type HostList struct {
	Host *Host `yaml:",inline,anchor"`
	Name     string `yaml:","`
}

type Host struct {
	*Host `yaml:",omitempty"`
	Name string `yaml:",inline,anchorname"`
	Hostname string
	Username string
	Password string
}

type Queue struct {
	Name     string `yaml:","`
	*Host    `yaml:",alias"`
}

func main() {
	var doc struct {
		HostList []*HostList `yaml:"hosts"`
		Hosts    []*Host     `yaml:"hosts2"`
		Queues   []*Queue    `yaml:"queues"`
	}
	host1 := &Host {
			Hostname: "host1.example.com",
			Username: "queue1",
			Password: "queue1",
	}
	host2 := &Host {
			Hostname: "host2.example.com",
			Username: "queue1",
			Password: "queue1",
	}

	doc.HostList = []*HostList{
		{
			Name: "host1",
			Host: host1,
		},{
			Name: "host2",
			Host: host2,
		},
	}
	doc.Queues = []*Queue {
		{
			Name: "queue",
			Host: host1,
		},{
			Name: "queue2",
			Host: host2,
		},
	}

	bytes, err := yaml.Marshal(doc)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Printf("%s", string(bytes))
}

I can't seem to work out how to get the names out of the ast.

go-yaml/encode.go

Lines 394 to 403 in 009880f

case structField.IsAutoAnchor:
anchorName := structField.RenderName
if fieldValue.Kind() == reflect.Ptr {
e.anchorPtrToNameMap[fieldValue.Pointer()] = anchorName
}
value = &ast.AnchorNode{
Start: token.New("&", "&", e.pos(column)),
Name: ast.String(token.New(anchorName, anchorName, e.pos(column))),
Value: value,
}

Can I please get some help?

Inconsistent struct tag use

I guess the intent of this is to allow

type T struct {
    A `foo`
}

to just work, but I don't think this is a common practice (at least, encoding/json doesn't do that), and I also think it's a bad idea.

At least, the current code does not check for this:

type T struct {
    A `"foo"` // does NOT do what we want
}

of course, it's easy to handle this particular case, but it's just one more inconsistency. I think it's much much better to say "we support tag name X (and Y)" and be done with it.

Cannot detect it is multiline string or object

Related to: #75 (fixed by #78)

if the receiver type is interface{} and unmarshaling with UnmarshalYAML([]byte), it seems impossible to detect it is multiline string or object.

e.g. the value in https://github.com/OAI/OpenAPI-Specification/blob/master/examples/v3.0/api-with-examples.yaml#L18 and the value in https://github.com/OAI/OpenAPI-Specification/blob/master/examples/v3.0/api-with-examples.yaml#L51 have same type but the result of unmarshal is not same, they are defined as interface{} and former one should be unmarshaled to map[string]interface{}, later one should unmarshaled to string

before #78, the passed bytes contains | at first character so we could detect it is string or object

is there any way to detect? (or, should revert the change?)

validation crash on missing nested required field

Hey @goccy,

I've run the following test:

package main 

import (
	"strings"

	"github.com/goccy/go-yaml"
	"gopkg.in/go-playground/validator.v9"
)

type Person struct {
	Name string `yaml:"name" validate:"required"`
	Age  int    `yaml:"age" validate:"gte=0,lt=120"`
	Addr *PersonAddress `yaml:"addr" validate:"required,dive,required"`
}
type PersonAddress struct {
	Number string `yaml:"number" validate:"required"`
	State string `yaml:"state" validate:"required"`
}


func main() {
	yml := `
name: itai
age: 10
addr:
  number: seven
` // missing State
	validate := validator.New()
	dec := yaml.NewDecoder(
		strings.NewReader(yml),
		yaml.Validator(validate),
	)
	var v Person
	err := dec.Decode(&v)
	if err == nil {
		panic("expected error")
	}
}

got the following crash log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x10effb2]

goroutine 1 [running]:
github.com/goccy/go-yaml.(*Decoder).decodeStruct(0xc0001fe000, 0x11b2480, 0xc0000a9fc0, 0x199, 0x1210c20, 0xc000105a80, 0x28, 0x11aa560)
        /Users/itai/work/go-yaml/decode.go:855 +0x1b12
github.com/goccy/go-yaml.(*Decoder).decodeValue(0xc0001fe000, 0x11b2480, 0xc0000a9fc0, 0x199, 0x1210c20, 0xc000105a80, 0x0, 0x2a)
        /Users/itai/work/go-yaml/decode.go:473 +0x1b0a
github.com/goccy/go-yaml.(*Decoder).decode(0xc0001fe000, 0x1189e00, 0xc0000a9fc0, 0x16, 0x1189e00, 0x1337a20)
        /Users/itai/work/go-yaml/decode.go:1243 +0xd2
github.com/goccy/go-yaml.(*Decoder).Decode(0xc0001fe000, 0x1189e00, 0xc0000a9fc0, 0x1006cbf, 0xc00008e058)
        /Users/itai/work/go-yaml/decode.go:1272 +0x272
main.main()
        /Users/itati/work/go-yaml/cmd/validate/cmd.go:34 +0x1f6
exit status 2

shell returned 1

It seems that structFieldMap doesn't contain "State" fieldName, so we crash on null pointer deref (structField.RenderName)

					structField := structFieldMap[fieldName]
					node, exists := keyToNodeMap[structField.RenderName]

cannot parse correctly escape sequences in the string

sample test: https://github.com/kyoh86/go-yaml/commit/54285c4d51ad6379854f0a2a49169250f5c2a765

failed result:

--- FAIL: TestDecoder (0.00s)
    --- FAIL: TestDecoder/"1":_"\"2\"" (0.00s)
        decode_test.go:935: failed to test ["1": "\"2\""], actual=[map[1:\]], expect=[map[1:"2"]]
    --- FAIL: TestDecoder/"1":_"\"" (0.00s)
        decode_test.go:935: failed to test ["1": "\""], actual=[map[1:\]], expect=[map[1:"]]
    --- FAIL: TestDecoder/"1":_"X\"z" (0.00s)
        decode_test.go:935: failed to test ["1": "X\"z"], actual=[map[1:X\]], expect=[map[1:X\"z]]
    --- FAIL: TestDecoder/"1":_"\\\\" (0.00s)
        decode_test.go:935: failed to test ["1": "\\\\"], actual=[map[1:\\\\]], expect=[map[1:\\]]
    --- FAIL: TestDecoder/"1":_"\\"2\\"" (0.00s)
        decode_test.go:935: failed to test ["1": "\\"2\\""], actual=[map[1:\\]], expect=[map[1:\"2\"]]
    --- FAIL: TestDecoder/"1":_"a\x2Fb\u002Fc\U0000002Fd" (0.00s)
        decode_test.go:935: failed to test ["1": "a\x2Fb\u002Fc\U0000002Fd"], actual=[map[1:a\x2Fb\u002Fc\U0000002Fd]], expect=[map[1:a/b/c/d]]
    --- FAIL: TestDecoder/"1":_"\a\b\t\n\v\f\r\e\#x20\"\/\\\N\_\L\P" (0.00s)
        decode_test.go:935: failed to test ["1": "\a\b\t\n\v\f\r\e\#x20\"\/\\\N\_\L\P"], actual=[map[1:\a\b\t\n\v\f\r\e\#x20\]], expect=[map[1:
            

  "/\��

]]
FAIL
FAIL    github.com/goccy/go-yaml        0.010s

Support for default values

It appears that the library doesn't support default values. During unmarshaling, all the values not present in the yaml are reset (or set to zero value).

gopkg.in/yaml.v2 supports this behaviour, which is great.

See the following failing test:

package yaml_test

import (
	"github.com/goccy/go-yaml"
	// "gopkg.in/yaml.v2"
	"strings"
	"testing"
)

func TestDecoder_DefaultValues(t *testing.T) {
	v  := struct {
		A string  `yaml:"a"`
		B string `yaml:"b"`
		c string // private
	}{
		B: "defaultBValue",
		c: "defaultCValue",
	}

	const src = `---
a: a_value
`
	if err := yaml.NewDecoder(strings.NewReader(src)).Decode(&v); err != nil {
		t.Fatalf(`parsing should succeed: %s`, err)
	}

	if v.A != "a_value" {
		t.Fatalf("v.A should be `a_value`, got `%s`", v.A)
	}

	if v.B != "defaultBValue" {
		t.Fatalf("v.B should be `defaultValue`, got `%s`", v.B)
	}

	if v.c != "defaultCValue" {
		t.Fatalf("v.c should be `defaultCValue`, got `%s`", v.c)
	}
}

Thanks.

Respect `json` struct tags

One of the reasons I use https://github.com/ghodss/yaml is because it supports the json struct tag -- so I don't have to have multiple definitions of the same fields:

// I DO NOT want this
type Foo struct {
    A `json:"foo" yaml:"foo"`
    B `json:"bar,omitempty", yaml:"bar,omitempty"`
    ...
}
// I WANT this to work with this YAML library
type Bar struct {
    A `json:"foo"` 
    B `json:"bar"`
}

I would really like this library to (1) by default look at the YAML tag, and (2) if not found, look at the JSON tag

Cannot parse yaml file

My config file cannot be parser. It complains on a unexpected key at 16,1. Removing the empty line before the "masters:" key, makes Unmarshall parse it.

bad.yml.txt

The yamlv.2 package parses it correctly, though.

When marshalling strings they are not escaped property

Example: https://play.golang.org/p/8eLm27d9NHe

I am escaping stars and backticks by having them wrapped in single/double quotes in my yaml file.

I have yaml in a file like bellow:

keyword: a test keyword
response:
- "**This line will print the first time** Fails to add when marshalling"
- "This line will be printed alone after marshalling and unmarshalling"

If you marshal it strips the outside quotes resulting in the following:

keyword: a test keyword
response:
- **This line will print the first time** Fails to add when marshalling
- This line will be printed alone after marshalling and unmarshalling

The result after marshalling is that the first line gets lost due to being parsed as a pointer when unmarshalling again. This also affects when reading and writing from files.

Parsing JSON is whitespace-dependent

When a YAML contains a JSON object that's formatted without whitespace, for example {"name":"John"}, the parser gives the following error:

[1:2] failed to parse flow mapping value node
    >  1 | {"name":"John"}

This does not happen when fields are separated by a space, i.e. {"name": "John"}

Playground: https://goplay.space/#5xOT86HOV2L

Cleanup ast/*

I want to cleanup ast/* . For one thing I want to work towards squeezing more performance out of the library, but for that I want to centralize object value assignment and allocation. I'd like to apply this to a wider area than ast/, but I'd like to start at ast/

For now, I'd like to disallow explicit instantiation of, and assignment to structs. i.e.

// disallow 
n := ast.Foo{Start: tk, Value: blah}
n.Start = ...
n.Value = ...
// enforce encapsulation
n := ast.Foo(tk, blah)
n.Start() // Get only
n.SetValue(...) // If mutation is required

Would you consider this type of change if I submit a PR?

DisallowUnknownField crash when inline tag provided

Hey @goccy,
I've added disallowUnknownField opt to TestDecoder_Inline.

func TestDecoder_Inline(t *testing.T) {
	type Base struct {
		A int
		B string
	}
	yml := `---
a: 1
b: hello
c: true
`
	var v struct {
		*Base `yaml:",inline"`
		C     bool
	}
	if err := yaml.NewDecoder(strings.NewReader(yml), yaml.DisallowUnknownField()).Decode(&v); err != nil {
		t.Fatalf("%+v", err)
	}
	if v.A != 1 {
		t.Fatal("failed to decode with inline key")
	}
	if v.B != "hello" {
		t.Fatal("failed to decode with inline key")
	}
	if !v.C {
		t.Fatal("failed to decode with inline key")
	}
}

The test yielded the following stack trace:

--- FAIL: TestDecoder_Inline (0.00s)
panic: reflect: NumField of non-struct type *yaml_test.Base [recovered]
        panic: reflect: NumField of non-struct type *yaml_test.Base

goroutine 27 [running]:
testing.tRunner.func1.1(0x11fb580, 0xc000332430)
        /usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:941 +0x3d0
testing.tRunner.func1(0xc0003306c0)
        /usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:944 +0x3f9
panic(0x11fb580, 0xc000332430)
        /usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:967 +0x15d
reflect.(*rtype).NumField(0x11ef5c0, 0x1208240)
        /usr/local/Cellar/go/1.14/libexec/src/reflect/type.go:973 +0xb7
github.com/goccy/go-yaml.structFieldMap(0x12aff80, 0x11ef5c0, 0x196, 0x12aff80, 0x11ef5c0)
        /Users/user/work/go-yaml/struct.go:117 +0xcb
github.com/goccy/go-yaml.(*Decoder).deleteStructKeys(0xc0003034a0, 0x11ef5c0, 0xc000332280, 0x196, 0xc000334450, 0x121d060, 0xc00032c8a0)
        /Users/user/work/go-yaml/decode.go:366 +0x82
github.com/goccy/go-yaml.(*Decoder).decodeStruct(0xc0003034a0, 0x1218840, 0xc000332280, 0x199, 0x12adb00, 0xc000320d00, 0x28, 0x121cca0)
        /Users/user/work/go-yaml/decode.go:778 +0xde7
github.com/goccy/go-yaml.(*Decoder).decodeValue(0xc0003034a0, 0x1218840, 0xc000332280, 0x199, 0x12adb00, 0xc000320d00, 0x0, 0x1a)
        /Users/user/work/go-yaml/decode.go:471 +0x1b0a
github.com/goccy/go-yaml.(*Decoder).decode(0xc0003034a0, 0x11f4780, 0xc000332280, 0x16, 0x11f4780, 0x1414800)
        /Users/user/work/go-yaml/decode.go:1234 +0xd2
github.com/goccy/go-yaml.(*Decoder).Decode(0xc0003034a0, 0x11f4780, 0xc000332280, 0x10565f9, 0x730071671f62b)
        /Users/user/work/go-yaml/decode.go:1263 +0x272
github.com/goccy/go-yaml_test.TestDecoder_Inline(0xc0003306c0)
        /Users/user/work/go-yaml/decode_test.go:1314 +0x1da
testing.tRunner(0xc0003306c0, 0x125ebf0)
        /usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:992 +0xdc
created by testing.(*T).Run
        /usr/local/Cellar/go/1.14/libexec/src/testing/testing.go:1043 +0x357
FAIL    github.com/goccy/go-yaml        0.812s
FAIL

Comments split files into documents

When using parse.ParseFile(), I find that my single-document yaml file will get split into multiple documents every time I hit a comment; is this intentional?

example YAML to reproduce this:

key1: data
key2: data
# Comment
key3: data

Another question I have is regarding the comment parsing; the only ast.Mode available is ParseComments (which is fine), however... calling .String() on *ast.File, or *ast.Document, does not display comments.

Single quoted string output is not consistent with input

Thanks for the amazing library.

Description

If the input yaml file contains single-quoted string, they are converted to double-quoted string after marshalling. Is this expected behaviour?

https://play.golang.org/p/JLbpVImLy2K

struct {
	A string `yaml:"username"`
	B string `yaml:"password"`
}

Input for unmarshal

username: '{{ requiredEnv "USERNAME" }}'
password: '{{ requiredEnv "PASSWORD" }}'

Output of marshal

username: "{{ requiredEnv \"USERNAME\" }}"
password: "{{ requiredEnv \"PASSWORD\" }}"

feature request | support multiple validation errors.

Hey @goccy,
validator support multiple validation errors.
When using Validator DecodeOption, the current implementation gets the first error that appears in the YAML and wraps it as ErrSyntax.
I would like to have a way to access all validation errors.
Perhaps we should introduce a new type: errors.ErrSyntaxList?

Allow new-line and indented map as value of slice

I want to decode new-line and indented map as value of slice.

like follows

a:
-
  b: d
  c: e
-
  b: f
  c: g
go-yaml/yaml goccy/go-yaml v1.4.1
decode can decode error
play.golang.org here here

I have tried this problem but have not solved it yet ... so I first register this issue.

[Question] Wrong indentation?

Given the following:

func main() {
	mystruct := Struct{
		Inner{
			map[string]string{"Woo": "Hoo"},
		},
	}
	yaml, err := yaml.Marshal(mystruct)

	if err != nil {
		fmt.Printf("Oh no! %v", err)
	}

	fmt.Printf("We have:\n%s\n", yaml)
}

type Struct struct {
	Inner Inner
}

type Inner struct {
	Map map[string]string
}

I'd expect the output to look like:

We have:
inner:
  map:
    Woo: Hoo

but instead I get:

We have:
inner:
  map:
  Woo: Hoo

Where attributes of map are not indented. Am I doing something wrong? If I use https://github.com/go-yaml/yaml the output is as expected

cannot parse correctly single-quoted strings

https://yaml.org/spec/1.2/spec.html#id2788097

The single-quoted style is specified by surrounding “'” indicators. Therefore, within a single-quoted scalar, such characters need to be repeated. This is the only form of escaping performed in single-quoted scalars. In particular, the “\” and “"” characters may be freely used. This restricts single-quoted scalars to printable characters. In addition, it is only possible to break a long single-quoted line where a space character is surrounded by non-spaces.

sample test: https://github.com/kyoh86/go-yaml/commit/ae724c11639ee791cd6fede2f8419ef6ea858d51?w=1

error on parsing block string

environment: go 1.13.4

(日本語でいいでしょうか・・・)

次の様な複数行文字列をパースしている際に文字列中でエラーが発生しました。

foo:
  value: |
     {
      "versions": [
            {
              "status": "CURRENT",
              "updated": "2011-01-21T11:33:21Z",
              "id": "v2.0",
              "links": [
                  {
                      "href": "http://127.0.0.1:8774/v2/",
                      "rel": "self"
                  }
              ]
          },
          {
              "status": "EXPERIMENTAL",
              "updated": "2013-07-23T11:33:21Z",
              "id": "v3.0",
              "links": [
                  {
                      "href": "http://127.0.0.1:8774/v3/",
                      "rel": "self"
                  }
              ]
          }
      ]
     }

.foo.value以下はすべて文字列となる、という想定で、gopkg.in/yaml.v2では問題なく動作していたものです。
エラーメッセージは次の様なものです。

--- FAIL: TestAPIWithExample (0.00s)
    unmarshalyaml_test.go:25: [66:51] failed to parse flow mapping value node
              62 | }
              63 |                             ]
              64 |                         },
              65 |                         {
            > 66 |                             "status": "EXPERIMENTAL",
              67 |                             "updated": "2013-07-23T11:33:21Z",
                                                                    ^
              68 |                             "id": "v3.0",
              69 |                             "links": [
              70 |                                 {
FAIL

当該部分は文字列なので、(インデントが足りないとかを除いて)途中でエラーが発生するというのは挙動として正しくないように思えます。

再現用Playground: https://play.golang.org/p/yLh0AZ5lQ62

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.