wi2l / jettison Goto Github PK
View Code? Open in Web Editor NEWHighly configurable, fast JSON encoder for Go
Home Page: https://pkg.go.dev/github.com/wI2L/jettison
License: MIT License
Highly configurable, fast JSON encoder for Go
Home Page: https://pkg.go.dev/github.com/wI2L/jettison
License: MIT License
I'm getting this amazing error on go version go1.18.1 linux/amd64
:
runtime: nameOff 0x1276120 out of range 0x1029000 - 0x17d770e
fatal error: runtime: name offset out of range
goroutine 57 [running]:
runtime.throw({0x1317f5a?, 0xc0001aec20?})
/usr/lib/go/src/runtime/panic.go:992 +0x71 fp=0xc00060ca70 sp=0xc00060ca40 pc=0x435751
runtime.resolveNameOff(0x11c5e60?, 0x1276120)
/usr/lib/go/src/runtime/type.go:198 +0x265 fp=0xc00060cac8 sp=0xc00060ca70 pc=0x45ba25
reflect.resolveNameOff(0x12ccb80?, 0x179e620?)
/usr/lib/go/src/runtime/runtime1.go:498 +0x19 fp=0xc00060cae8 sp=0xc00060cac8 pc=0x460ed9
reflect.(*rtype).nameOff(...)
/usr/lib/go/src/reflect/type.go:729
reflect.(*rtype).String(0x179e620)
/usr/lib/go/src/reflect/type.go:799 +0x25 fp=0xc00060cb08 sp=0xc00060cae8 pc=0x4af745
reflect.(*rtype).ptrTo(0x179e620)
/usr/lib/go/src/reflect/type.go:1456 +0x65 fp=0xc00060cb98 sp=0xc00060cb08 pc=0x4b29e5
reflect.PointerTo(...)
/usr/lib/go/src/reflect/type.go:1442
reflect.PtrTo(...)
/usr/lib/go/src/reflect/type.go:1437
github.com/wI2L/jettison.newMarshalerTypeInstr({0x17b0fe0, 0x179e620}, 0x0)
/home/fiatjaf/comp/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:139 +0x58 fp=0xc00060cbd0 sp=0xc00060cb98 pc=0x9a2638
github.com/wI2L/jettison.newInstruction({0x17b0fe0?, 0x179e620}, 0x20?, 0xe6?)
/home/fiatjaf/comp/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:94 +0x52 fp=0xc00060cc08 sp=0xc00060cbd0 pc=0x9a21f2
github.com/wI2L/jettison.cachedInstr({0x17b0fe0?, 0x179e620})
/home/fiatjaf/comp/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:45 +0x85 fp=0xc00060cc40 sp=0xc00060cc08 pc=0x9a1f85
github.com/wI2L/jettison.encodeInterface(0xc000022d00?, {0xc00037d000, 0xf, 0x1000}, {{0x17a6490, 0xc000040068}, {0x131a373, 0x23}, 0x5, 0xb1, ...})
/home/fiatjaf/comp/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:70 +0x87 fp=0xc00060ccd0 sp=0xc00060cc40 pc=0x99bec7
github.com/wI2L/jettison.encodeStruct(0xc0001e50e0, {0xc00037d000?, 0x99b885?, 0x2021b00?}, {{0x17a6490, 0xc000040068}, {0x131a373, 0x23}, 0x5, 0xb1, ...}, ...)
/home/fiatjaf/comp/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:244 +0x4f0 fp=0xc00060cdf0 sp=0xc00060ccd0 pc=0x99d1b0
github.com/wI2L/jettison.newStructFieldsInstr.func2(0x1346e501a90b46?, {0xc00037d000?, 0x7f2645981538?, 0x40?}, {{0x17a6490, 0xc000040068}, {0x131a373, 0x23}, 0x5, 0xb1, ...})
/home/fiatjaf/comp/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:319 +0x65 fp=0xc00060ce78 sp=0xc00060cdf0 pc=0x9a3a25
github.com/wI2L/jettison.marshalJSON({0x12a17a0?, 0xc0001e50e0?}, {{0x17a6490, 0xc000040068}, {0x131a373, 0x23}, 0x5, 0xb1, 0x0, 0x0})
/home/fiatjaf/comp/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:167 +0xd9 fp=0xc00060cf40 sp=0xc00060ce78 pc=0x9a5499
github.com/wI2L/jettison.MarshalOpts({0x12a17a0, 0xc0001e50e0}, {0xc00060d028, 0x4, 0xc0003cbc00?})
/home/fiatjaf/comp/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:142 +0x1a9 fp=0xc00060d000 sp=0xc00060cf40 pc=0x9a5289
github.com/lnbits/infinity/utils.JSONMarshal({0x12a17a0?, 0xc0001e50e0?})
/home/fiatjaf/comp/infinity/utils/json.go:24 +0x6c fp=0xc00060d058 sp=0xc00060d000 pc=0x9ad2cc
github.com/lnbits/infinity/api/apiutils.SendJSON({0x17a57a8, 0xc00002e2a0}, {0x12a17a0?, 0xc0001e50e0?})
/home/fiatjaf/comp/infinity/api/apiutils/jsonresponse.go:10 +0x33 fp=0xc00060d088 sp=0xc00060d058 pc=0xb7c733
github.com/lnbits/infinity/api.LnurlScan({0x17a57a8, 0xc00002e2a0}, 0xc0005ecb00)
/home/fiatjaf/comp/infinity/api/wallet.go:347 +0x786 fp=0xc00060d5c8 sp=0xc00060d088 pc=0xf14ba6
net/http.HandlerFunc.ServeHTTP(0xc0003a28f0?, {0x17a57a8?, 0xc00002e2a0?}, 0xc0005ecb00?)
/usr/lib/go/src/net/http/server.go:2084 +0x2f fp=0xc00060d5f0 sp=0xc00060d5c8 pc=0x6d8e6f
For my use case, I only want to apply the AllowList
to the top level of my object. I'm thinking that rather than having an AllowList
, there could be an Allower interface:
type Allower interface {
Allow(jsonKey string) bool
Recurse(jsonKey string) Allower
}
Recurse
could return nil
to indicate that there is no filtering with the key.
If this idea is acceptable to you, I might try my hand at building it sometime in the next year. While I would use it immediately if it were available, I can work around not having it for a while.
The current AllowList
and DenyList
could be re-implemented as things that generate an Allower
.
When trying to marshal a structure that has a field declared as an interface, an error gets returned.
Tested using Go 1.21, MacOS 14.0, M1 Pro.
The following test demonstrates it:
package lib
import (
"testing"
"github.com/wI2L/jettison"
)
type Circle struct {
Radius int `json:"radius"`
}
func (c Circle) Area() int {
return c.Radius * c.Radius
}
type Shape interface {
Area() int
}
type Foo struct {
S Shape `json:"s"`
}
func TestJettisonMarshal(t *testing.T) {
foo := Foo{S: Circle{Radius: 1}}
_, err := jettison.Marshal(foo)
if err != nil {
t.Fatal(err)
}
}
returns:
=== RUN TestJettisonMarshal
runtime: nameOff 0x2af780 out of range 0x10029c000 - 0x1002d5a08
fatal error: runtime: name offset out of range
goroutine 6 [running]:
runtime.throw({0x1002704f4?, 0x14000003ba0?})
/usr/local/go/src/runtime/panic.go:1077 +0x40 fp=0x14000125570 sp=0x14000125540 pc=0x1001994c0
runtime.resolveNameOff(0x14000125608?, 0x2af780)
/usr/local/go/src/runtime/type.go:119 +0x214 fp=0x140001255d0 sp=0x14000125570 pc=0x1001c2f04
reflect.resolveNameOff(0x1400000e101?, 0x2d3f48?)
/usr/local/go/src/runtime/runtime1.go:604 +0x1c fp=0x140001255f0 sp=0x140001255d0 pc=0x1001c8b1c
reflect.(*rtype).nameOff(...)
/usr/local/go/src/reflect/type.go:526
reflect.(*rtype).String(0x1002d3f48)
/usr/local/go/src/reflect/type.go:542 +0x24 fp=0x14000125610 sp=0x140001255f0 pc=0x1001fa964
fmt.(*pp).handleMethods(0x1400007ad00, 0x418108?)
/usr/local/go/src/fmt/print.go:673 +0x21c fp=0x14000125870 sp=0x14000125610 pc=0x100206b9c
fmt.(*pp).printArg(0x1400007ad00, {0x1002ce3c0?, 0x1002d3f48}, 0x73)
/usr/local/go/src/fmt/print.go:756 +0x5fc fp=0x14000125910 sp=0x14000125870 pc=0x1002075cc
fmt.(*pp).doPrintf(0x1400007ad00, {0x10026e5ea, 0x1a}, {0x14000125ab8?, 0x1, 0x1})
/usr/local/go/src/fmt/print.go:1077 +0x2dc fp=0x14000125a20 sp=0x14000125910 pc=0x100209dcc
fmt.Sprintf({0x10026e5ea, 0x1a}, {0x14000125ab8, 0x1, 0x1})
/usr/local/go/src/fmt/print.go:239 +0x4c fp=0x14000125a80 sp=0x14000125a20 pc=0x10020462c
github.com/wI2L/jettison.(*UnsupportedTypeError).Error(0x1002d5701?)
/Users/ziemekobel/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:67 +0x50 fp=0x14000125ad0 sp=0x14000125a80 pc=0x100264060
fmt.(*pp).handleMethods(0x1400007a9c0, 0x0?)
/usr/local/go/src/fmt/print.go:667 +0x158 fp=0x14000125d30 sp=0x14000125ad0 pc=0x100206ad8
fmt.(*pp).printArg(0x1400007a9c0, {0x1002b03e0?, 0x140000242b0}, 0x76)
/usr/local/go/src/fmt/print.go:756 +0x5fc fp=0x14000125dd0 sp=0x14000125d30 pc=0x1002075cc
fmt.(*pp).doPrintln(0x1400007a9c0, {0x14000125f48?, 0x1, 0x14000024220?})
/usr/local/go/src/fmt/print.go:1223 +0x3c fp=0x14000125e50 sp=0x14000125dd0 pc=0x10020acfc
fmt.Sprintln({0x14000125f48, 0x1, 0x1})
/usr/local/go/src/fmt/print.go:321 +0x3c fp=0x14000125ea0 sp=0x14000125e50 pc=0x10020489c
testing.(*common).Fatal(0x14000003a00, {0x14000125f48?, 0x14000073f48?, 0x1001de19c?})
/usr/local/go/src/testing/testing.go:1075 +0x3c fp=0x14000125f00 sp=0x14000125ea0 pc=0x10021f75c
ef-studio/catalyst/tests/lib.TestJettisonMarshal(0x14000003a00)
/Users/ziemekobel/ws/ef-studio/backend/go/catalyst/tests/lib/jettison_test.go:30 +0x94 fp=0x14000125f60 sp=0x14000125f00 pc=0x100268bd4
Marshalling panics when a struct containing a map field and a custom MarshalJSON() is provided.
Tested on Apple M1 Pro, running macOS 14.0.
The below test code:
type Fields struct {
AdditionalProperties map[string]string `json:"-"`
}
func (a Fields) MarshalJSON() ([]byte, error) {
var err error
object := make(map[string]json.RawMessage)
for fieldName, field := range a.AdditionalProperties {
object[fieldName], err = json.Marshal(field)
if err != nil {
return nil, fmt.Errorf("error marshaling '%s': %w", fieldName, err)
}
}
return json.Marshal(object)
}
func TestJettison(t *testing.T) {
v := Fields{
AdditionalProperties: map[string]string{
"foo": "bar",
},
}
_, err := jettison.Marshal(v)
if err != nil {
t.Fatal(err)
}
}
results in:
unexpected fault address 0x6bff37c970
fatal error: fault
[signal SIGBUS: bus error code=0x1 addr=0x6bff37c970 pc=0x100fa32d8]
goroutine 6 [running]:
runtime.throw({0x101105eaf?, 0x100fa07b4?})
/usr/local/go/src/runtime/panic.go:1077 +0x40 fp=0x1400005d910 sp=0x1400005d8e0 pc=0x100fcabd0
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:858 +0x178 fp=0x1400005d970 sp=0x1400005d910 pc=0x100fe26c8
runtime.evacuated(...)
/usr/local/go/src/runtime/map.go:205
runtime.mapiternext(0x1400005dab8)
/usr/local/go/src/runtime/map.go:897 +0xe8 fp=0x1400005d9f0 sp=0x1400005d980 pc=0x100fa32d8
runtime.mapiterinit(0xb?, 0x0?, 0x10117aa40?)
/usr/local/go/src/runtime/map.go:864 +0x2a0 fp=0x1400005da20 sp=0x1400005d9f0 pc=0x100fa31b0
ef-studio/catalyst/tests/lib.Fields.MarshalJSON({0x1400005db40?})
/Users/ziemekobel/ws/ef-studio/backend/go/catalyst/tests/lib/jettison_test.go:21 +0x58 fp=0x1400005db20 sp=0x1400005da20 pc=0x101105078
github.com/wI2L/jettison.encodeJSONMarshaler({0x10118e100?, 0x1400005dd98}, {0x1400016a000, 0x0, 0x1000}, {{0x1011b98c0, 0x10131db00}, {0x10110eada, 0x23}, 0x5, ...}, ...)
/Users/ziemekobel/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:692 +0x7c fp=0x1400005dbc0 sp=0x1400005db20 pc=0x1010fb9ec
github.com/wI2L/jettison.encodeMarshaler(0x1400005dd98, {0x1400016a000, 0x0, 0x1000}, {{0x1011b98c0, 0x10131db00}, {0x10110eada, 0x23}, 0x5, 0x0, ...}, ...)
/Users/ziemekobel/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:668 +0x2b4 fp=0x1400005dc80 sp=0x1400005dbc0 pc=0x1010fb524
github.com/wI2L/jettison.newMarshalerTypeInstr.newJSONMarshalerInstr.func5(0x1400000e1f8?, {0x1400016a000?, 0x1400005dd58?, 0x1010f7134?}, {{0x1011b98c0, 0x10131db00}, {0x10110eada, 0x23}, 0x5, 0x0, ...})
/Users/ziemekobel/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:241 +0x68 fp=0x1400005dd10 sp=0x1400005dc80 pc=0x1010fdec8
github.com/wI2L/jettison.cachedInstr.wrapInlineInstr.func1(0x14000108ea0, {0x1400016a003, 0x0, 0x1000}, {{0x1011b98c0, 0x10131db00}, {0x10110eada, 0x23}, 0x5, 0x0, ...})
/Users/ziemekobel/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:406 +0x94 fp=0x1400005dd90 sp=0x1400005dd10 pc=0x1010fd094
github.com/wI2L/jettison.marshalJSON({0x10118e100, 0x14000108ea0?}, {{0x1011b98c0, 0x10131db00}, {0x10110eada, 0x23}, 0x5, 0x0, 0x0, 0x0})
/Users/ziemekobel/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:167 +0xe0 fp=0x1400005de70 sp=0x1400005dd90 pc=0x1011007b0
github.com/wI2L/jettison.Marshal({0x10118e100?, 0x14000108ea0?})
/Users/ziemekobel/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:115 +0x84 fp=0x1400005df10 sp=0x1400005de70 pc=0x101100674
ef-studio/catalyst/tests/lib.TestJettison(0x140001524e0)
/Users/ziemekobel/ws/ef-studio/backend/go/catalyst/tests/lib/jettison_test.go:36 +0x7c fp=0x1400005df60 sp=0x1400005df10 pc=0x10110525c
From the docs it seams that field selection through an AllowList, should only apply to first-level elements:
AllowList sets the list of first-level fields which are to be considered when encoding a struct. The fields are identified by the name that is used in the final JSON payload. See DenyFields documentation for more information regarding joint use with this option.
However, it looks like they are applied to nested struct elements as well:
Hey,
I've tried to replace encoding/json
with jettison, since jettison has the omitnil
tag.
There are certain fields that I am using a custom type for. The idea behind the type is to allow a HTTP request to set a field to null
via a PATCH
request, but not automatically null
all omitted fields. For that to work, null fields must not be see as empty when unmarshalling. However, when marshalling, these null-values have no worth and should be omitted. Is something like that possible?
Let's say you had the following type:
type Thing {
A *NullableString
B *NullableString
}
If you now had a resource where both A
and B
already had a value of Hello
and you'd send the following request:
{
"A": null,
Then field A
should be nulled, while B
will stay unchanged, since A was explicitly defined, but B was not.
The easy solution would be to make two versions of all structs. One for requests and one for replies, however I think that isn't desirable, as it bloats the code and doesn't allow sharing code to work on these structs.
Here's a small example. The second case will panic with json: error calling MarshalJSON for type *flows_service.NullableString: json: invalid value
.
package flows_service
import (
"encoding/json"
"fmt"
"testing"
"github.com/wI2L/jettison"
)
type NullableString struct {
// Set indicates whether unmarshalled JSON contained the field.
Set bool
// This field is only relevant if Set is `true`
NonNull bool
// Val; Only relevant if NonNull and Set are `true`.
Val string
}
// MarshalJSON implements json.Marshaler.
func (v *NullableString) MarshalJSON() ([]byte, error) {
if !v.Set || !v.NonNull {
return nil, nil
}
return json.Marshal(v.Val)
}
// UnmarshalJSON implements json.Unmarshaler.
func (v *NullableString) UnmarshalJSON(data []byte) error {
// If this method was called, the value was set.
v.Set = true
if string(data) == "null" {
return nil
}
if err := json.Unmarshal(data, &v.Val); err != nil {
return err
}
v.NonNull = true
return nil
}
type Something struct {
Field *NullableString `json:"field,omitempty,omitnil"`
}
func TestMarshalling(t *testing.T) {
cases := []*Something{
{},
{
Field: &NullableString{},
},
{
Field: &NullableString{
Set: true,
},
},
}
for i, c := range cases {
t.Run(fmt.Sprintf("%d", i), func(t *testing.T) {
data, err := jettison.Marshal(c)
if err != nil {
panic(err)
}
t.Log(string(data))
})
}
}
Hi, I just updated to Go 1.18 and jettison (v0.7.3) started failing when marshaling maps. This code:
package main
import "github.com/wI2L/jettison"
func main() {
jettison.Marshal(map[string]interface{}{
"test": 123,
})
}
will fail using Go 1.18 with this stacktrace:
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100 pc=0x460e05]
goroutine 1 [running]:
runtime.throw({0x4ca188?, 0xc00011c078?})
/home/flusflas/.gvm/gos/go1.18/src/runtime/panic.go:992 +0x71 fp=0xc00008da38 sp=0xc00008da08 pc=0x430bd1
runtime.sigpanic()
/home/flusflas/.gvm/gos/go1.18/src/runtime/signal_unix.go:802 +0x3a9 fp=0xc00008da88 sp=0xc00008da38 pc=0x444789
sync.(*Pool).Get(0x56b100)
/home/flusflas/.gvm/gos/go1.18/src/sync/pool.go:129 +0x25 fp=0xc00008dac0 sp=0xc00008da88 pc=0x460e05
github.com/wI2L/jettison.encodeSortedMap(0x56b0c0, {0xc000138000, 0x1, 0x1000}, {{0x4e5730, 0xc00012c028}, {0x4c8fa0, 0x23}, 0x5, 0x0, ...}, ...)
/home/flusflas/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:415 +0x7a fp=0xc00008dc00 sp=0xc00008dac0 pc=0x49a13a
github.com/wI2L/jettison.encodeMap(0x2?, {0xc000138000, 0x0, 0x1000}, {{0x4e5730, 0xc00012c028}, {0x4c8fa0, 0x23}, 0x5, 0x0, ...}, ...)
/home/flusflas/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:364 +0x337 fp=0xc00008dce0 sp=0xc00008dc00 pc=0x499cd7
github.com/wI2L/jettison.newMapInstr.func1(0x56b000?, {0xc000138000?, 0xc00010c2b0?, 0xc000138000?}, {{0x4e5730, 0xc00012c028}, {0x4c8fa0, 0x23}, 0x5, 0x0, ...})
/home/flusflas/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:400 +0x72 fp=0xc00008dd70 sp=0xc00008dce0 pc=0x49ff52
github.com/wI2L/jettison.wrapInlineInstr.func1(0xc00010e150, {0xc000138000?, 0x203000?, 0x0?}, {{0x4e5730, 0xc00012c028}, {0x4c8fa0, 0x23}, 0x5, 0x0, ...})
/home/flusflas/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:406 +0x65 fp=0xc00008dde0 sp=0xc00008dd70 pc=0x4a0165
github.com/wI2L/jettison.marshalJSON({0x4b2a80?, 0xc00010e150?}, {{0x4e5730, 0xc00012c028}, {0x4c8fa0, 0x23}, 0x5, 0x0, 0x0, 0x0})
/home/flusflas/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:167 +0xd9 fp=0xc00008dea8 sp=0xc00008dde0 pc=0x4a0eb9
github.com/wI2L/jettison.Marshal({0x4b2a80?, 0xc00010e150?})
/home/flusflas/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:115 +0xbf fp=0xc00008df48 sp=0xc00008dea8 pc=0x4a0d7f
main.main()
/home/flusflas/main/main.go:6 +0x7a fp=0xc00008df80 sp=0xc00008df48 pc=0x4a601a
I tried to look for any reported bug with pools in Go 1.18, but I didn't find anything, and I feel a little lost here.
Jsoniter is a drop in replacement for JSON, I thought you could post a benchmark there if you like to showcase more efficient JSON.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.