kelindar / binary Goto Github PK

View Code? Open in Web Editor NEW

201.0 6.0 24.0 111 KB

Generic and fast binary serializer for Go

Home Page: https://emitter.io

License: MIT License

Go 100.00%

encoding golang codec decoding generic binary serialization

binary's Introduction

Hey 👋, I'm Roman!

I love building stuff, currently working as a Principal Engineer in Careem on experimentation, optimization and machine learning platforms. Prior to Careem, I worked as Head of Data Science in AirAsia and as a Principal Engineer in Grab, the super app of South East Asia where we built Product Insights & Experimentation Platforms amongst other things. I also obtained a PhD in Computer Science and Human-Computer Interaction with Trinity College Dublin & IBM Research, and worked as an engineer at various successful companies in Europe, building things like online gaming platforms, autonomous helicopters, or particle/matter collision simulators!

🚀 Distributed Systems I have designed and open-sourced

emitter-io/emitter - high performance, distributed and low latency publish-subscribe platform
kelindar/talaria - distributed, highly available, and low latency time-series database for Presto

📦 Golang Libraries I made to help me in building software faster or explore a certain idea

kelindar/async - concurrency patterns library using promises for Go
kelindar/approx - probabilistic data structures (e.g. morris counting, count-min sketch, topk) in Go
kelindar/binary - generic and fast binary serializer for Go
kelindar/bitmap - zero-allocation dense bitmap index for Go
kelindar/column - columnar in-memory db (structures-of-arrays) with querying and indexing in Go
kelindar/goap - goal oriented action planning library in Go
kelindar/intmap - fast uint32-to-uint32 hash map, about 20-50% faster than Go map
kelindar/tile - 2D grid engine, built with data and cache friendly ways, includes pathfinding and observers
kelindar/timeline - Scheduler of events for near real-time systems in Go
kelindar/evolve - Go implementation of a binary genetic algorithm with random binary crossover & mutation
kelindar/loader - Go library that allows for downloading things from a URL (http, s3, ...)
kelindar/ecs - high-performance entity component system in pure Go
kelindar/event - simple internal event bus for Go applications
kelindar/lua - wrapper around LUA script executor for long-running scripts
kelindar/tcp - simple TCP server in Go
kelindar/process - Go library for process usage retrieval
kelindar/simd - auto-vectorized SIMD math functions for Go
kelindar/smutex - simple sharded mutual exclusion in Go
kelindar/xxrand - XXH3-Based Pseudorandom Number Generator
kelindar/iostream - Simple binary reader and writer for io.Reader/io.Writer streams
kelindar/simplex - Simplex noise in Go
emitter-io/address - Go library providing a set of utilities for dealing with network addressing
emitter-io/stats - high-performance library for stats and metric aggregation
emitter-io/config - library for hierarchical config management

🧪 Experiments in which I tried with various ideas

kelindar/gocc - tool to transpile C to Go assembly using clang & objdump
kelindar/hashbench - benchmarking various hash functions in Go
kelindar/etop - command-line utility for emitter.io cluster status
kelindar/spiral - procedurally generating a spiral galaxy in Go
kelindar/markov - experimenting with markov-chain procgen in Go
kelindar/pmml2lua - trying to convert PMML decision trees into LUA code
kelindar/spike-box - what if we can data-bind js/DOM transparently over the network?
kelindar/misakai-baker - static site generator for C# / .Net people
manycore/dashboard - profiling your parallelized app by categorizing issues and visually browsing them
manycore/matmul - matrix multiplication benchmark suite for parallel performance measurement
manycore/harvester - operating system and hardware performance counter collector utility

🎨 Emitter Demos I have prepared for the project

chat - building a chat with emitter
actor - distributed actor model with emitter
client-server - how to create a client/server application with emitter
platformer - making an online platformer with emitter
retain - how to use message retention in emitter
share - how to use shared subscriptions in emitter
iss - tracking international space station in real-time
presence - demo of the channel presence for emitter

📚 Blogs & Papers I have written in the past

Technical Blog - My random blog posts around experimentation, performance and open source
Ph.D Thesis - Supporting visual diagnosis of performance problems in multi-core and parallel software
SIGCHI'14 Paper - Design considerations for parallel performance tools
IEEE Journal Paper - Parallel Performance Problems on Shared-Memory Multicore Systems: Taxonomy and Observation

binary's People

Contributors

Stargazers

Watchers

binary's Issues

Self-referencing data structure causes stack overflow by endless recursion

Consider the following program:


import (
        "fmt"

        "github.com/kelindar/binary"
)

type Thing struct {
        N int
        P *Thing
}

func main() {
        x := Thing{N: 1}
        bin, _ := binary.Marshal(x)
        fmt.Printf("asdf %d\n", len(bin))
}

Running this causes a fatal error:

runtime: sp=0xc020300350 stack=[0xc020300000, 0xc040300000]
fatal error: stack overflow

runtime stack:
runtime.throw(0x50aa19, 0xe)
	/usr/lib/golang/src/runtime/panic.go:1117 +0x72
runtime.newstack()
	/usr/lib/golang/src/runtime/stack.go:1069 +0x7ed
runtime.morestack()
	/usr/lib/golang/src/runtime/asm_amd64.s:458 +0x8f

goroutine 1 [running]:
reflect.resolveTypeOff(0x4e84e0, 0x8840, 0x40ca1b)
	/usr/lib/golang/src/runtime/runtime1.go:503 +0x4c fp=0xc020300360 sp=0xc020300358 pc=0x462e4c
reflect.(*rtype).typeOff(...)
	/usr/lib/golang/src/reflect/type.go:690
reflect.(*rtype).ptrTo(0x4e84e0, 0x4db001)
	/usr/lib/golang/src/reflect/type.go:1384 +0x36c fp=0xc020300408 sp=0xc020300360 pc=0x48110c
reflect.PtrTo(...)
	/usr/lib/golang/src/reflect/type.go:1379
github.com/kelindar/binary.scanCustomCodec(0x531f60, 0x4e84e0, 0xc00775c0f0, 0x0, 0x0)
	/home/user/go/pkg/mod/github.com/kelindar/[email protected]/scanner.go:247 +0x5b fp=0xc020300530 sp=0xc020300408 pc=0x4da6fb
github.com/kelindar/binary.scanType(0x531f60, 0x4e84e0, 0x4db00f, 0x1, 0x0, 0x0)
	/home/user/go/pkg/mod/github.com/kelindar/[email protected]/scanner.go:56 +0x50 fp=0xc020300680 sp=0xc020300530 pc=0x4d8dd0
github.com/kelindar/binary.scanType(0x531f60, 0x4f5120, 0x4f5120, 0x0, 0x0, 0x4f5180)
	/home/user/go/pkg/mod/github.com/kelindar/[email protected]/scanner.go:137 +0x991 fp=0xc0203007d0 sp=0xc020300680 pc=0x4d9711
github.com/kelindar/binary.scanType(0x531f60, 0x4e3d80, 0x4db013, 0x1, 0x0, 0x0)
	/home/user/go/pkg/mod/github.com/kelindar/[email protected]/scanner.go:66 +0x43c fp=0xc020300920 sp=0xc0203007d0 pc=0x4d91bc
github.com/kelindar/binary.scanType(0x531f60, 0x4f5120, 0x4f5120, 0x0, 0x0, 0x4f5180)

(with many more similar lines).

Apparently the *Thing pointer in the struct causes the problem. This means that in its current state this library cannot be used to serialize e.g. linked lists.

cannot use l (variable of type int) as uintptr value in assignment

Hello!
I'm trying to compile encoding+deconidng code to wasi targert with tinygo 0.30.0 and have the following errors:

$ make
tinygo build -o main.wasm -target ./target.json -opt=2 -gc=precise .
# github.com/kelindar/binary
../../../../go/pkg/mod/github.com/kelindar/[email protected]/convert.go:24:19: cannot use l (variable of type int) as uintptr value in assignment
../../../../go/pkg/mod/github.com/kelindar/[email protected]/convert.go:25:19: cannot use l (variable of type int) as uintptr value in assignment
make: *** [Makefile:17: main.wasm] Error 1

My target.json:

{
    "inherits": [ "wasi" ],
    "ldflags": [
        "--initial-memory=2097152",
        "--max-memory=67108864",
        "-zstack-size=65536"
    ]
}

It looks like an easy problem, let's fix it :-)

Panics

Hello, thanks for the great library.

We've encountered some issues with panicing pointers.

func main() {
	t := time.Now()
	data := []*A{{&t}}
	b, _ := binary.Marshal(data)
	var i []*A
	binary.Unmarshal(b, &i)

	data2 := []*B{{t}}
	b2, _ := binary.Marshal(data2)
	var j []*B
	binary.Unmarshal(b2, &j)
	log.Println(j)

	data3 := []*B{}
	b3, _ := binary.Marshal(data3)
	var h []*B
	binary.Unmarshal(b3, &h)
	log.Println(h)
}

type A struct {
	T *time.Time
}

type B struct {
	T time.Time
}

They all break in the same part:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x109bfea]

goroutine 1 [running]:
time.(*Time).UnmarshalBinary(0x2, {0xc0000b2130, 0x1, 0x1})
	/usr/local/go/src/time/time.go:1234 +0x8a
reflect.Value.call({0x10e9dc0, 0xc0000ac0b0, 0x12285b8}, {0x10ec1f5, 0x4}, {0xc0000d3c98, 0x1, 0xc0000d3c10})
	/usr/local/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x10e9dc0, 0xc0000ac0b0, 0xf}, {0xc0000d3c98, 0x1, 0x1})
	/usr/local/go/src/reflect/value.go:339 +0xc5
github.com/kelindar/binary.(*customCodec).DecodeTo(0x10dbd40, 0xc000098450, {0x10e9dc0, 0xc0000ac0b0, 0xc0000ac0b0})
	/Users/mark/go/pkg/mod/github.com/kelindar/[email protected]/codecs.go:323 +0x197
github.com/kelindar/binary.(*reflectStructCodec).DecodeTo(0x10cf4e0, 0xc0000ac0a8, {0x10dbd40, 0xc0000ac0b0, 0xc0000ac0b0})
	/Users/mark/go/pkg/mod/github.com/kelindar/[email protected]/codecs.go:273 +0x148
github.com/kelindar/binary.(*reflectPointerCodec).DecodeTo(0xc000096330, 0xc000098450, {0x10cf4e0, 0xc0000ac0a8, 0xc0000a4180})
	/Users/mark/go/pkg/mod/github.com/kelindar/[email protected]/codecs.go:246 +0x15c
github.com/kelindar/binary.(*reflectSliceCodec).DecodeTo(0xc000096340, 0xc000098450, {0x10d0e20, 0xc0000a4168, 0x106bdd2})
	/Users/mark/go/pkg/mod/github.com/kelindar/[email protected]/codecs.go:94 +0x4f7
github.com/kelindar/binary.(*Decoder).Decode(0xc000098450, {0x10cea20, 0xc0000a4168})
	/Users/mark/go/pkg/mod/github.com/kelindar/[email protected]/decoder.go:71 +0x119
github.com/kelindar/binary.Unmarshal({0xc0000b6000, 0x12, 0x40}, {0x10cea20, 0xc0000a4168})
	/Users/mark/go/pkg/mod/github.com/kelindar/[email protected]/decoder.go:34 +0xba
main.main()
	/Users/mark/go2/main.go:19 +0x10f
exit status 2

I'm having a look at t fix, but my reflect knowledge is limited.

unsupported type interface {}

Hello! 👋

We have data of the form:

type NotificationData struct {
	Metadata map[string]interface{}
}

// Notification model
type Notification struct {
	Data NotificationData
}

And it breaks with:

unsupported type interface {}

Is this something you would consider supporting?

Can try and get a repeatable example if you need!

typo in module package name

github.com/Kelindar/binary: github.com/Kelindar/[email protected]: parsing go.mod:
module declares its path as: github.com/kelindar/binary
but was required as: github.com/Kelindar/binary

add examples to docs

I found in the kelindar source code that it is possible to customize serialization (custom GetBinaryCodec or custom MarshalBinary+UnmarshalBinary methods), but there is only one sentence about this in the documentation: Support for custom BinaryMarshaler and BinaryUnmarshaler for tighter packing control

I think we need describe this in details and with examples of code

[]interface{} failed

binary: unsupported type interface {}
binary.Marshal failed due to []interface{}

json.Marshall has no issue with it.

`ReadString` of `streamReader` may only read partial of the string with trailing '\0'

Because the implementation of Slice method uses the Read method of io.Reader wrongly.

func (r *streamReader) Slice(n int) (buffer []byte, err error) {
	if n <= 10 {
		buffer = r.scratch[:n]
	} else {
		buffer = make([]byte, n, n)
	}

	_, err = r.Read(buffer)
	return
}

According to the doc of io.Reader, Read method reads up to len(p) bytes into p. It may actually read less. The unread part has the zero value.

The correct way is to use io.ReadFull.

This issue occurred in real life when I was trying to decode a bunch of the data from a file.

Tags, Ignore Field

It does not look like there is a way to ignore struct fields using tags such as json:"-"
Tried to use as a drop-in replacement to encoding/json and got unsupported type *thirdParty.Struct.

Is ignoring struct fields a feature that is on the roadmap?

Non Support for Pointer struct Fields

for example :

type (
tesState struct {
time *timeStruct
}
)

type timeStruct struct {
time int64
}

if we try to encode this .. it will return
binary: unsupported type *timeStruct

because timeStruct is a pointer

but gob has this one supported.. but it block unexpected fields

kelindar / binary Goto Github PK

binary's Introduction

Hey 👋, I'm Roman!

binary's People

Contributors

Stargazers

Watchers

Forkers

binary's Issues

Recommend Projects

Recommend Topics

Recommend Org