Save the program at the bottom to e.g. test_zstd.go. Run:

$ go run test_zstd.go
signal: killed

The program crashes before even the first line gets executed.

This is go 1.8 installed via brew.

Mac OS 10.12.3 (16D32)

I installed zstd with go get -u github.com/DataDog/zstd

$ go version
go version go1.8 darwin/amd64

package main

import (
	"bytes"
	"fmt"
	"log"

	"github.com/DataDog/zstd"
)

func main() {
	fmt.Printf("Starting the program\n")
	n := 16 * 1024
	d := make([]byte, n, n)
	for i := 0; i < n; i++ {
		d[i] = byte(i % 256)
	}
	c, err := zstd.Compress(nil, d)
	if err != nil {
		log.Fatalf("zstd.Compress() failed with %s\n", err)
	}
	fmt.Printf("After compression: %d bytes\n", len(c))
	dec, err := zstd.Decompress(nil, c)
	if err != nil {
		log.Fatalf("zstd.Decompress() failed with %s\n", err)
	}
	if !bytes.Equal(d, dec) {
		log.Fatalf("decompressed data different than original data\n")
	}
	fmt.Printf("Finishing program\n")
}

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.13.4 linux/amd64

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go:/data/projects/go/"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build896654248=/tmp/go-build -gno-record-gcc-switches"

What did you do?

go build

If possible, provide a recipe for reproducing the error.
If you have issues building, please parse the output of go build -x

What did you expect to see?

github.com/DataDog/zstd

/tmp/cc4tDPwc.s: Assembler messages:
/tmp/cc4tDPwc.s:726: Error: no such instruction: shlx %r10,%r9,%r10' /tmp/cc4tDPwc.s:746: Error: no such instruction: shlx %r9,%rax,%rax'
/tmp/cc4tDPwc.s:797: Error: no such instruction: shrx %r9,%rax,%r9' /tmp/cc4tDPwc.s:901: Error: no such instruction: shlx %rcx,%r14,%rcx'
/tmp/cc4tDPwc.s:910: Error: no such instruction: shlx %rax,%r15,%rax' /tmp/cc4tDPwc.s:933: Error: no such instruction: shlx %rbx,%rcx,%rbx'
/tmp/cc4tDPwc.s:958: Error: no such instruction: shlx %r10,%rcx,%rcx' /tmp/cc4tDPwc.s:1030: Error: no such instruction: shrx %r9,%rax,%r9'
/tmp/cc4tDPwc.s:1046: Error: no such instruction: `shlx %rax,%rdx,%rax'
FAIL github.com/DataDog/zstd [build failed]

What did you see instead?

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.9.1 darwin/amd64

What did you do?

If possible, provide a recipe for reproducing the error.
If you have issues building, please parse the output of go build -x

I'd like to upgrade the zstd we used from 0.7 to 1.3 with legacy support, what's the recommended way to do that?

What did you expect to see?

zstd 1.3 could decompress data which is compressed with zstd 0.7

What did you see instead?

It failed with "Unknown frame descriptor"

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.10 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/managata/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/managata/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build158247966=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ dd if=/dev/zero of=0.raw bs=1M count=100
$ zstd -3 0.raw
$ cat s.go

package main

import (
  "io"
  "io/ioutil"
  "os"

  "github.com/DataDog/zstd"
)

func main() {
  r, _ := os.Open("0.raw")
  w, _ := os.Create("0.stream")
  z := zstd.NewWriterLevel(w, 3)
  io.Copy(z, r)
  r.Close()
  z.Close()
  w.Close()

  b, _ := ioutil.ReadFile("0.raw")
  c, _ := zstd.CompressLevel(nil, b, 3)
  z.Close()
  ioutil.WriteFile("0.simple", c, 0644)
}

$ go build
$ ./s
$ ls -alp 0.*

-rw-r--r-- 1 m h 104857600 18-03-17 14:49:56 0.raw
-rw-r--r-- 1 m h      9615 18-03-17 14:49:56 0.raw.zst
-rw-r--r-- 1 m h      9611 18-03-17 15:09:59 0.simple
-rw-r--r-- 1 m h     38409 18-03-17 15:09:59 0.stream

What did you expect to see?

'0.stream' should be same size as '0.simple'.

What did you see instead?

0.stream is much larger than '0.simple'.

Did I misse something?
In my test (https://github.com/managata/zstd), ZSTD_initCStream() is working as expect.

Hi! I've encountered a strange error in backup recovery tool WAL-G using Zstd compression.
Here's a call stack. Who do you think, can this be caused by corrupted data on WAL-G side or is it a problem in Zstd bindings?
Here https://github.com/DataDog/zstd/blob/1.x/zstd_stream.go#L240 code does not check that src has at least one byte.

What version of Go are you using (go version)?

1.10

What operating system and processor architecture are you using (go env)?

Linux 86_64

What did you do?

wal-g backup-fetch of 11Tb database

What did you expect to see?

What did you see instead?

panic: runtime error: index out of range

goroutine 4949956 [running]:
github.com/x4m/wal-g/vendor/github.com/DataDog/zstd.(*reader).Read(0xc460b9c360, 0xc43aa28000, 0x1400000, 0x1400000, 0x1400000, 0x0, 0x0)
        /home/robot-gerrit/workspace/wal-g-build-deb/src/github.com/x4m/wal-g/obj-x86_64-linux-gnu/src/github.com/x4m/wal-g/vendor/github.com/DataDog/zstd/zstd_stream.go:240 +0x924
github.com/x4m/wal-g.fastCopy(0xf123a0, 0xc44de66c80, 0x7f5e745d9198, 0xc460b9c360, 0xc460b9c360, 0xc4464e5e60, 0xc420c3bf28)
        /home/robot-gerrit/workspace/wal-g-build-deb/src/github.com/x4m/wal-g/obj-x86_64-linux-gnu/src/github.com/x4m/wal-g/

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.14

What operating system and processor architecture are you using (go env)?

mac os 10.15.6 Intel Core i7

What did you do?

If possible, provide a recipe for reproducing the error.
If you have issues building, please parse the output of go build -x

I want to process two sources of data, one is compressed by zstd, but the other is not, at the end , they will both call the DeCompress function provided by you, will DeCompress function distinguish them？

What did you expect to see?

When uncompressed data call DeCompress function, DeCompress function will do noting

What did you see instead?

Hi,

Consider adding // #cgo CFLAGS: -O2 directive to zstd.go ?

Although i don't know the performance impact (of neither build time nor run time) whether zstd is helped by this flag or if even -O3 is better.

Otherwise, maybe it would be better to document existence of CGO_CFLAGS environment variable.

Thanks

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.15.8 linux/amd64

What operating system and processor architecture are you using (go env)?

GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOROOT="/usr/lib/go-1.15"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.15/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build135663132=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I tried to work with package "archive/tar".

package main

import (
	"archive/tar"
	"io"
	"os"

	"github.com/DataDog/zstd"
)

func main() {
	out, err := os.Create("out.tar.zstd")
	if err != nil {
		panic(err)
	}
	defer out.Close()

	in := "infile.txt"
	fi, err := os.Stat(in)
	if err != nil {
		panic(err)
	}

	zw := zstd.NewWriter(out)
	defer zw.Close()

	tw := tar.NewWriter(zw)
	defer tw.Close()

	hdr, err := tar.FileInfoHeader(fi, fi.Name())
	if err != nil {
		panic(err)
	}

	if err := tw.WriteHeader(hdr); err != nil {
		panic(err)
	}

	file, err := os.OpenFile(in, os.O_RDONLY, 0)
	if err != nil {
		panic(err)
	}
	defer file.Close()

	if _, err := io.Copy(tw, file); err != nil {
		panic(err)
	}
}

What did you expect to see?

Generate a tar and compressed file.

What did you see instead?

panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 1 [running]:
github.com/DataDog/zstd.(*Writer).Write.func1(0xc000020080, 0xc00010cb80, 0xc00010cb80, 0x200, 0x200)
        /root/go/pkg/mod/github.com/!data!dog/[email protected]/zstd_stream.go:181 +0xe5
github.com/DataDog/zstd.(*Writer).Write(0xc000020080, 0xc00010cb80, 0x200, 0x200, 0x1648, 0x0, 0x0)
        /root/go/pkg/mod/github.com/!data!dog/[email protected]/zstd_stream.go:181 +0xf5
archive/tar.(*Writer).writeRawHeader(0xc00010ca80, 0xc00010cb80, 0x50aba0, 0x30, 0x0, 0x0)
        /usr/lib/go-1.15/src/archive/tar/writer.go:392 +0x7d
archive/tar.(*Writer).writeUSTARHeader(0xc00010ca80, 0xc00010caa8, 0xa, 0x0)
        /usr/lib/go-1.15/src/archive/tar/writer.go:125 +0x1be
archive/tar.(*Writer).WriteHeader(0xc00010ca80, 0xc00011a000, 0x59e34c, 0xa)
        /usr/lib/go-1.15/src/archive/tar/writer.go:97 +0x2e5
main.main()
        /root/test/main.go:34 +0x287

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11.1 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/xxxx/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/xxxx/golang"
GOPROXY=""
GORACE=""
GOROOT="/da1/xxxx/go"
GOTMPDIR=""
GOTOOLDIR="/da1/xxxx/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build569555389=/tmp/go-build -gno-record-gcc-switches"

What did you do?

If possible, provide a recipe for reproducing the error.
If you have issues building, please parse the output of go build -x

go get github.com/DataDog/zstd

What did you expect to see?

No error and I got it

What did you see instead?

github.com/DataDog/zstd

/tmp/ccAv7mRw.s: Assembler messages:
/tmp/ccAv7mRw.s:126: Error: no such instruction: shlx %r11,%r9,%r9' /tmp/ccAv7mRw.s:154: Error: no such instruction: shlx %r10,%rcx,%rcx'
/tmp/ccAv7mRw.s:199: Error: no such instruction: shrx %rcx,%r9,%r9' /tmp/ccAv7mRw.s:286: Error: no such instruction: shlx %rsi,%rcx,%rcx'
/tmp/ccAv7mRw.s:314: Error: no such instruction: shlx %rax,%rsi,%rax' /tmp/ccAv7mRw.s:339: Error: no such instruction: shlx %r13,%rsi,%rsi'
/tmp/ccAv7mRw.s:357: Error: no such instruction: shlx %r10,%rsi,%rsi' /tmp/ccAv7mRw.s:419: Error: no such instruction: shrx %rsi,%rcx,%r9'
/tmp/ccAv7mRw.s:437: Error: no such instruction: `shlx %rsi,%rcx,%rcx'

Is there some reason why I shouldn't treat the compression step as a WriteCloser? Why does that return a public struct instead of an interface, like how decompression does?

Thanks!

Go Versions benchmarks

Tests and benchs (payload is mr from silesia corpus):

ᐅ eval "$(gimme 1.4.2)"
go version go1.4.2 linux/amd64
ᐅ go test -bench .
PASS
BenchmarkUnsafeDecompress         30      47481504 ns/op     209.99 MB/s
BenchmarkFzlibDecompress          30      48686951 ns/op     204.79 MB/s
BenchmarkCzlibDecompress          20      53068259 ns/op     187.88 MB/s
BenchmarkZlibDecompress       10     158092207 ns/op      63.07 MB/s
BenchmarkUnsafeCompress        2     664396361 ns/op      15.01 MB/s
BenchmarkFzlibCompress         2     703111095 ns/op      14.18 MB/s
BenchmarkCzlibCompress         2     737086124 ns/op      13.53 MB/s
BenchmarkZlibCompress          1    2163086773 ns/op       4.61 MB/s

ᐅ eval "$(gimme 1.6.1)"
go version go1.6.1 linux/amd64
ᐅ go test -bench .
PASS
BenchmarkUnsafeDecompress-4       30      74960391 ns/op     133.01 MB/s
BenchmarkFzlibDecompress-4        20      71395036 ns/op     139.65 MB/s
BenchmarkCzlibDecompress-4         1    1929882049 ns/op       5.17 MB/s
BenchmarkZlibDecompress-4          5     202957234 ns/op      49.13 MB/s
BenchmarkUnsafeCompress-4          2     716153507 ns/op      13.92 MB/s
BenchmarkFzlibCompress-4           2     712060351 ns/op      14.00 MB/s
BenchmarkCzlibCompress-4           2     976679206 ns/op      10.21 MB/s
BenchmarkZlibCompress-4            1    1808744762 ns/op       5.51 MB/s

ᐅ eval "$(gimme 1.6.1)"
go version go1.6.1 linux/amd64
ᐅ export GODEBUG=cgocheck=0
ᐅ go test -bench .
PASS
BenchmarkUnsafeDecompress-4       20      64997217 ns/op     153.40 MB/s
BenchmarkFzlibDecompress-4        20      69500434 ns/op     143.46 MB/s
BenchmarkCzlibDecompress-4        20      58652419 ns/op     169.99 MB/s
BenchmarkZlibDecompress-4         10     138611158 ns/op      71.93 MB/s
BenchmarkUnsafeCompress-4          2     671414987 ns/op      14.85 MB/s
BenchmarkFzlibCompress-4           2     676213336 ns/op      14.74 MB/s
BenchmarkCzlibCompress-4           2     754176802 ns/op      13.22 MB/s
BenchmarkZlibCompress-4            1    1482097992 ns/op       6.73 MB/s

First of all, thanks for a nice library!

I ran go test -v and got the output zstd_test.go:117: PAYLOAD env was not set.
However I don't know what value to set to PAYLOAD environment variable.

Could you add an explanation to README?
Thanks!

Hello,
can you update to zstd 1.4.4 library ?
regards,
Nicolas.

github.com/DataDog/zstd

'+bmi2' is not a recognized feature for this target (ignoring feature)

go version
go version go1.10.4 linux/amd64

go env
GOARCH="amd64"
GOBIN="/home/mayurw/go/bin"
GOCACHE="/home/mayurw/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/mayurw/go"
GORACE=""
GOROOT="/usr/lib/go-1.10"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build555451646=/tmp/go-build -gno-record-gcc-switches"

I executed the command
env GOOS=linux GOARCH=arm64 go build
it produced output as bolow
go build github.com/DataDog/zstd: build constraints exclude all Go files in ~/go/src/github.com/DataDog/zstd

I'm really not getting what error it is. Please provide solution to go cross compilation from linux to ARM architecture binaries..

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 windows/amd64

What operating system and processor architecture are you using (go env)?

Windows 10

go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\ciczhang\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\NextGenPO\GoWorkplace
set GOPROXY=
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\ciczhang\AppData\Local\Temp\go-build594543906=/tmp/go-build -gno-record-gcc-switches

What did you do?

Building in windows 10 failed

C:\Users\ciczhang\AppData\Local\Temp\go-link-108982731\000004.o: In function COVER_buildDictionary': /cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/cover.c:678: undefined reference to __getreent'
/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/cover.c:678: undefined reference to __getreent' /cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/cover.c:710: undefined reference to __getreent'
/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/cover.c:710: undefined reference to __getreent' /cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/cover.c:714: undefined reference to __getreent'
C:\Users\ciczhang\AppData\Local\Temp\go-link-108982731\000004.o:/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/cover.c:714: more undefined references to __getreent' follow C:\Users\ciczhang\AppData\Local\Temp\go-link-108982731\000006.o: In function ss_mintrosort':
/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/divsufsort.c:444: undefined reference to __assert_func' /cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/divsufsort.c:517: undefined reference to __assert_func'
/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/divsufsort.c:516: undefined reference to __assert_func' /cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/divsufsort.c:503: undefined reference to __assert_func'
/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/divsufsort.c:502: undefined reference to __assert_func' C:\Users\ciczhang\AppData\Local\Temp\go-link-108982731\000006.o:/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/divsufsort.c:499: more undefined references to __assert_func' follow
C:\Users\ciczhang\AppData\Local\Temp\go-link-108982731\000009.o: In function FASTCOVER_buildDictionary': /cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/fastcover.c:395: undefined reference to __getreent'
/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/fastcover.c:395: undefined reference to __getreent' /cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/fastcover.c:434: undefined reference to __getreent'
/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/fastcover.c:434: undefined reference to __getreent' /cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/fastcover.c:430: undefined reference to __getreent'
C:\Users\ciczhang\AppData\Local\Temp\go-link-108982731\000009.o:/cygdrive/c/NextGenPO/GoWorkplace/src/bitbucket-eng-sjc1.cisco.com/bitbucket/nextg/lds-microservice/vendor/github.com/DataDog/zstd/fastcover.c:430: more undefined references to `__getreent' follow
collect2.exe: error: ld returned 1 exit status

What did you expect to see?

Should build successfully

What did you see instead?

I initially got gcc not found error, so I installed TDM-GCC-64 and add path C:\TDM-GCC-64\bin, then, I got above error

Testing: "/builddir/build/BUILD/zstd-1.3.4/_build/src/github.com/DataDog/zstd"
+ GOPATH=/builddir/build/BUILD/zstd-1.3.4/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
# github.com/DataDog/zstd [github.com/DataDog/zstd.test]
./zstd_test.go:30:53: constant 54564654653 overflows int
FAIL	github.com/DataDog/zstd [build failed]

with Go 1.11.1.

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10.1

What operating system and processor architecture are you using (go env)?

centeros 7.5

What did you do?

If possible, provide a recipe for reproducing the error.
If you have issues building, please parse the output of go build -x

What did you expect to see?

What did you see instead?

when i run go install
DataDog/zstd exec: "gcc": executable file not found in $PATH

We're moving other repos away from travis-ci. Can you switch the tests to run in CircleCI instead?

Following up on #4

I have started working on updating it to 1.1.3 but doesn't seem straightforward as some of the methods we use are deprecated in 1.1.2:

API : zbuff : changed : prototypes now generate deprecation warnings

I haven't done a lot yet so if anyone have a PR handy that would be great, otherwise I'll try to work on it soon

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.12.1 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\mfiles\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=e:\gopath\
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=e:\temp\wintemp\go-build152196250=/tmp/go-build -gno-record-gcc-switches

Package version: 809b919c325d7887bff7bd876162af73db53e878 (v.1.4.0 tag)

What did you do?

Using standard zstd.Decompress function.

Reproducer: badref.zip - includes main function and sample data)

Using the commandline tool is fine:

zstd.exe -d 2aa9ddad029fa86d4e4831b22051a3b8414f015f.zst
2aa9ddad029fa86d4e4831b22051a3b8414f015f.zst: 137 bytes

What did you expect to see?

The length of the decompressed data (137 bytes)

What did you see instead?

panic: Corrupted block detected

What did you do?

I compressed an empty file with zstd:

$ touch empty

$ zstd empty
empty                :1300.00%   (     0 =>     13 bytes, empty.zst)

Then I compressed an empty slice in Go:

package main

import (
	"log"

	"github.com/DataDog/zstd"
)

func main() {
	input := []byte{}

	output, err := zstd.CompressLevel(nil, input, 0)
	if err != nil {
		log.Fatalf("Error compressing: %s", err)
	}

	log.Printf("input      [len = %d]: %#v", len(input), input)
	log.Printf("compressed [len = %d]: %#v", len(output), output)
}

What did you expect to see?

Both outputs are the same.

What did you see instead?

13 bytes in zstd cli output:

$ ls -la empty*
-rw-r--r--  1 bobrik  wheel   0 Oct  1 11:07 empty
-rw-r--r--  1 bobrik  wheel  13 Oct  1 11:07 empty.zst

$ cat empty.zst | hexdump -C
00000000  28 b5 2f fd 24 00 01 00  00 99 e9 d8 51           |(./.$.......Q|
0000000d

9 bytes in Go output:

$ go run zstd.go
2018/10/01 11:09:09 input      [len = 0]: []byte{}
2018/10/01 11:09:09 compressed [len = 9]: []byte{0x28, 0xb5, 0x2f, 0xfd, 0x20, 0x0, 0x1, 0x0, 0x0}

Last 4 bytes are missing, which means no optional checksum:

• https://tools.ietf.org/id/draft-kucherawy-dispatch-zstd-00.html#comp_zstd_frames

Checksums are great and I think there should be a way to enable them.

What version of Go are you using (go version)?

go version go1.13.6 windows/amd64

What operating system and processor architecture are you using (go env)?

Windows 10

set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\homepc\AppData\Local\go-build
set GOENV=C:\Users\homepc\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\homepc\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\homepc\AppData\Local\Temp\go-build692364802=/tmp/go-build -gno-record-gcc-switches

What did you do?

Build in windows 10 failed

C:\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:\Users\homepc\AppData\Local\Temp\go-link-980381163\000004.o: In function `COVER_buildDictionary':
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/cover.c:678: undefined reference to `__getreent'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/cover.c:678: undefined reference to `__getreent'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/cover.c:714: undefined reference to `__getreent'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/cover.c:714: undefined reference to `__getreent'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/cover.c:710: undefined reference to `__getreent'
C:\Users\homepc\AppData\Local\Temp\go-link-980381163\000004.o:/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/cover.c:710: more undefined references to `__getreent' follow
C:\Users\homepc\AppData\Local\Temp\go-link-980381163\000006.o: In function `ss_mintrosort':
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/divsufsort.c:495: undefined reference to `__assert_func'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/divsufsort.c:494: undefined reference to `__assert_func'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/divsufsort.c:509: undefined reference to `__assert_func'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/divsufsort.c:499: undefined reference to `__assert_func'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/divsufsort.c:498: undefined reference to `__assert_func'
C:\Users\homepc\AppData\Local\Temp\go-link-980381163\000006.o:/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/divsufsort.c:513: more undefined references to `__assert_func' follow
C:\Users\homepc\AppData\Local\Temp\go-link-980381163\000009.o: In function `FASTCOVER_ctx_init':
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/fastcover.c:311: undefined reference to `__getreent'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/fastcover.c:311: undefined reference to `__getreent'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/fastcover.c:318: undefined reference to `__getreent'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/fastcover.c:318: undefined reference to `__getreent'
/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/fastcover.c:324: undefined reference to `__getreent'
C:\Users\homepc\AppData\Local\Temp\go-link-980381163\000009.o:/cygdrive/c/Users/homepc/go/pkg/mod/github.com/!data!dog/[email protected]/fastcover.c:324: more undefined references to `__getreent' follow
collect2.exe: error: ld returned 1 exit status

What did you expect to see?

Should build successfully

What did you see instead?

I initially got gcc not found error, so I installed TDM-GCC-64 and add path C:\TDM-GCC-64\bin, then, I got above error

I think it's related to this issue #63

Basic example from cgo (https://golang.org/cmd/cgo/, where C code may call function pointers received ...) compiles and runs successfully, so I don't think there are some problems with my local installation.

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.12

What operating system and processor architecture are you using (go env)?

darwin+amd64

What did you do?

export GOOS=linux
and go build -x

What did you expect to see?

successful build

What did you see instead?

output of go build -x:

WORK=/var/folders/28/5svspm_d5f3fnhfgdvnn_n6090q7g6/T/go-build576916263
mkdir -p $WORK/b168/
go build github.com/DataDog/zstd: build constraints exclude all Go files in ...```

I just did the first thing to try this out, but it doesn't work.

Test program

package main

import (
    "io"
    "log"
    "os"

    "github.com/DataDog/zstd"
)

func main() {
    zr := zstd.NewReader(os.Stdin)

    _, err := io.Copy(os.Stdout, zr)
    if err != nil {
        log.Fatal(err)
    }
}

Error

Then I run this:

$ echo hello | zstd | go run program.go

But I get:

2016/07/19 21:30:35 failed to decompress: Unknown frame descriptor

Am I missing something obvious?

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.9.2

What operating system and processor architecture are you using (go env)?

windows 7, intel i7

What did you do?

go build
If possible, provide a recipe for reproducing the error.
If you have issues building, please parse the output of go build -x
go build -x
WORK=C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774
mkdir -p $WORK\github.com\olebedev\config_obj
mkdir -p $WORK\github.com\olebedev
mkdir -p $WORK\go.uber.org\zap_obj
cd F:\gopath\src\github.com\olebedev\config
"C:\Go\pkg\tool\windows_amd64\compile.exe" -o "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774\github.com\olebedev\config.a" -trimpath "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774" -goversion go1.9.2 -p github.com/olebedev/config -complete -buildid 74bc28f638efc355043676b5e6a90c19642bf116 -D /F/gopath/src/github.com/olebedev/config -I "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774" -I "F:\gopath\pkg\windows_amd64" -pack "F:\gopath\src\github.com\olebedev\config\config.go" "F:\gopath\src\github.com\olebedev\config\doc.go"
mkdir -p $WORK\go.uber.org
mkdir -p $WORK\github.com\davecgh\go-spew\spew_obj
cd F:\gopath\src\go.uber.org\zap
"C:\Go\pkg\tool\windows_amd64\compile.exe" -o "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774\go.uber.org\zap.a" -trimpath "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774" -goversion go1.9.2 -p go.uber.org/zap -complete -buildid 254bc71fa5beb9daad916e503e345838c3a98757 -D /F/gopath/src/go.uber.org/zap -I "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774" -I "F:\gopath\pkg\windows_amd64" -pack "F:\gopath\src\go.uber.org\zap\array.go" "F:\gopath\src\go.uber.org\zap\config.go" "F:\gopath\src\go.uber.org\zap\doc.go" "F:\gopath\src\go.uber.org\zap\encoder.go" "F:\gopath\src\go.uber.org\zap\error.go" "F:\gopath\src\go.uber.org\zap\field.go" "F:\gopath\src\go.uber.org\zap\flag.go" "F:\gopath\src\go.uber.org\zap\global.go" "F:\gopath\src\go.uber.org\zap\http_handler.go" "F:\gopath\src\go.uber.org\zap\level.go" "F:\gopath\src\go.uber.org\zap\logger.go" "F:\gopath\src\go.uber.org\zap\options.go" "F:\gopath\src\go.uber.org\zap\sink.go" "F:\gopath\src\go.uber.org\zap\stacktrace.go" "F:\gopath\src\go.uber.org\zap\sugar.go" "F:\gopath\src\go.uber.org\zap\time.go" "F:\gopath\src\go.uber.org\zap\writer.go"
mkdir -p $WORK\github.com\davecgh\go-spew
cd F:\gopath\src\github.com\davecgh\go-spew\spew
"C:\Go\pkg\tool\windows_amd64\compile.exe" -o "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774\github.com\davecgh\go-spew\spew.a" -trimpath "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774" -goversion go1.9.2 -p github.com/davecgh/go-spew/spew -complete -buildid a2dd230e6cba51607e484b8fc6f8c43980cd9dec -D /F/gopath/src/github.com/davecgh/go-spew/spew -I "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774" -pack "F:\gopath\src\github.com\davecgh\go-spew\spew\bypass.go" "F:\gopath\src\github.com\davecgh\go-spew\spew\common.go" "F:\gopath\src\github.com\davecgh\go-spew\spew\config.go" "F:\gopath\src\github.com\davecgh\go-spew\spew\doc.go" "F:\gopath\src\github.com\davecgh\go-spew\spew\dump.go" "F:\gopath\src\github.com\davecgh\go-spew\spew\format.go" "F:\gopath\src\github.com\davecgh\go-spew\spew\spew.go"
mkdir -p $WORK\github.com\DataDog\zstd_obj
mkdir -p $WORK\github.com\DataDog
cd F:\gopath\src\github.com\DataDog\zstd
CGO_LDFLAGS="-g" "-O2" "C:\Go\pkg\tool\windows_amd64\cgo.exe" -objdir "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774\github.com\DataDog\zstd\_obj\" -importpath github.com/DataDog/zstd -- -I "C:\Users\ADMINI1\AppData\Local\Temp\go-build892563774\github.com\DataDog\zstd\_obj\" -g -O2 errors.go zstd.go zstd_stream.go

github.com/DataDog/zstd

exec: "gcc": executable file not found in %PATH%

What did you expect to see?

build pass

What did you see instead?

$ go build

github.com/DataDog/zstd

exec: "gcc": executable file not found in %PATH%

What version of Go are you using (go version)?

go version go1.15.6 linux/arm64

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/home/iurt/.cache/go-build"
GOENV="/home/iurt/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/iurt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/iurt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_arm64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build790893092=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Building golang-github-datadog-zstd 1.4.5-patch1 rpm package. It sometimes passes but most of the time fails.

$ go test -x
WORK=/tmp/go-build935713179
mkdir -p $WORK/b001/
mkdir -p $WORK/b019/
mkdir -p $WORK/b006/
mkdir -p $WORK/b010/
mkdir -p $WORK/b029/
mkdir -p $WORK/b013/
mkdir -p $WORK/b027/
mkdir -p $WORK/b012/
mkdir -p $WORK/b030/
mkdir -p $WORK/b011/
mkdir -p $WORK/b017/
mkdir -p $WORK/b057/
cp /home/iurt/.cache/go-build/07/071283d23ffd9eabfe3ce2a77213fdede77a69bca99ce3f55bad15bd0e82dc52-d $WORK/b057/_cgo_gotypes.go
mkdir -p $WORK/b024/
mkdir -p $WORK/b009/
cp /home/iurt/.cache/go-build/45/458a10e66e97e71eb214780d398c888757bb200bd46d52129d734e108e41ad57-d $WORK/b057/cgo.cgo1.go
cp /home/iurt/.cache/go-build/eb/ebc0b501bd5111b65729e9e43bc6238dc3804efad9dcf7b94cb214311e9b3641-d $WORK/b057/_cgo_import.go
mkdir -p $WORK/b026/
mkdir -p $WORK/b008/
mkdir -p $WORK/b005/
mkdir -p $WORK/b020/
mkdir -p $WORK/b004/
mkdir -p $WORK/b036/
mkdir -p $WORK/b014/
mkdir -p $WORK/b021/
mkdir -p $WORK/b032/
mkdir -p $WORK/b056/
mkdir -p $WORK/b028/
mkdir -p $WORK/b018/
mkdir -p $WORK/b037/
mkdir -p $WORK/b016/
mkdir -p $WORK/b025/
mkdir -p $WORK/b023/
mkdir -p $WORK/b035/
mkdir -p $WORK/b022/
mkdir -p $WORK/b042/
mkdir -p $WORK/b015/
mkdir -p $WORK/b003/
mkdir -p $WORK/b040/
mkdir -p $WORK/b039/
mkdir -p $WORK/b034/
mkdir -p $WORK/b038/
mkdir -p $WORK/b041/
mkdir -p $WORK/b033/
mkdir -p $WORK/b055/
mkdir -p $WORK/b031/
mkdir -p $WORK/b054/
cp /home/iurt/.cache/go-build/25/25252efbf11bccaafa97b69fa8ea82f65bd48a38dac9f05c8e65eb6a4d3c3d15-d $WORK/b054/_cgo_gotypes.go
cp /home/iurt/.cache/go-build/26/269217122756ca3cb5842639012f3826cf3cc6a360b6d60c8f0058ca2520158b-d $WORK/b054/errors.cgo1.go
cp /home/iurt/.cache/go-build/6c/6c6e2587e25cc71d1c138038e14ef40d0d73d668c076809c4864e93ae32fd819-d $WORK/b054/zstd.cgo1.go
cp /home/iurt/.cache/go-build/88/8812e04c91798cf5b035e0d5a583accbf4ffd0bda99fe9df31d7f24139cd7de4-d $WORK/b054/zstd_ctx.cgo1.go
cp /home/iurt/.cache/go-build/19/195c0895ff8c3d72f0a206add7f2ee648863722ddc770efd511ca51388fb332e-d $WORK/b054/zstd_stream.cgo1.go
cp /home/iurt/.cache/go-build/96/96e686d4d25f928e81ae4810ef9a14627bcb29e9ae7a6027892006311342e50b-d $WORK/b054/_cgo_import.go
cat >$WORK/b054/vet.cfg << 'EOF' # internal
{
	"ID": "_/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1",
	"Compiler": "gc",
	"Dir": "/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1",
	"ImportPath": "_/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1",
	"GoFiles": [
		"/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1/errors_test.go",
		"/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1/helpers_test.go",
		"/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1/zstd_ctx_test.go",
		"/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1/zstd_stream_test.go",
		"/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1/zstd_test.go",
		"$WORK/b054/_cgo_gotypes.go",
		"$WORK/b054/errors.cgo1.go",
		"$WORK/b054/zstd.cgo1.go",
		"$WORK/b054/zstd_ctx.cgo1.go",
		"$WORK/b054/zstd_stream.cgo1.go",
		"$WORK/b054/_cgo_import.go"
	],
	"NonGoFiles": [],
	"ImportMap": {
		"C": "C",
		"bytes": "bytes",
		"errors": "errors",
		"fmt": "fmt",
		"io": "io",
		"io/ioutil": "io/ioutil",
		"log": "log",
		"math/rand": "math/rand",
		"os": "os",
		"runtime": "runtime",
		"runtime/cgo": "runtime/cgo",
		"runtime/debug": "runtime/debug",
		"sync": "sync",
		"syscall": "syscall",
		"testing": "testing",
		"time": "time",
		"unsafe": "unsafe"
	},
	"PackageFile": {
		"bytes": "/usr/lib/golang/pkg/linux_arm64/bytes.a",
		"errors": "/usr/lib/golang/pkg/linux_arm64/errors.a",
		"fmt": "/usr/lib/golang/pkg/linux_arm64/fmt.a",
		"io": "/usr/lib/golang/pkg/linux_arm64/io.a",
		"io/ioutil": "/usr/lib/golang/pkg/linux_arm64/io/ioutil.a",
		"log": "/usr/lib/golang/pkg/linux_arm64/log.a",
		"math/rand": "/usr/lib/golang/pkg/linux_arm64/math/rand.a",
		"os": "/usr/lib/golang/pkg/linux_arm64/os.a",
		"runtime": "/usr/lib/golang/pkg/linux_arm64/runtime.a",
		"runtime/cgo": "/usr/lib/golang/pkg/linux_arm64/runtime/cgo.a",
		"runtime/debug": "/usr/lib/golang/pkg/linux_arm64/runtime/debug.a",
		"sync": "/usr/lib/golang/pkg/linux_arm64/sync.a",
		"syscall": "/usr/lib/golang/pkg/linux_arm64/syscall.a",
		"testing": "/usr/lib/golang/pkg/linux_arm64/testing.a",
		"time": "/usr/lib/golang/pkg/linux_arm64/time.a"
	},
	"Standard": {
		"bytes": true,
		"errors": true,
		"fmt": true,
		"io": true,
		"io/ioutil": true,
		"log": true,
		"math/rand": true,
		"os": true,
		"runtime": true,
		"runtime/cgo": true,
		"runtime/debug": true,
		"sync": true,
		"syscall": true,
		"testing": true,
		"time": true,
		"unsafe": true
	},
	"PackageVetx": {
		"bytes": "/home/iurt/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d",
		"errors": "/home/iurt/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d",
		"fmt": "/home/iurt/.cache/go-build/6f/6f1a3e1f52c31e8342fdd695153c6d7a005caa69858e5628fb700df89b134021-d",
		"io": "/home/iurt/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d",
		"io/ioutil": "/home/iurt/.cache/go-build/5b/5b9172b5fe9e92464466e5c1459ae73de4a6c1cd1f7f4f795aa7a0e5ba8af6ac-d",
		"log": "/home/iurt/.cache/go-build/76/76b9459e6e6ba3a56312401978135470861b5d3081d732284d55b15b625db629-d",
		"math/rand": "/home/iurt/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d",
		"os": "/home/iurt/.cache/go-build/37/376397066f40ef1fac96c02af9c4a0699a0e5aa019bc41dafa3d93954e7fe7c8-d",
		"runtime": "/home/iurt/.cache/go-build/2d/2d26f8ed83dc6de73bc9496e28d0e378ec6945d8641b2016bae22f9c0ddb929b-d",
		"runtime/cgo": "/home/iurt/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d",
		"runtime/debug": "/home/iurt/.cache/go-build/7d/7d513fcf6ecfbc1b567a4f405fb6a1576ef05c49c8d868faa043913005d19976-d",
		"sync": "/home/iurt/.cache/go-build/2d/2d26f8ed83dc6de73bc9496e28d0e378ec6945d8641b2016bae22f9c0ddb929b-d",
		"syscall": "/home/iurt/.cache/go-build/77/7703a56ffad7e2d1533a584fb504cc800de76c80580922a8bc5d78959e8316c1-d",
		"testing": "/home/iurt/.cache/go-build/b4/b498e06c7950c7340d58bb00bea7e5178e197b2e50ce118ef05a726701692326-d",
		"time": "/home/iurt/.cache/go-build/77/7703a56ffad7e2d1533a584fb504cc800de76c80580922a8bc5d78959e8316c1-d"
	},
	"VetxOnly": false,
	"VetxOutput": "$WORK/b054/vet.out",
	"SucceedOnTypecheckFailure": false
}
EOF
cd /home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1
TERM='dumb' CC='/usr/bin/gcc' GCCGO='' /usr/lib/golang/pkg/tool/linux_arm64/vet -atomic -bool -buildtags -errorsas -ifaceassert -nilfunc -printf -stringintconv $WORK/b054/vet.cfg
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile _/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1.test=/home/iurt/.cache/go-build/3c/3ca69d193d6105bb30753889fdeeabd9d4730f14161e44dc46396646fca98c2a-d
packagefile os=/usr/lib/golang/pkg/linux_arm64/os.a
packagefile reflect=/usr/lib/golang/pkg/linux_arm64/reflect.a
packagefile testing=/usr/lib/golang/pkg/linux_arm64/testing.a
packagefile testing/internal/testdeps=/usr/lib/golang/pkg/linux_arm64/testing/internal/testdeps.a
packagefile runtime=/usr/lib/golang/pkg/linux_arm64/runtime.a
packagefile _/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1=/home/iurt/.cache/go-build/9c/9ce3c57ebbeb1fb2591398f8a3258b10779e0f296a0c501cd622c305e4784541-d
packagefile errors=/usr/lib/golang/pkg/linux_arm64/errors.a
packagefile internal/oserror=/usr/lib/golang/pkg/linux_arm64/internal/oserror.a
packagefile internal/poll=/usr/lib/golang/pkg/linux_arm64/internal/poll.a
packagefile internal/syscall/execenv=/usr/lib/golang/pkg/linux_arm64/internal/syscall/execenv.a
packagefile internal/syscall/unix=/usr/lib/golang/pkg/linux_arm64/internal/syscall/unix.a
packagefile internal/testlog=/usr/lib/golang/pkg/linux_arm64/internal/testlog.a
packagefile io=/usr/lib/golang/pkg/linux_arm64/io.a
packagefile sync=/usr/lib/golang/pkg/linux_arm64/sync.a
packagefile sync/atomic=/usr/lib/golang/pkg/linux_arm64/sync/atomic.a
packagefile syscall=/usr/lib/golang/pkg/linux_arm64/syscall.a
packagefile time=/usr/lib/golang/pkg/linux_arm64/time.a
packagefile internal/unsafeheader=/usr/lib/golang/pkg/linux_arm64/internal/unsafeheader.a
packagefile math=/usr/lib/golang/pkg/linux_arm64/math.a
packagefile strconv=/usr/lib/golang/pkg/linux_arm64/strconv.a
packagefile unicode=/usr/lib/golang/pkg/linux_arm64/unicode.a
packagefile unicode/utf8=/usr/lib/golang/pkg/linux_arm64/unicode/utf8.a
packagefile bytes=/usr/lib/golang/pkg/linux_arm64/bytes.a
packagefile flag=/usr/lib/golang/pkg/linux_arm64/flag.a
packagefile fmt=/usr/lib/golang/pkg/linux_arm64/fmt.a
packagefile internal/race=/usr/lib/golang/pkg/linux_arm64/internal/race.a
packagefile io/ioutil=/usr/lib/golang/pkg/linux_arm64/io/ioutil.a
packagefile runtime/debug=/usr/lib/golang/pkg/linux_arm64/runtime/debug.a
packagefile runtime/trace=/usr/lib/golang/pkg/linux_arm64/runtime/trace.a
packagefile sort=/usr/lib/golang/pkg/linux_arm64/sort.a
packagefile strings=/usr/lib/golang/pkg/linux_arm64/strings.a
packagefile bufio=/usr/lib/golang/pkg/linux_arm64/bufio.a
packagefile regexp=/usr/lib/golang/pkg/linux_arm64/regexp.a
packagefile runtime/pprof=/usr/lib/golang/pkg/linux_arm64/runtime/pprof.a
packagefile internal/bytealg=/usr/lib/golang/pkg/linux_arm64/internal/bytealg.a
packagefile internal/cpu=/usr/lib/golang/pkg/linux_arm64/internal/cpu.a
packagefile runtime/internal/atomic=/usr/lib/golang/pkg/linux_arm64/runtime/internal/atomic.a
packagefile runtime/internal/math=/usr/lib/golang/pkg/linux_arm64/runtime/internal/math.a
packagefile runtime/internal/sys=/usr/lib/golang/pkg/linux_arm64/runtime/internal/sys.a
packagefile log=/usr/lib/golang/pkg/linux_arm64/log.a
packagefile math/rand=/usr/lib/golang/pkg/linux_arm64/math/rand.a
packagefile runtime/cgo=/usr/lib/golang/pkg/linux_arm64/runtime/cgo.a
packagefile internal/reflectlite=/usr/lib/golang/pkg/linux_arm64/internal/reflectlite.a
packagefile math/bits=/usr/lib/golang/pkg/linux_arm64/math/bits.a
packagefile internal/fmtsort=/usr/lib/golang/pkg/linux_arm64/internal/fmtsort.a
packagefile path/filepath=/usr/lib/golang/pkg/linux_arm64/path/filepath.a
packagefile context=/usr/lib/golang/pkg/linux_arm64/context.a
packagefile regexp/syntax=/usr/lib/golang/pkg/linux_arm64/regexp/syntax.a
packagefile compress/gzip=/usr/lib/golang/pkg/linux_arm64/compress/gzip.a
packagefile encoding/binary=/usr/lib/golang/pkg/linux_arm64/encoding/binary.a
packagefile text/tabwriter=/usr/lib/golang/pkg/linux_arm64/text/tabwriter.a
packagefile compress/flate=/usr/lib/golang/pkg/linux_arm64/compress/flate.a
packagefile hash/crc32=/usr/lib/golang/pkg/linux_arm64/hash/crc32.a
packagefile hash=/usr/lib/golang/pkg/linux_arm64/hash.a
EOF
cd .
/usr/lib/golang/pkg/tool/linux_arm64/link -o $WORK/b001/zstd-1.4.5-patch1.test -importcfg $WORK/b001/importcfg.link -s -w -buildmode=exe -buildid=vE0vSAiEtirN37vxl8na/CtaR4_McDVI7Sxq5Z1TH/505bYDzzK5WzQ7NgBjWv/vE0vSAiEtirN37vxl8na -extld=gcc /home/iurt/.cache/go-build/3c/3ca69d193d6105bb30753889fdeeabd9d4730f14161e44dc46396646fca98c2a-d
$WORK/b001/zstd-1.4.5-patch1.test -test.timeout=10m0s
--- FAIL: TestStreamCompressionDecompressionParallel (0.00s)
    --- FAIL: TestStreamCompressionDecompressionParallel/#119 (0.01s)
        zstd_stream_test.go:115: Did not read the same �MA�.p != Hello World!
    --- FAIL: TestStreamCompressionDecompressionParallel/#159 (0.01s)
        zstd_stream_test.go:115: Did not read the same �MA�.p != Hello World!
    --- FAIL: TestStreamCompressionDecompressionParallel/#117 (0.02s)
        zstd_stream_test.go:115: Did not read the same �MA�.p != Hello World!
    --- FAIL: TestStreamCompressionDecompressionParallel/#163 (0.04s)
        zstd_stream_test.go:115: Did not read the same �MA�.p != Hello World!
FAIL
exit status 1
FAIL	_/home/iurt/rpmbuild/BUILD/zstd-1.4.5-patch1	5.307s
rm -r $WORK/b001/

Not sure if relevant, but the build machine has 32 cores.

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.12 windows/amd64

And gcc (i686-posix-dwarf-rev2, Built by MinGW-W64 project) 4.9.1

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Go\bin;C:\Go
set GOPROXY=
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=d:\Temp\go-build450666390=/tmp/go-build -gno-record-gcc-switches

What did you do?

go build

What did you expect to see?

What did you see instead?

github.com/DataDog/zstd

cc1.exe: sorry, unimplemented: 64-bit mode not compiled in
Compilation finished with exit code 2

Hi, thanks so much for releasing this binding.

I'm trying to track down a strange panic when this library is combined with some others. It's entirely possible that it has nothing to do with DataDog/zstd, but i can't reduce my test case any further.

Would you be able to comment on golang/go#19468 (comment) ?

Is there any chance the C code in this library is writing to memory it doesn't own?

In order for others to use the only currently supported vendor solution aka Go modules, this project needs a go.mod file.

This would also help with confusion some people have over the import name (DataDog vs datadog) - since the module file will contain the package name, it'll be the only proper import path after the module file gets merged.

Which one it should have is probably up for discussion (or as the author decides).

I am using
go version go1.12.14 windows/amd64

I was trying to decompress using the streaming api and getting the error:

zr := zstd.NewReader(bytes.NewBuffer(cbuf))
buf, err = ioutil.ReadAll(zr)
if err != nil {
return nil, err
}
err = zr.Close()
return

failed to decompress: Unknown frame descriptor

Reason behind that the service I am using to download data is adding some byte to make the block of fixed size.
Without padding it is working fine.
sample padding code:
n := len(itemList[i].comp) % 16
if n > 0 {
n = 16 - n
for ; n > 0; n-- {
itemList[i].comp = append(itemList[i].comp, padding[0])
}
}

I was expecting it will be able to ignore the extra bytes but that is not the case.
Can you please help me how to resolve this.

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.12.6

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=D:\goproject\bin
set GOCACHE=C:\Users\Administrator\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=D:\goproject
set GOPROXY=
set GORACE=
set GOROOT=D:\go\1.12.6
set GOTMPDIR=
set GOTOOLDIR=D:\go\1.12.6\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

What did you do?

write to an existing zstd file using io.writer

filename := "./test.zstd"
file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY, 0644)
_ = err
w := zstd.NewWriter(file)
w.Write([]byte("aaaaa"))

What did you expect to see?

Zstd can continue to compress the bytes written without affecting the decompression

What did you see instead?

When I use zstd.NewReader and ioutil.ReadAll, I get an error: failed to decompress: Corrupted block detected

It seems af19ae6 forgot to change the README:

All C files from Zstd have been removed from the distribution; instead the shared library is used. This version has been tested against zStd library version 1.0.

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.12.1 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\mfiles\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=e:\gopath\
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=e:\temp\wintemp\go-build152196250=/tmp/go-build -gno-record-gcc-switches

Package version: 809b919c325d7887bff7bd876162af73db53e878 (v.1.4.0 tag)

What did you do?

Fuzz test. Using ref, refErr := zstd.Decompress(nil, data).

Sample data: af046bc7b9294e2e85367bf8eb2bfa4de760bc06.zst.gz (gzipped to make github happy).

I got plenty of examples of this.

What did you expect to see?

An error. The commandline returns:

zstd.exe -d af046bc7b9294e2e85367bf8eb2bfa4de760bc06.zst
eb2bfa4de760bc06.zst : 0 MB...     eb2bfa4de760bc06.zst : Read error (39) : premature end

What did you see instead?

No error returned.

Win7 build cross-linux64
go build github.com/DataDog/zstd: no buildable Go source files in E:\go\test\src\github.com\DataDog\zstd

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.12.5 darwin/amd64

cc --version
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

What operating system and processor architecture are you using (go env)?

go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/sdomeshok/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/sdomeshok/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12.5/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/sdomeshok/Documents/Dev/terraform-provider-kafka/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/y5/8tn17qyd1mj08jlcfss9fvxc0000gq/T/go-build010092139=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Attempt to build an upstream that relies on zstd

What did you expect to see?

zstd successfully building.

What did you see instead?

# github.com/DataDog/zstd
In file included from ../../../go/pkg/mod/github.com/!data!dog/[email protected]/zstd.go:6:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
In file included from /usr/local/include/stdint.h:59:
/usr/local/include/stdint.h:2:10: error: #include nested too deeply
#include <stddef.h>
         ^
/usr/local/include/stdint.h:59:11: error: #include nested too deeply
# include <stdint.h>
          ^
/usr/local/include/stdint.h:72:11: error: #include nested too deeply
# include <sys/types.h>
          ^
/usr/local/include/stdint.h:76:10: error: #include nested too deeply
#include <limits.h>
         ^
/usr/local/include/stdint.h:82:11: error: #include nested too deeply
# include <inttypes.h>
          ^
5 errors generated.

Hi,

Our application stores and loads compressed data on disk using this library. We recently upgraded our application from 0727e17 (tag v1.3.0) to aebefd9 (tag v1.3.4).

After this library upgrade, there were some complaints from users. After generating some JSON data, compressing it, storing it, then later loading it and decompressing it, the data could no longer be parsed (e.g. json.Unmarshal: invalid character '\x00' in string literal).

The issue occurred on (at least) both Windows and macOS, with both Go 1.9.7 and 1.10.3.

I assume it was caused by memory corruption in CGO data buffers.

Reverting this library back to tag v1.3.0 seems to have completely resolved the issue going-forward.

We're not yet able to reproduce the issue, and not all our users were affected (perhaps it's related to OS memory pressure?), but, just a heads-up that this library upgrade was implicated. Once we have an internal reproducer we may be able to bisect it (or hopefully, blame something else and disregard this entire issue).

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.15.9 linux/amd64

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/xx/.cache/go-build"
GOENV="/home/xx/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/xx/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/xx/go"
GOPRIVATE=""
GOPROXY=""
GOROOT="/usr/lib/go-1.15"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.15/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build605241201=/tmp/go-build -gno-record-gcc-switches"

What did you do?

m wondering if we could export the ctx creation to the users so that they can make some customition, i have noticed the very early issue #17 and i did it myself by adding the following code to zstd_streaming.go, but maybe it's really not a good impl.

#cgo CPPFLAGS: -DZSTD_MULTITHREAD
#cgo CFLAGS: -pthread

func (w *Writer) SetNbWorkers(n int) error {
	if w.firstError != nil {
		return w.firstError
	}
	if err := getError(int(C.ZSTD_CCtx_setParameter(w.ctx, C.ZSTD_c_nbWorkers, C.int(n)))); err != nil {
		w.firstError = err
		return err
	}
	return nil
}

What did you expect to see?

compress&decompress ctx creation & customition avaliable

What did you see instead?

ctx created by the lib internally and it's not accessable outside

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.8.1

What operating system and processor architecture are you using (go env)?

darwin amd64

What did you do?

I used zstd.Writer() to compress very small chunks (a few bytes) at a time.

This is a reproduction case:

package main

import (
	"github.com/datadog/zstd"
	"os"
)

func main() {
	input, _ := os.Open("input")
	output, _ := os.Create("output")
	z := zstd.NewWriter(output)
	for {
		var tmp [1]byte
		_, err := input.Read(tmp[:])
		if err != nil {
			break
		}
		z.Write(tmp[:])
	}
	z.Close()
}

What did you expect to see?

Compression.

What did you see instead?

ZSTD is very slow (several times worse than any variant of compressing or not compressing) and balloons the file size (several times larger than the input) when used in this manner. I get files that are several times larger and take several times longer to compress.

I don't think it makes sense to provide an io.Writer without buffering into a minimum chunk size, as done by snappy with their BufferedWriter:

https://github.com/golang/snappy/blob/master/encode.go#L126

The following zstd functions are not currently wrapped or exposed:

• Simple dictionary API

  • ZSTD_compress_usingDict
  • ZSTD_decompress_usingDict

• Bulk processing dictionary API

  • ZSTD_createCDict
  • ZSTD_compress_usingCDict
  • ZSTD_createDDict
  • ZSTD_decompress_usingDDict

What version of Go are you using (go version)?

$ go version
go version go1.9.4 darwin/amd64

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/bobrik/cf-repos/els"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.9.4/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.9.4/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/xv/lkn605ns2c7fy9vtkz49dhcr0000gn/T/go-build106115125=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

$ GOOS=linux go install -v github.com/DataDog/zstd

What did you expect to see?

Succesful compilation with no errors

What did you see instead?

github.com/DataDog/zstd
go build github.com/DataDog/zstd: build constraints exclude all Go files in /tmp/dd/src/github.com/DataDog/zstd

It works just fine if I compile on linux directly or on macos directly, but not with cross-compilation.

Suppose a zstd.reader reads from a source which emits (and flushes) 1 small zstd block, then keeps the connection open (a typical use case would be compressed instant messaging).

If the block is too small, then the zstd.reader will hang forever trying to read a minimum amount of data from its source before actually processing it and passing it to zstd.

This is due to zstd.reader.Read calling TryReadFull to fill its compression buffer of initial size ZSTD_DStreamInSize (which is ZSTD_BLOCKSIZE_MAX + ZSTD_blockHeaderSize, larger than any zstd block). TryReadFull will keep trying to read to fill that buffer without trying to pass what it got so far on each Read to zstd.

The way to fix this issue would be to not call TryReadFull but instead simply Read on the underlying reader, and pass the intermediary result to zstd to check if a block is ready, if not try again, until zstd can read a block.

There could be a slight performance impact to calling zstd multiple times (crossing the Go/C boundary) so we could make this behaviour dependent on some new "low-latency" flag or something if it proves to hurt performance too much.

I am facing the following issue. Its during decompression
For compression the 'Data' that i am pushing is json data from an api. Compression is working perfectly.
Can some one help me with this.

Compression

var rs = new stream.Readable({ objectMode: true });
var chunks = [];
rs.push(Data);
rs.push(null);
rs.pipe()
.pipe(compressStream())
.on('data', function(chunk) {
chunks.push(chunk);
})
.on('end',function(){
Buffer.concat(chunks);
});

Decompression

var ds = new stream.Readable();
ds.push(chunks.toString());
ds.push(null);
ds.pipe(decompressStream())
.pipe(process.stdout);

Error

events.js:141
throw er; // Unhandled 'error' event
^

Error: Unknown frame descriptor
at Error (native)

Has anyone tried the official 1.0 with the Go Bindings?

https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/

Hello,

Would you consider cutting a new version that includes all fixes residing in master (empty slice error, empty payload, etc)?

I am packaging Datadog/zstd in Debian (https://tracker.debian.org/pkg/golang-github-datadog-zstd) as part of https://github.com/Shopify/sarama dependencies and having a new version instead of backporting patches would be helpful.

Thank you,
Chris

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.12.1 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\mfiles\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=e:\gopath\
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=e:\temp\wintemp\go-build152196250=/tmp/go-build -gno-record-gcc-switches

Package version: 809b919c325d7887bff7bd876162af73db53e878 (v.1.4.0 tag)

What did you do?

Fuzz test. Using ref, refErr := zstd.Decompress(nil, data).

It seems like the decompressor will attempt to allocate whatever value is returned from ZSTD_getDecompressedSize, meaning you can crash the decoder with a very small payload. The value is also cast from uint64 to int allowing negative values, which will panic: runtime error: makeslice: len out of range. Note that int can also be 32 bits.

Sample: a43e4d2eb6da97e23c23964ee90a093262f69564.zst.gz (gzipped to make github happy).

What did you expect to see?

No crash. The decoder should fall back to streaming over a certain size.

What did you see instead?

runtime: VirtualAlloc of 52983526064128 bytes failed with errno=1455
fatal error: out of memory

runtime stack:
runtime.throw(0x5c33c3, 0xd)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/panic.go:617 +0x79
runtime.sysMap(0xc000400000, 0x303030400000, 0x6ab778)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/mem_windows.go:122 +0x138
runtime.(*mheap).sysAlloc(0x693a40, 0x303030304000, 0x693a50, 0x181818182)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/malloc.go:633 +0x1d4
runtime.(*mheap).grow(0x693a40, 0x181818182, 0x0)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/mheap.go:1232 +0x49
runtime.(*mheap).allocSpanLocked(0x693a40, 0x181818182, 0x6ab788, 0x203000000000000)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/mheap.go:1150 +0x3d1
runtime.(*mheap).alloc_m(0x693a40, 0x181818182, 0x303030300101, 0x10000)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/mheap.go:977 +0xd0
runtime.(*mheap).alloc.func1()
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/mheap.go:1048 +0x53
runtime.(*mheap).alloc(0x693a40, 0x181818182, 0x10101, 0xc00008dce8)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/mheap.go:1047 +0x91
runtime.largeAlloc(0x303030303030, 0x101, 0x1cadd8)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/malloc.go:1055 +0xa0
runtime.mallocgc.func1()
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/malloc.go:950 +0x4d
runtime.systemstack(0xff5fe0)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/asm_amd64.s:351 +0x6b
runtime.mstart()
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/proc.go:1153

goroutine 1 [running]:
runtime.systemstack_switch()
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/asm_amd64.s:311 fp=0xc00008dc68 sp=0xc00008dc60 pc=0x451b40
runtime.mallocgc(0x303030303030, 0x599e20, 0xc000004001, 0x4390000)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/malloc.go:949 +0x8b1 fp=0xc00008dd08 sp=0xc00008dc68 pc=0x40cb91
runtime.makeslice(0x599e20, 0x303030303030, 0x303030303030, 0x4ce327)
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/slice.go:49 +0x73 fp=0xc00008dd38 sp=0xc00008dd08 pc=0x43d853
github.com/DataDog/zstd.Decompress(0x0, 0x0, 0x0, 0x4390000, 0x12, 0x200000, 0x0, 0x0, 0x0, 0x0, ...)
	e:/temp/wintemp/go-fuzz-build588063767/gopath/src/github.com/DataDog/zstd/zstd.go:130 +0x44a fp=0xc00008de08 sp=0xc00008dd38 pc=0x4c3c9a
github.com/klauspost/compress/zstd.Fuzz(0x4390000, 0x12, 0x200000, 0x0)
	e:/gopath/src/github.com/klauspost/compress/zstd/fuzz_decompress.go:69 +0x2bb fp=0xc00008df10 sp=0xc00008de08 pc=0x4d541b
github.com/dvyukov/go-fuzz/go-fuzz-dep.Main(0x5cb4c8)
	e:/temp/wintemp/go-fuzz-build588063767/gopath/src/github.com/dvyukov/go-fuzz/go-fuzz-dep/main.go:54 +0xbd fp=0xc00008df80 sp=0xc00008df10 pc=0x4dacbd
main.main()
	e:/temp/wintemp/go-fuzz-build588063767/gopath/src/github.com/klauspost/compress/zstd/go.fuzz.main/main.go:10 +0x34 fp=0xc00008df98 sp=0xc00008df80 pc=0x4db554
runtime.main()
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/proc.go:200 +0x20c fp=0xc00008dfe0 sp=0xc00008df98 pc=0x42e4ac
runtime.goexit()
	e:/temp/wintemp/go-fuzz-build588063767/goroot/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0x453cf1

goroutine 4 [runnable]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000080000)
	e:/gopath/src/github.com/klauspost/compress/zstd/blockdec.go:186
created by github.com/klauspost/compress/zstd.newBlockDec
	e:/gopath/src/github.com/klauspost/compress/zstd/blockdec.go:105 +0x174
exit status 2

What did you do?

Tried to compress empty slice.

What did you expect to see?

Empty slice is compressed consistently with zstd:

$ touch empty

$ zstd empty
empty                :1300.00%   (     0 =>     13 bytes, empty.zst)

$ ls -la empty empty.zst
-rw-r--r--  1 bobrik  staff   0 Sep 30 17:09 empty
-rw-r--r--  1 bobrik  staff  13 Sep 30 17:09 empty.zst

$ cat empty.zst | hexdump -C
00000000  28 b5 2f fd 24 00 01 00  00 99 e9 d8 51           |(./.$.......Q|
0000000d

What did you see instead?

Error ErrEmptySlice (Bytes slice is empty) is generated: 6114686.

cc @Viq111

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.14.4 windows/amd64

What operating system and processor architecture are you using (go env)?

windows 10
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\dell\AppData\Local\go-build
set GOENV=C:\Users\dell\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\Go\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=D:\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=D:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\dell\AppData\Local\Temp\go-build673146550=/tmp/go-build -gno-record-gcc-switches

What did you do?

I tried to compile configtxgen in the following path:$GOPATH/src/github.com/hyperledger/fabric (master),using git bash.

What did you expect to see?

should've built successively...

What did you see instead?

The compiler gives me information like what is shown below:

github.com/hyperledger/fabric/cmd/configtxgen

D:\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:\Users\dell\AppData\Local\Temp\go-link-429055843\000004.o: In function COVER_buildDictionary': /cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/cover.c:714: undefined reference to __getreent'
/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/cover.c:714: undefined reference to __getreent' /cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/cover.c:710: undefined reference to __getreent'
/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/cover.c:710: undefined reference to __getreent' /cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/cover.c:678: undefined reference to __getreent'
C:\Users\dell\AppData\Local\Temp\go-link-429055843\000004.o:/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/cover.c:678: more undefined references to __getreent' follow C:\Users\dell\AppData\Local\Temp\go-link-429055843\000006.o: In function ss_mintrosort':
/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/divsufsort.c:509: undefined reference to __assert_func' /cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/divsufsort.c:513: undefined reference to __assert_func'
/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/divsufsort.c:512: undefined reference to __assert_func' /cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/divsufsort.c:508: undefined reference to __assert_func'
/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/divsufsort.c:517: undefined reference to __assert_func' C:\Users\dell\AppData\Local\Temp\go-link-429055843\000006.o:/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/divsufsort.c:516: more undefined references to __assert_func' follow
C:\Users\dell\AppData\Local\Temp\go-link-429055843\000009.o: In function FASTCOVER_ctx_init': /cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/fastcover.c:324: undefined reference to __getreent'
/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/fastcover.c:324: undefined reference to __getreent' /cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/fastcover.c:311: undefined reference to __getreent'
/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/fastcover.c:311: undefined reference to __getreent' /cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/fastcover.c:318: undefined reference to __getreent'
C:\Users\dell\AppData\Local\Temp\go-link-429055843\000009.o:/cygdrive/d/Go/go/src/github.com/hyperledger/fabric/vendor/github.com/DataDog/zstd/fastcover.c:318: more undefined references to `__getreent' follow
collect2.exe: error: ld returned 1 exit status

make: *** [Makefile:209: build/bin/configtxgen] Error 2

########################################
Thank you very much if you can help me solve this issue.I will really appreciate it.

If I make multiple calls to Write, the resulting compressed data stream length is much greater than buffering the data and making a single call to Write.

— Is this expected behaviour? The reason I ask is that if chained with other writers, one cannot make any guarantees as to how they may parcel up their data.

Code (error handling omitted):

func main() {

	b := &bytes.Buffer{}
	for i := 0; i < 500; i++ {
		b.Write([]byte("Hello World! "))
	}
	data1 := b.Bytes()
	fmt.Println("data len", len(data1))

	// Compress 1
	buffer1 := &bytes.Buffer{}
	w1 := zstd.NewWriterLevel(buffer1, CompressionLevel)
	w1.Write(data1)
	w1.Close()

	fmt.Println("buffer1 len", buffer1.Len())

	// Compress 2
	buffer2 := &bytes.Buffer{}
	w2 := zstd.NewWriterLevel(buffer2, CompressionLevel)
	for i := 0; i < 500; i++ {
		w2.Write([]byte("Hello World! "))
	}
	w2.Close()

	fmt.Println("buffer2 len", buffer2.Len())
}

Output:

data len 6500
buffer1 len 33
buffer2 len 5014

Regards

zstd improved the multi-threading code present in pzstd in 1.1.0 and moved it up to the main library. I'd love to be able to use multiple threads - at least when reading zstd compressed streams.

Thanks in advance!

What version of Go are you using (go version)?

go version go1.8.1 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/count"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.8.1/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.8.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/qx/x2xsgzgx5y90lmwwkhs3w91r0000gn/T/go-build691925268=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"