segmentio / asm Goto Github PK
View Code? Open in Web Editor NEWGo library providing algorithms optimized to leverage the characteristics of modern CPUs
License: MIT No Attribution
Go library providing algorithms optimized to leverage the characteristics of modern CPUs
License: MIT No Attribution
Maybe replace the < 32 bytes implementation to avoid the double scan + two functions call, if it happens to be an issue with production data.
While running a test suite on Go 1.17.3 on an M1 Mac with github.com/segmentio/encoding v0.3.2 and github.com/segmentio/asm v1.1.0, I ran into segmentio/encoding#84 again. I believe the fix in segmentio/encoding#85 was incomplete and may not apply to non-Intel CPUs, or maybe the rules are slightly different in newer Go and/or on ARM.
The same test code triggers it:
package main
import (
"testing"
"github.com/segmentio/encoding/json"
)
type Foo struct {
Source struct {
Table string
}
}
func TestUnmarshal(t *testing.T) {
input := []byte(`{"source": {"table": "1234567"}}`)
r := &Foo{}
json.Unmarshal(input, r)
}
Run the same way:
go mod init segtest
go mod tidy
go test -v -race -trimpath ./...
And the results are:
=== RUN TestUnmarshal
fatal error: checkptr: pointer arithmetic result points to invalid allocation
goroutine 35 [running]:
runtime.throw({0x1010ceb85, 0x40})
runtime/panic.go:1198 +0x54 fp=0xc00004cb00 sp=0xc00004cad0 pc=0x100f81794
runtime.checkptrArithmetic(0xc00016a040, {0xc00004cb90, 0x1, 0x1})
runtime/checkptr.go:69 +0xbc fp=0xc00004cb30 sp=0xc00004cb00 pc=0x100f5106c
github.com/segmentio/asm/ascii.ValidPrintString({0xc00016a020, 0x20})
github.com/segmentio/[email protected]/ascii/valid_print_default.go:16 +0x80 fp=0xc00004cba0 sp=0xc00004cb30 pc=0x101092be0
github.com/segmentio/asm/ascii.ValidPrint(...)
github.com/segmentio/[email protected]/ascii/valid_print.go:7
github.com/segmentio/encoding/ascii.ValidPrint(...)
github.com/segmentio/[email protected]/ascii/valid_print.go:10
github.com/segmentio/encoding/json.internalParseFlags({0xc00016a020, 0x20, 0x20})
github.com/segmentio/[email protected]/json/parse.go:33 +0x148 fp=0xc00004cc50 sp=0xc00004cba0 pc=0x1010b9788
github.com/segmentio/encoding/json.Parse({0xc00016a020, 0x20, 0x20}, {0x101113200, 0xc000112530}, 0x0)
github.com/segmentio/[email protected]/json/json.go:303 +0xc8 fp=0xc00004cdb0 sp=0xc00004cc50 pc=0x1010b91c8
github.com/segmentio/encoding/json.Unmarshal({0xc00016a020, 0x20, 0x20}, {0x101113200, 0xc000112530})
github.com/segmentio/[email protected]/json/json.go:285 +0x58 fp=0xc00004ce60 sp=0xc00004cdb0 pc=0x1010b8ff8
segtest.TestUnmarshal(0xc000107520)
segtest/seg_test.go:18 +0xe4 fp=0xc00004cec0 sp=0xc00004ce60 pc=0x1010c3584
testing.tRunner(0xc000107520, 0x1011405f0)
testing/testing.go:1259 +0x19c fp=0xc00004cfc0 sp=0xc00004cec0 pc=0x101035e1c
runtime.goexit()
runtime/asm_arm64.s:1133 +0x4 fp=0xc00004cfc0 sp=0xc00004cfc0 pc=0x100fb59f4
created by testing.(*T).Run
testing/testing.go:1306 +0x5bc
I've tried to run on an Intel CPU with -tags purego
, but it won't compile due to duplicated function definitions. Perhaps add a build !purego to the generated _amd64.go files to get this to work and add go test -race -tags purego
to your test suite?
Thanks!
-- Aaron
Hi folks,
This library is accelerated using assembly. Have you made any performance comparison of using cgo for corresponding acceleration? Although cgo itself has a remarkable overhead around 100ns for each call, it might be negligible given larger batch inputs. On the other hand, assembly itself also has the shortcomings such that it could not be inlined, so it would be interesting if performance comparison is available. Thank you~
Some of the implementations in this repository are drop-in replacement for standard Go library functions. Contributing them back upstream, to golang/go, could benefit the ecosystem as a whole with little drawback to Twilio Segment, as segmentio/asm is already released under the permissive MIT license. However, we did not anticipate the attribution requirement of this license to prevent its adoption in the standard library (example: utf8.Valid
CL). It is a non-starter to include derivatives of MIT-licensed code in the Go standard library, because it would force all people who distribute Go program binaries to include the license and copyright assignment to Segment.
As a result, would you consider re-licensing the code in this repository to permit its integration into Go without attribution?
For example: MIT-0 ("MIT No Attribution License").
Thank you!
When segmentio/asm v1.2.0
is vendored into a module, building a Docker image fails.
build layer in Dockerfile:
RUN CGO_ENABLED=1 GO111MODULE=off go install /cmd/main.go
error logs:
[2022-12-06 16:56:34] INFO docker build: #10 37.51 # github.com/segmentio/<REPO>/cmd/<BINARY>
| [2022-12-06 16:56:34] INFO docker build: #10 37.51 /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
| [2022-12-06 16:56:34] INFO docker build: #10 37.51 /usr/bin/ld: /tmp/go-link-506352844/go.o: in function `github.com/segmentio/<REPO>/vendor/github.com/segmentio/asm/bswap.swap64.abi0':
| [2022-12-06 16:56:34] INFO docker build: #10 37.51 /go/src/github.com/segmentio/<REPO>/vendor/github.com/segmentio/asm/bswap/swap64_amd64.s:14: undefined reference to `github.com/segmentio/asm/cpu.X86'
| [2022-12-06 16:56:34] INFO docker build: #10 37.51 collect2: error: ld returned 1 exit status
| [2022-12-06 16:56:34] INFO docker build: #10 37.51
| [2022-12-06 16:56:35] INFO docker build: #10 37.83 # github.com/segmentio/<REPO>/cmd/<BINARY>
| [2022-12-06 16:56:35] INFO docker build: #10 37.83 /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
| [2022-12-06 16:56:35] INFO docker build: #10 37.83 /usr/bin/ld: /tmp/go-link-65482911/go.o: in function `github.com/segmentio/<REPO>/vendor/github.com/segmentio/asm/bswap.swap64.abi0':
| [2022-12-06 16:56:35] INFO docker build: #10 37.83 /go/src/github.com/segmentio/<REPO>/vendor/github.com/segmentio/asm/bswap/swap64_amd64.s:14: undefined reference to `github.com/segmentio/asm/cpu.X86'
| [2022-12-06 16:56:35] INFO docker build: #10 37.83 collect2: error: ld returned 1 exit status
| [2022-12-06 16:56:35] INFO docker build: #10 37.83
It seems like this is the line of code within segmentio/asm that's missing a reference but I'm unsure how to fix this.
Playground: https://play.golang.org/p/QlHEKjlenGp
Result:
it looks like random error, sometimes there is no mistake.
Apologies if this isn't the right place to ask this question, but could find any other means.
Noticed ARM64 being worked on, does that imply there might be support for Apple Silicon in the future?
Apple Silicon deviates from ARM64 by supporting W^X. Don't know if it's applicable to your algorithms though.
Prefer x/sys/cpu over the current asm/cpu
package. It also has the advantage of taking into account whether an OS supports XMM and YMM registers.
go.mod file hasn't require github.com/mmcloughlin/avo v0.x.0
The bits.CountByte
function in https://github.com/segmentio/parquet-go is a good candidate to bring to https://github.com/segmentio/asm
Here is a suggestion for the signature:
package mem
func Count(b []byte, v byte) int
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.