Transform
Transform is a Go package that provides a simple pattern for performing chainable data transformations on streams of bytes. It conforms to the io.Reader interface and is useful for operations such as converting data formats, audio/video resampling, image transforms, log filters, regex line matching, etc.
The transutil package provides few examples that work with JSON such as JSONToMsgPack, MsgPackToJSON, JSONToPrettyJSON, JSONToUglyJSON, JSONToProtoBuf, and ProtoBufToJSON. It also includes a handy Gzipper and Gunzipper.
Getting Started
Installing
To start using Transform, install Go and run go get:
$ go get -u github.com/tidwall/transformUsing
Below are a few very simple examples of custom transformers.
ToUpper
Convert a string to uppper case. Unicode aware. In this example we only process one rune at a time.
func ToUpper(r io.Reader) io.Reader {
br := bufio.NewReader(r)
return transform.NewTransformer(func() ([]byte, error) {
c, _, err := br.ReadRune()
if err != nil {
return nil, err
}
return []byte(strings.ToUpper(string([]rune{c}))), nil
})
}msg := "Hello World"
data, err := ioutil.ReadAll(ToUpper(bytes.NewBufferString(msg)))
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))Output:
HELLO WORLD
Rot13
The Rot13 cipher.
func Rot13(r io.Reader) io.Reader {
buf := make([]byte, 256)
return transform.NewTransformer(func() ([]byte, error) {
n, err := r.Read(buf)
if err != nil {
return nil, err
}
for i := 0; i < n; i++ {
if buf[i] >= 'a' && buf[i] <= 'z' {
buf[i] = ((buf[i] - 'a' + 13) % 26) + 'a'
} else if buf[i] >= 'A' && buf[i] <= 'Z' {
buf[i] = ((buf[i] - 'A' + 13) % 26) + 'A'
}
}
return buf[:n], nil
})
}msg := "Hello World"
data, err := ioutil.ReadAll(Rot13(bytes.NewBufferString(msg)))
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))Output:
Uryyb Jbeyq
RegExp Line Matcher
A line reader that filters lines that match on a RegExp pattern.
func LineMatch(r io.Reader, pattern string) io.Reader {
br := bufio.NewReader(r)
return NewTransformer(func() ([]byte, error) {
for {
line, err := br.ReadBytes('\n')
matched, _ := regexp.Match(pattern, line)
if matched {
return line, err
}
if err != nil {
return nil, err
}
}
})
}logs := `
23 Apr 17:32:23.604 [INFO] DB loaded in 0.551 seconds
23 Apr 17:32:23.605 [WARN] Disk space is low
23 Apr 17:32:23.054 [INFO] Server started on port 7812
23 Apr 17:32:23.141 [INFO] Ready for connections
`
data, err := ioutil.ReadAll(LineMatch(bytes.NewBufferString(logs), "WARN"))
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))Output:
23 Apr 17:32:23.605 [WARN] Disk space is low
LineTrimSpace
A line reader that trims the spaces from all lines.
func LineTrimSpace(r io.Reader, pattern string) io.Reader {
br := bufio.NewReader(r)
return transform.NewTransformer(func() ([]byte, error) {
for {
line, err := br.ReadBytes('\n')
if len(line) > 0 {
line = append(bytes.TrimSpace(line), '\n')
}
return line, err
}
})
}phrases := " lacy timber \n"
phrases += "\t\thybrid gossiping\t\n"
phrases += " coy radioactivity\n"
phrases += "rocky arrow \n"
out, err := ioutil.ReadAll(LineTrimSpace(bytes.NewBufferString(phrases)))
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)Output:
lacy timber
hybrid gossiping
coy radioactivity
rocky arrow
Chaining
A reader that matches lines on the letter 'o', trims the space from the lines, and transforms everything to upper case.
phrases := " lacy timber \n"
phrases += "\t\thybrid gossiping\t\n"
phrases += " coy radioactivity\n"
phrases += "rocky arrow \n"
r := ToUpper(LineTrimSpace(LineMatch(bytes.NewBufferString(phrases), "o")))
// Pass the string though the transformer.
out, err := ioutil.ReadAll(r)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)Output:
HYBRID GOSSIPING
COY RADIOACTIVITY
ROCKY ARROW
Transutil package
The github.com/tidwall/transform/transutil package includes additional examples.
func Gunzipper(r io.Reader) io.Reader
func Gzipper(r io.Reader) io.Reader
func JSONToMsgPack(r io.Reader) io.Reader
func JSONToPrettyJSON(r io.Reader) io.Reader
func JSONToProtoBuf(r io.Reader, pb proto.Message, multimessage bool) io.Reader
func JSONToUglyJSON(r io.Reader) io.Reader
func MsgPackToJSON(r io.Reader) io.Reader
func ProtoBufToJSON(r io.Reader, pb proto.Message, multimessage bool) io.Reader
Contact
Josh Baker @tidwall
License
Transform source code is available under the ISC License.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent