sv / kdbgo Goto Github PK

Im trying to use UnmarshalDict any i got it to work with all fields but a list of char

My table structure

c          | t f a
-----------| -----
id         | s    
dispname   | C    
order      | i    
update_time| z    

My Struct for this table

type Linkgroup struct {
	Id          string    `json:"id"`
	Dispname    string    `json:"dispname"`
	Order       int32     `json:"order"`
	Update_time time.Time `json:"update_time"`
}

i do a db.Call("0!select from linkgroups")
Then i do the following

tbl := t.Data.(kdb.Table)
dict := tbl.Index(2)
var data2 models.Linkgroup
err := kdb.UnmarshalDict(dict, data2)
fmt.Println(data2)

All the fields work except Dispname.

Thank you for any help, hopefully you still look at this repo

Alternatives: https://gof.rs/projects/uuid/ or https://github.com/google/uuid

reason: package is not maintained/versioned and has some issues nu7hatch/gouuid#26

I couldn't find a description how the project is licensed. It would be good to provide this information.

Some of the v0 api is not very familiar to regular users of net package and can be made more modular.

v1beta branch uses more straightforward Conn and Dial for managing connections

Here is my code, please check it out.

package main

import (
	"bytes"
	"fmt"
	"log"
	"os"
	"os/exec"
	"strconv"

	kdb "github.com/sv/kdbgo"
)

var testPort = 0

func StartQ(ready chan struct{}) {
	fmt.Println("Starting q process on random port")
	_, err := exec.LookPath("q")
	if err != nil {
		log.Fatal("installing q is in your future")
	}
	cmd := exec.Command("q", "-p", "0W", "-q")
	stdin, err := cmd.StdinPipe()
	if err != nil {
		log.Fatal("Failed to connect stdin", err)
	}
	stdin.Write([]byte(".z.pi:.z.pg:.z.ps:{value 0N!x};system\"p\"\n"))
	stdout, err := cmd.StdoutPipe()
	if err != nil {
		log.Fatal("Failed to connect stdout", err)
	}
	stderr, err := cmd.StderrPipe()
	if err != nil {
		log.Fatal("Failed to connect stderr", err)
	}
	err = cmd.Start()
	if err != nil {
		log.Fatal("Failed to start q", err)
	}
	buf := make([]byte, 16)
	n, err := stdout.Read(buf)
	if err != nil {
		log.Fatal("Failed to fill port number for child process: ", err)
	}
	testPort, err = strconv.Atoi(string(buf[:bytes.IndexByte(buf, 'i')]))
	if err != nil {
		fmt.Println("Failed to setup listening port", string(buf[:n]), err)
		cmd.Process.Kill()
		os.Exit(1)
	}
	fmt.Println("Listening port is ", testPort)
	go func() {
		for {
			buf := make([]byte, 256)
			n, err := stderr.Read(buf)
			fmt.Println("Q stderr output:", string(buf[:n]))
			if err != nil {
				fmt.Println("Q stderr error:", err)
				return
			}
		}
	}()
	go func() {
		for {
			buf := make([]byte, 256)
			_, err := stdout.Read(buf)
			//fmt.Println("Q stdout output:", string(buf[:n]))
			if err != nil {
				fmt.Println("Q stdout error:", err)
				return
			}
		}
	}()
	ready <- struct{}{}
	cmd.Wait()
	//stdin.Write([]byte("show `exiting_process;\nexit 0\n"))
}

func main() {
	ready := make(chan struct{})
	go StartQ(ready)

	<-ready
	log.Println("Connect to kdb+ through unix socket.")
	con, err := kdb.DialUnix("localhost", testPort, "")
	if err != nil {
		panic(err)
	}

	log.Println("Close the connection")
	err = con.Close()
	if err != nil {
		panic(err)
	}
}

In https://github.com/sv/kdbgo/blob/master/kdb.go#L208 the port, which is an int value is casted to a string by string(port).
This is not correct.
Consider using strconv.Itoa(port) or fmt.Sprintf.

I am thinking of addressing the installing q is in your future to install Q on Travis CI since I believe some of the tests are failing when run with Q installed on my machine

I was thinking we could check in the 32 bit binary of Q into the repository and run that. Do you have any approach in mind?

With 64-bit personal edition installed, "go test -v" failed at line 47 in kdb_test.go.
Error message:

panic: runtime error: slice bounds out of range

goroutine 1 [running]:
github.com/sv/kdbgo.TestMain(0xc0000bc200)
  /home/even/Experiments/kdb/kdbgo/kdb_test.go:47 +0x813
main.main()
  _testmain.go:80 +0x13e
exit status 2

panic: runtime error: slice bounds out of range

goroutine 162 [running]:
bufio.(*Reader).Read(0xc4202181e0, 0xc4216b5c88, 0x8, 0x8, 0x8, 0x16c96e0, 0x0)
/usr/local/go/src/bufio/bufio.go:227 +0x3bf
io.ReadAtLeast(0x1888be0, 0xc4202181e0, 0xc4216b5c88, 0x8, 0x8, 0x8, 0x16c9760, 0x1767701, 0xc4216b5c88)
/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x1888be0, 0xc4202181e0, 0xc4216b5c88, 0x8, 0x8, 0x8, 0xc420036f00, 0xc420070218)
/usr/local/go/src/io/io.go:327 +0x58
encoding/binary.Read(0x1888be0, 0xc4202181e0, 0x1890300, 0x1cb5b00, 0x1708000, 0xc4216b5348, 0xc420704380, 0xc4216efad0)
/usr/local/go/src/encoding/binary/binary.go:245 +0xf48
github.com/sv/kdbgo.Decode(0xc4202181e0, 0xc420222028, 0x1, 0xc421213f40, 0x0)
/github.com/sv/kdbgo/decode.go:141 +0x95
github.com/sv/kdbgo.(*KDBConn).Call(0xc4201e2230, 0xc424473dc0, 0x3d, 0x0, 0x0, 0x0, 0xc42000e018, 0xc420e1bed8, 0x1)
/github.com/sv/kdbgo/kdb.go:85 +0x179

I just started looking into this as I've been reaching a segmentation fault in kdb+ when passing strings over IPC (via AsyncCall, Call, and WriteMessage). The issue seems to lie in the Compress function which appears to truncate some input objects. When I remove compression from the Encode function, the object is sent as expected. I will upload more details later, but when I try passing several strings over IPC, each greater than 800 characters, I receive the seg fault in kdb+.

Can Decode also be used to decode data from disk, or is this a different format?

Hi, we are trying to make multiple select queries in parallel to retrieve data from kdb.

query:- select open:first price, high:max price, low:min price, close:last price, volume:sum volume by sym from trade where time within time1 time 2

we are encountering an error : "Slice bounds out of range" from Decode function.

panic: slice bounds out of range
goroutine 106 [running]:
bufio.(*Reader).Read(0xc000049380, 0xc00112d328, 0x8, 0x8, 0xc00f669c08, 0x441cd3, 0x8)
C:/Go/src/bufio/bufio.go:234 +0x3ce
io.ReadAtLeast(0xe339e0, 0xc000049380, 0xc00112d328, 0x8, 0x8, 0x8, 0x5, 0xc000dc7560, 0x0)
C:/Go/src/io/io.go:310 +0x8f
io.ReadFull(...)
C:/Go/src/io/io.go:329
encoding/binary.Read(0xe339e0, 0xc000049380, 0xe4a140, 0x1487ec0, 0xbd22c0, 0xc00112d2f8, 0x0, 0xc00f669e28)
github.com/sv/kdbgo.Decode(0xc000049380, 0xc00011c270, 0xc00011c201, 0xc000621120, 0x0)

This error occurs only when there is huge data for processing.
We don't see this error when there is only one select query.
Could some one please look into this ? and recommend a fix.

Environment
kdb windows 32bit
go version go1.12.7 windows/amd64
OS: windows

Hey, I noticed that the function UnmarshalDict have some behaviour around field names that may be undesirable (certainly is for my usecase).

The main issue is that when it looks up a fieldname it will make the first character uppercase if necessary, but no further changes. That means that if a dict field uses snake case, like this:
this_is_a_field the corresponding field in a struct would have to be:

type MyStruct struct {
    This_is_a_field int
}

Which both brake go conventions and is also confusing.
For my own purpose I rewrote the function the following way:

func unmarshalResponse(t kdb.Dict) (*OrderResponse, error) {              
    var (
        keys = t.Key.Data.([]string)
        vals = t.Value.Data.([]*kdb.K)                                    
        v    = new(OrderResponse)                                         
    )
    vv := reflect.Indirect(reflect.ValueOf(v))                            
    for i := range keys {
        val := vals[i].Data
        fv := vv.FieldByName(strcase.ToCamel(keys[i]))                    
        if !fv.IsValid() {
            return nil, fmt.Errorf("field does not exist %q", keys[i])    
        }
        if fv.CanSet() && reflect.TypeOf(val).AssignableTo(fv.Type()) {   
            fv.Set(reflect.ValueOf(val))                                  
        }                                                                 
    }
    return v, nil                                                         
}                                                                         

It uses a thirdparty library to convert the name to CamelCase, but that could be done locally, I was just being lazy.
(an added benefit with this version is that it will error out if a correct struct field can not be found).

In addition it might be beneficial to support tags as an alternative to strict field naming similar to how the json marshalling works, so it would be possible to do something along these lines.

type MyStruct struct {
    Foo int `kdb:"this_is_a_field"`
}

I'm happy to write up a PR if you think this would be beneficial.

I'm also working on functionality to Marshal structs and maps into Dicts that might be useful as well.

time:= &kdb.K{kdb.KN, kdb.NONE, []int64{t2.Nanoseconds()}}
when I use time.Duration to upsert KDB I got the
ERROR: logging before flag.Parse: E0223 11:52:50.294900 13024 decode.go:129] Failed to read message header: EOF