Giter Club home page Giter Club logo

base62's Introduction

Lytics Command Line Tool & Developers Aid

The goal of this tool is to provide CLI access to the Lytics API. It also functions as a developers aid to enable writing and testing LQL (Lytics Query Language) as easily as possible.

We would love any feature requests or ideas that would make this useful to you.

Installation

Download a binary from the releases page and rename to lytics:

# linux/amd64
curl -Lo lytics https://github.com/lytics/lytics/releases/download/latest/lytics_linux \
  && chmod +x lytics \
  && sudo mv lytics /usr/local/bin/

# darwin/amd64
curl -Lo lytics https://github.com/lytics/lytics/releases/download/latest/lytics_mac \
  && chmod +x lytics \
  && sudo mv lytics /usr/local/bin/

Or install from source:

git clone https://github.com/lytics/lytics.git
go build
go install

Or install from the repository via go:

go get -u github.com/lytics/lytics

Usage

All examples use JQ to prettify the JSON output.

export LIOKEY="your_api_key"
lytics --help

Segment Scan Usage

Exporting CSV files, with usage.

Example

# Scan a segment by id
lytics segment scan ab93a9801a72871d689342556b0de2e9 | jq '.'

# Scan a segment by slug
lytics segment scan last_2_hours | jq '.'

# write out this segment to a temp file so we can play with JQ
lytics segment scan last_2_hours > /tmp/users.json

# same thing but with an "ad hoc query"
lytics segment scan '
FILTER AND (
    lastvisit_ts > "now-2d"
    EXISTS email
)
FROM user
' > /tmp/users.json

# use JQ to output a few fields
cat /tmp/users.json | \
 jq -c ' {country: .country, city: .city, org: .org, uid: ._uid, visitct: .visitct} '

# create a CSV file from these users
echo "country,city,org,uid,visitct\n" > /tmp/users.csv
cat /tmp/users.json | \
 jq -r ' [ .country, .city, .org,  ._uid, .visitct ] | @csv ' >> /tmp/users.csv

Lytics Watch Usage

  1. Create NAME.lql (any name) file in a folder.
  2. Assuming you already have data collected, it will use our API to show recent examples against that LQL.

You can open and edit in an editor. Every time you edit it will print resulting users it interpreted from recent data to our API.

Example

# get your API key from the web app account settings screen
export LIOKEY="your_api_key"

cd /path/to/your/project

# create an LQL file
# - utilize the Lytics app "Data -> Data Streams" section to see
#   data fields you are sending to Lytics.

# you can create this in an editor as well
echo '
SELECT
   user_id,
   name,
   todate(ts),
   match("user.") AS user_attributes,
   map(event, todate(ts))   as event_times   KIND map[string]time  MERGEOP LATEST

FROM default
INTO USER
BY user_id
ALIAS my_query
' > default.lql


# start watching
lytics schema queries watch .

# now edit JSON results of how data is interpreted is output

Lytics Watch With Custom Data

  1. Create NAME.lql (any name) file in a folder.
  2. Create NAME.json (any name, must match LQL file name) in folder.
  3. Run the lytics watch command from the folder with files.
  4. Edit .lql, or .json files, upon change the evaluated result of the .lql, JSON will be output.

Example

# get your API key from web app account settings
export LIOKEY="your_api_key"

cd /tmp

# start watching in background
lytics schema queries watch &

# create an LQL file
echo '
SELECT
   user_id,
   name,
   todate(ts),
   match("user.") AS user_attributes,
   map(event, todate(ts))   as event_times   KIND map[string]time  MERGEOP LATEST

FROM data
INTO USER
BY user_id
ALIAS hello
' > hello.lql

# Create an array of JSON events to feed into LQL query
echo '[
    {"user_id":"dump123","name":"Down With","company":"Trump", "event":"project.create", "ts":"2016-11-09"},
    {"user_id":"another234","name":"No More","company":"Trump", "event":"project.signup","user.city":"Portland","user.state":"Or", "ts":"2016-11-09"}
]' > hello.json

SegmentML example

# replace {your model name here} with target_audience::source_audience

# generates tables
lytics segmentml --output all {your model name here}
lytics segmentml --output features {your model name here}
lytics segmentml --output predictions {your model name here}
lytics segmentml --output overview {your model name here}

# for CSV output
lytics --format csv segmentml --output all {your model name here}

# for JSON
lytics --format json segmentml --output all {your model name here}

base62's People

Contributors

araddon avatar jerrodrurik avatar markhayden avatar tuxxy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

base62's Issues

tests are failing, things are broken

Hey lytics. Nice project you have here. I'm using it in something for work. Unfortunately the latest merge here has broken tests. If you run go test . at this projects root you will see that the latest commit broke things with the ? string somehow.

commit 2c06157bd55d0d09bb15e7648606e065a8a5079f has passing tests, so I am pinning to that commit for the time being.

the latest commit: 6146185d8a0363f33780de1629e292efcd12d2cd doesn't have passing tests, and in my project's tests that depend on this, it runs into out of bounds errors

You should probably look into automating the CI, so that you don't break your test :)

use DecodeString panic

base62 version:v0.0.0-20180808010106-0ee4de5a5d6d

base62.StdEncoding.DecodeString("sY1XtY6IMcuzmVs0c4h79XBSB7qVEtSt.WdkJg")

panic: runtime error: index out of range [27] with length 27 [recovered]
panic: runtime error: index out of range [27] with length 27

This is NOT base62. Does not agree with GMP / GnuPG / Saltpack packages

Reference Strings

For reference, I believe this is what base62 encoding and decoding should look like:

Raw   : "Hello, 世界"
Base64: SGVsbG8sIOS4lueVjA (18 chars)
Base62: 1wJfrzvdbuFbL65vcS (18 chars)

Raw   : "Hello World"
Base64: SGVsbG8gV29ybGQ (15 chars)
Base62: 73XpUgyMwkGr29M (15 chars)

Raw   : [ 0, 0, 0, 0, 255, 255, 255, 255 ]
Base64: AAAAAP____8 (11 chars)
Base62: 000004gfFC3 (11 chars)

Raw   : [ 255, 255, 255, 255, 0, 0, 0, 0 ]
Base64: _____wAAAAA (11 chars)
Base62: LygHZwPV2MC (11 chars)

Reference Implementation

I generated that using https://github.com/keybase/saltpack/encoding/basex, which seems to be correct and agree with other implementations:

package main

import (
	"encoding/base64"
	"fmt"

	"github.com/keybase/saltpack/encoding/basex"
)

func main() {
	for _, src := range [][]byte{
		[]byte("Hello, 世界"),
		[]byte("Hello World"),
		{0, 0, 0, 0, 255, 255, 255, 255},
		{255, 255, 255, 255, 0, 0, 0, 0},
	} {
		// Uses the GMP character set "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
		b62 := basex.Base62StdEncoding.EncodeToString(src)
		b64 := base64.RawURLEncoding.EncodeToString(src)

		fmt.Printf("Base64: %s (%d chars)\n", b64, len(b64))
		fmt.Printf("Base62: %s (%d chars)\n", b62, len(b62))
		fmt.Println("")
	}
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.