Giter Club home page Giter Club logo

analyst's Introduction

Lytics Command Line Tool & Developers Aid

The goal of this tool is to provide CLI access to the Lytics API. It also functions as a developers aid to enable writing and testing LQL (Lytics Query Language) as easily as possible.

We would love any feature requests or ideas that would make this useful to you.

Installation

Download a binary from the releases page and rename to lytics:

# linux/amd64
curl -Lo lytics https://github.com/lytics/lytics/releases/download/latest/lytics_linux \
  && chmod +x lytics \
  && sudo mv lytics /usr/local/bin/

# darwin/amd64
curl -Lo lytics https://github.com/lytics/lytics/releases/download/latest/lytics_mac \
  && chmod +x lytics \
  && sudo mv lytics /usr/local/bin/

Or install from source:

git clone https://github.com/lytics/lytics.git
go build
go install

Or install from the repository via go:

go get -u github.com/lytics/lytics

Usage

All examples use JQ to prettify the JSON output.

export LIOKEY="your_api_key"
lytics --help

Segment Scan Usage

Exporting CSV files, with usage.

Example

# Scan a segment by id
lytics segment scan ab93a9801a72871d689342556b0de2e9 | jq '.'

# Scan a segment by slug
lytics segment scan last_2_hours | jq '.'

# write out this segment to a temp file so we can play with JQ
lytics segment scan last_2_hours > /tmp/users.json

# same thing but with an "ad hoc query"
lytics segment scan '
FILTER AND (
    lastvisit_ts > "now-2d"
    EXISTS email
)
FROM user
' > /tmp/users.json

# use JQ to output a few fields
cat /tmp/users.json | \
 jq -c ' {country: .country, city: .city, org: .org, uid: ._uid, visitct: .visitct} '

# create a CSV file from these users
echo "country,city,org,uid,visitct\n" > /tmp/users.csv
cat /tmp/users.json | \
 jq -r ' [ .country, .city, .org,  ._uid, .visitct ] | @csv ' >> /tmp/users.csv

Lytics Watch Usage

  1. Create NAME.lql (any name) file in a folder.
  2. Assuming you already have data collected, it will use our API to show recent examples against that LQL.

You can open and edit in an editor. Every time you edit it will print resulting users it interpreted from recent data to our API.

Example

# get your API key from the web app account settings screen
export LIOKEY="your_api_key"

cd /path/to/your/project

# create an LQL file
# - utilize the Lytics app "Data -> Data Streams" section to see
#   data fields you are sending to Lytics.

# you can create this in an editor as well
echo '
SELECT
   user_id,
   name,
   todate(ts),
   match("user.") AS user_attributes,
   map(event, todate(ts))   as event_times   KIND map[string]time  MERGEOP LATEST

FROM default
INTO USER
BY user_id
ALIAS my_query
' > default.lql


# start watching
lytics schema queries watch .

# now edit JSON results of how data is interpreted is output

Lytics Watch With Custom Data

  1. Create NAME.lql (any name) file in a folder.
  2. Create NAME.json (any name, must match LQL file name) in folder.
  3. Run the lytics watch command from the folder with files.
  4. Edit .lql, or .json files, upon change the evaluated result of the .lql, JSON will be output.

Example

# get your API key from web app account settings
export LIOKEY="your_api_key"

cd /tmp

# start watching in background
lytics schema queries watch &

# create an LQL file
echo '
SELECT
   user_id,
   name,
   todate(ts),
   match("user.") AS user_attributes,
   map(event, todate(ts))   as event_times   KIND map[string]time  MERGEOP LATEST

FROM data
INTO USER
BY user_id
ALIAS hello
' > hello.lql

# Create an array of JSON events to feed into LQL query
echo '[
    {"user_id":"dump123","name":"Down With","company":"Trump", "event":"project.create", "ts":"2016-11-09"},
    {"user_id":"another234","name":"No More","company":"Trump", "event":"project.signup","user.city":"Portland","user.state":"Or", "ts":"2016-11-09"}
]' > hello.json

SegmentML example

# replace {your model name here} with target_audience::source_audience

# generates tables
lytics segmentml --output all {your model name here}
lytics segmentml --output features {your model name here}
lytics segmentml --output predictions {your model name here}
lytics segmentml --output overview {your model name here}

# for CSV output
lytics --format csv segmentml --output all {your model name here}

# for JSON
lytics --format json segmentml --output all {your model name here}

analyst's People

Contributors

araddon avatar slindberg avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

analyst's Issues

Cumulative data

Dimensioned data often needs to be visualized cumulatively, but many charting libraries don't have native support for converting to cumulative values. Add a mechanism for converting dimensioned data to cumulative format. The mechanism must be applied before transforms that can change data into a non-numeric format.

Indexed reduce functions

Individual data points are often object hashes. Add reducing function(s) that allow a field to be safely indexed and used as if it existed in the flat row data.

Test coverage

Unit tests should cover 80% of the codebase as a minimum.

Convert array output data to a scalar value

Add a mechanism for turning dimensioned data output (in array format) into a scalar value, allowing for operations like selecting the first/last/nth element, adding, averaging, etc.

Complex field reducer functions

Many fields are non-scalar values that contain objects with nested values (top(), valuect()), and should support reducing these nested values (count(?), sum, average(?)).

Max reducer

Add a reducer function for finding the maximum value of a field.

Remove d3 as a hard dependency

Since crossfilter is independent of d3, and d3 is orders of magnitude larger, analyst should not depend on d3 for core functionality. This currently means implementing events without d3.dispatch, and creating shims for d3 date functions.

Source/metric cloning

A source should be able to be created from an existing source, such that underlying data/crossfilters are isolated.

A metric should be able to be created from an existing metric that is an effective clone -- same dimension, reductions, transforms, etc. New mutators do not affect the metric it was cloned from.

An alternative to metric cloning would be to return a new metric every time a method is called that modifies it, allowing a 'base' metric to be created that serves as a starting point for other metrics. This would also eliminate the need for dimension sharing logic that can potentially cause confusion.

Better error handling

There's no way to tell if a request for data failed or timed out. This is bad. source.fetch() should probably return a promise object.

Metric composition

Multiple metrics (potentially with different sources) should be able to be combined into a single metric that behaves as any other metric. Events from constituent metrics should propagate up when needed. For example, non-dimensioned metrics should be able to be combined using simple arithmetic, and dimensioned metrics of the same length should be able to be mapped similarly.

Dimensioned output shaping

The final output of a dimensioned metric uses the crossfilter format of an object with key/value properties. This is inconvenient for many charting libraries, which require different data formats. Add a post-value shaping mechanism and add common shaping functions:

  • array of objects with x and y properties
  • array of 2 (or n) length numerically indexed arrays
  • split keys/values into their own arrays

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.