Giter Club home page Giter Club logo

mohawk's Introduction

Go Report Card Build Status

Mohawk

Mohawk

Mohawk is a metric data storage engine, it's fun, fast, light and easy to use.

Table of contents

Source Modules

Introduction

Mohawk is a metric data storage engine that uses a plugin architecture for data storage and a simple REST API as the primary interface.

Mohawk can use different storage plugins for different use cases. Different storage plugins may vary in speed, persistence and scale ability. Mohawk use a subset of Hawkular's REST API inheriting Hawkular's ecosystem of clients and plugins.

Compatibility

Mohawk is tested(1) with Hawkular plugins, like Hawkular Grafana Plugin and clients like Python and Ruby. Mohawk also work with Heapster to automagically scrape metrics from Kubernetes / OpenShift clusters.

Mohawk can also serve as Prometheus scraping endpoint.

(1) Mohawk implement only a subset of Hawkular's API, some functionality may be missing.

Installation

Using a Copr repository for Fedora

sudo dnf copr enable yaacov/mohawk
sudo dnf install mohawk

Using Dockerhub repository

docker run -v [PATH TO KEY AND CERT FILES]:/root/ssh:Z yaacov/mohawk

Building from source

# Create a directory for sources
mkdir -p ${GOPATH}/src/github.com/MohawkTSDB && cd ${GOPATH}/src/github.com/MohawkTSDB

# Clone the sources from the git repository
git clone https://github.com/MohawkTSDB/mohawk.git
cd mohawk

# Update vedor sources
make vendor

# Build, test and install
make clean
make
make test
make install

# Set up tls secrets for testing
make secret

Running the server

The server command line interface

Request usage message.

mohawk -h
Usage of mohawk:
...
mohawk --options=help
Storage options:
...

Running mohawk without tls and using the sqlite back end.

mohawk --storage sqlite
2017/01/03 10:06:50 Start server, listen on http://0.0.0.0:8080

Running the server with tls, gzip encoding support and using the memory storage.

Remember to set up the server.key and server.pem files in your path
mohawk --storage memory --tls --gzip --port 8443
2016/12/01 14:23:48 Start server, listen on https://0.0.0.0:8443
Mock Certifications for testing

The server requires certification to serve https requests. Users can use self signed credentials files for testing.

To create a self signed credentials use this bash commands:

openssl ecparam -genkey -name secp384r1 -out server.key
openssl req -new -x509 -sha256 -key server.key -out server.pem -days 3650

If running from source, the Makefile has a utility for generating secrets:

make secret

Running the server for this examples

Using TLS server requires certification files, default file names are server.key and server.pem .

mohawk --tls --gzip --port 8443

Reading and writing data

Common queries

# get server status
curl -ks https://localhost:8443/hawkular/metrics/status

# get a list of all metrics
curl -ks https://localhost:8443/hawkular/metrics/metrics

# post some data (timestamp is in ms)
curl -ks -X POST https://localhost:8443/hawkular/metrics/gauges/raw \
     -d "[{\"id\":\"machine/example.com/test\", \"data\":[{\"timestamp\": 1492434911769, \"value\": 42}]}]"

# read some data (variables can be start, end and bucketDuration)
curl -ks https://localhost:8443/hawkular/metrics/gauges/machine%2Fexample.com%2Ftest/raw?start=1492434911760

Tags

# set tags
curl -ks -X PUT https://localhost:8443/hawkular/metrics/gauges/tags \
     -d "[{\"id\":\"machine/example.com/test\",\"tags\":{\"type\":\"node\",\"hostname\":\"example.com\"}}]"

# look for metrics by tag value (using a regexp)
curl -ks https://localhost:8443/hawkular/metrics/metrics?tags=hostname:.*\.com

Search

# read multiple data points using an ids list
curl -ks -X POST https://localhost:8443/hawkular/metrics/gauges/raw/query \
     -d "{\"ids\": [\"machine/example.com/test\"], \"start\": 1492434811769, \"end\": 1492435911769}"

# read multiple data points using a regexp search on tags
curl -ks -X POST https://localhost:8443/hawkular/metrics/gauges/raw/query \
     -d "{\"tags\": \"name:.*test|.*machine\", \"start\": \"-12h\"}"

Aggregation

# using relative start time, and aggregation time of 2mn
curl -ks -X POST https://localhost:8443/hawkular/metrics/gauges/raw/query \
     -d "{\"ids\": [\"machine/example.com/test\"], \"start\":  \"-6h\", \"bucketDuration\": \"2mn\"}"

Data encoding, using gzip data encoding

# using the zcat utility to decode gzip message
curl -ks -H "Accept-Encoding: gzip" https://localhost:8443/hawkular/metrics/metrics | zcat
# sendig gziped data file with curl's --data-binary flag
curl -ks -H "Content-Encoding: gzip" -X PUT "https://localhost:8443/hawkular/metrics/gauges/tags" \
     --data-binary @tags.json.gz

Issues? Questions?

Github Project / Mohawk TSDB

mohawk's People

Contributors

boaz0 avatar nimrodshn avatar yaacov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

mohawk's Issues

Add basic auth

Currently we only support bearer auth, we need to support basic auth also

alerts struct to alert buffer should be compatible with prometheus

we currently just dump our alert struct to alert buffer.
prometheus also just dump it's alert struct to alert buffer.

to work with systems the expect prometheus alert struct, we need to dump a struct the looks like a prometheus alert strucs ( this alert mode should have a cli flag like --dump-full-alerts or other flag )

Proposal: Using vendor to manage packages

May I suggest using vndr to vendor packages in this project.

For example, to build the command line interface, mohawk is using github.com/spf13/cobra and github.com/spf13/viper. This will require from anyone who builds mohawk from source to do go get -u -v github.com/spf13/viper before compiling.

This is not always good as at the time of work the master branch was pointing at commit a while today master points at commit b which breaks the API that was used in commit a and that ends in failing to compiling mohawk in the future.

Moving alert worker to different threads

I think we should move from 'one thread to rule them all' model to 'one thread for each alert' [1] this will allow different alerting frequencies for different metrics (currently alerts worker run for all metrics at the same rate AlertsInterval) at the future I think we should probably group alerts by their Interval or give each alert an Interval/Frequency field.
[1] Essentialy, moving this line -

go a.run()

Up to this for loop-
for _, alert := range a.Alerts {

This is far term goal for me..
cc: @yaacov

Adding alert frequency

Adding some more control on the frequency of the alert worker - If user knows he only pushes data once / twice a day (or even once an hour) there is no need to check every ten seconds, on the other hand if he/she knows he pushed every millisecond then checking once an ten seconds (like we do now) is irrelevant..

  • add a command line ( + config file ) argument to set the alert check loop interval in seconds [1], now hardcodded to 10s.
  • fallback to 5s, alerts should fire immediately on change.

[1] https://github.com/MohawkTSDB/mohawk/blob/master/alerts/alerts.go#L78

Adding complex alerts

Adding complex alerting i.e.

Alert #1 :  "free memory too low"
Alert #2: "cpu_usage too low"
Alert #3: Alert #1 && Alert #2 are on.

Need an /alerts api endpoint

Add a /hawkular/alerts/raw api endpoint:
[ EDIT: always filter by tenant ]

a. get all alerts

Extra points for:

b. filter by state
c. filter by alert_id ( using regex )

Additional points if:

Add a /hawkular/alerts/status api endpoint:
a. get status of alert runner ( e.g. interval, buffer url, time of last complete check cycle ... )

Proposal: Using http.HandlerFunc instead of struct in middleware

I would like to suggest a more readable way to create and use middlewares.

Instead of creating a structure with parameters, we can write a function that get those parameters and return a new function that implements the logic of the middleware.

For example, instead of writing a authentication middleware like this

type AuthMw struct {
  Token string
  RegexpPath string
  next http.Handler
}

func (amw *AuthMw) SetNext(h http.Handler) {
  amw.next = h
}

func (amw *AuthMw) ServeHTTP(w ..., r ...) {
   ...
  amw.next(w, r)
}

It will be done in the following way:

func AuthDecorator(token, regexpPath string) Decorator {
   pRegex := regexp.MustCompile(regexpPath)
   tokenVal := "Bearer " + token
  return Decorator(func(h http.HandlerFunc) http.HandlerFunc {
    ...
    h(w,r)
  })
}

In that way we wrote less code and it is easier to reason what the middleware does.

In addition, we could easily wrap middlewares together

var m Middleware
for ix, mw := range middlewares {
 if ix == 0 {
  m = mw
 } else {
  m = mw(m)
 }
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.