mephux / envdb Goto Github PK

View Code? Open in Web Editor NEW

520.0 26.0 31.0 42.41 MB

LEGACY use https://github.com/mephux/kolide -- Envdb - Ask your environment questions with osquery.

License: Apache License 2.0

CSS 16.84% JavaScript 64.96% HTML 3.36% Shell 0.61% Go 13.81% Makefile 0.42%

envdb's Introduction

!!!!!!!!!!!!!!!!!! LEGACY use https://github.com/mephux/kolide

Envdb - Environment Database

NOTE: envdb is still beta software.

Envdb turns your production, dev, cloud, etc environments into a database cluster you can search using osquery as the foundation.

Envdb allows you to register each computer, server or asset as a node in a cluster. Once a new node is connected it becomes available for search from the Envdb ui. Envdb was built using golang so the whole application, node client and server comes as one single binary. This makes it really easy to deploy and get working in seconds.

Video Intro: https://youtu.be/ydYr7Ykwzy8

How it works.

Envdb wraps the osquery process with a node agent (node as in cluster node) that can communicate back to a central location. When that node gets a new query, it's executed and then sent back to the tcp server for rendering. Once the request is processed it's then sent to any avaliable web clients using websockets.

Envdb has an embedded sqlite database for node storage and saved searches.

ui --websockets--> server --tcp--> node client.

Download

Pre-built versions (deb/tar.gz) of envdb are avaliable for linux 386/amd64. linux downloads

Building on macosx is easy tho, checkout the section below.

Building

Make sure you have Go installed. I used Go version 1.4.1.

deps

go get github.com/jteeuwen/go-bindata/...
go get github.com/elazarl/go-bindata-assetfs

make

git clone https://github.com/mephux/envdb.git
cd envdb
make

Usage

NOTICE: The default username and password for the UI.
- username: [email protected]
- password: envdb

usage: envdb [<flags>] <command> [<flags>] [<args> ...]

The Environment Database - Ask your environment questions

Flags:
  --help       Show help.
  --debug      Enable debug logging.
  --dev        Enable dev mode. (read assets from disk and
               enable debug output)
  -q, --quiet  Remove all output logging.
  --version    Show application version.

Commands:
  help [<command>]
    Show help for a command.

  server [<flags>]
    Start the tcp server for node connections.

  node --server=127.0.0.1 [<flags>] <node-name>
    Register a new node.

  users [<flags>]
    User Management (Default lists all users).

Server

$ envdb help server

  usage: envdb [<flags>] server [<flags>] [<command>]

  Start the tcp server for node connections.

  Flags:
    -p, --port=3636  Port for the server to listen on.
    -P, --http-port=8080  
                     Port for the web server to listen on.

  Args:
    [<command>]  Daemon command. (start,status,stop)

Running the server without a `command` (start, stop or status) will run the server in the foreground.

* Note: By default this will start the tcp server on port 3636 and the web server on port 8080.

Node Client

$ envdb help node

  usage: envdb [<flags>] node --server=127.0.0.1 [<flags>] <node-name>

  Register a new node.

  Flags:
    -s, --server=127.0.0.1  
                     Address for server to connect to.
    -p, --port=PORT  Port to use for connection.

  Args:
    <node-name>  A name used to uniquely identify this node.

`sudo envdb node --server <ip to server> SomeBoxName`

That's it - it's really that simple.

User Management

list users envdb users
add a new user envdb users --add
remove a new user envdb users --remove <email>

More UI

Self-Promotion

Like envdb? Follow the repository on GitHub and if you would like to stalk me, follow mephux on Twitter and GitHub.

TODO

add in memeory pagination for results.
Node/Server auth, verification and validation.
Code cleanup (will continue forever).

envdb's People

Contributors

Stargazers

Watchers

envdb's Issues

limited options for last node in list

Thank you for fixing the node scroll bar.

I had to redeploy the envdb 0.4.1 binary because 0.4.0 clients wouldn't reconnect to the server running 0.4.1.

I have enough nodes that at any browser window size I need to scroll to see them all.
The last node in my list is offline and I would like to delete it.
There isn't enough screen space for the full menu to appear so I can't delete the node.

I have the same issue for any other node in the list that happens to be positioned at/near the bottom of the browser window but I can work around them by adjusting the scroll bar and then trying again.

With the last item in the list that isn't an option.

Have you experienced this?

Explaining similarities and differences to osquery

Facebook's osquery have similar concepts.

It would help if the README file would include a quick comparison.

envdb.io and query sharing.

I got envdb.io for the project and I want to build a web service for submiting useful queries and loading them from the EnvDB ui. I opened this issue to track feature concepts or use submitted feedback.

Additional backend database support

Is there any WIP to allow Envdb to use something other than sqlite? I'd love to be able to use Postgres so I could throw this up on Heroku. From the looks of things, since you're using xorm this should be pretty easy and mostly just a matter of some configuration changes. I might be able to work on a PR if this isn't already happening.

Very fresh linux box make error

This is a pretty bare bones ubuntu 14.04 box w/ go 1.4.2 installed w/ gvm. It looks like a Makefile issue... in that maybe it's trying to run bindata the target before installing bindata the dependency.

Here is the tail end of make --debug

Considering target file `all'.
 File `all' does not exist.
  Considering target file `deps'.
   File `deps' does not exist.
    Considering target file `bindata'.
     File `bindata' does not exist.
     Finished prerequisites of target file `bindata'.
    Must remake target `bindata'.
Putting child 0x019e5430 (bindata) PID 23109 on the chain.
Live child 0x019e5430 (bindata) PID 23109
==> Embedding Assets
Reaping winning child 0x019e5430 PID 23109
make: go-bindata: Command not found
Live child 0x019e5430 (bindata) PID 23111
Reaping losing child 0x019e5430 PID 23111
make: *** [bindata] Error 127
Removing child 0x019e5430 PID 23111 from chain.

also, make is:

GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-pc-linux-gnu

Building on OS X

I'm starting to play around with envdb but ran into a couple errors while compiling for OS X 10.10.3.

The main one is that -ldflags "-s" throws a deprecation error:

==> Embedding Assets
==> Building envdb
# _/Users/mojo/scratch/envdb
ld: warning: option -s is obsolete and being ignored

The resulting envdb binary then threw an error when trying to run it:

~/scratch/envdb/bin master! $ ./envdb 
Bus error: 10

Removing the -ldflags from the Makefile worked for me.

A minor error is that the version of grep that OS X ships with doesn't have a Perl regex mode, so the -P throws a usage warning in your grep for Version:

==> Embedding Assets
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
        [-e pattern] [-f file] [--binary-files=value] [--color=when]
        [--context[=num]] [--directories=action] [--label] [--line-buffered]
        [--null] [pattern] [file ...]

Other than that everything is looking good so far -- thanks!

scroll nodes list

First off, envdb is a great idea. Great job.
How do I enable scroll for the nodes list?
The nodes list cuts off at screen size (using Chrome).

can't load package: package _/home/dan/envdb: cannot find package "_/home/dan/envdb" in any of:

$ make
==> Embedding Assets
==> Installing dependencies
can't load package: package _/home/dan/envdb: cannot find package "_/home/dan/envdb" in any of:
    /usr/lib/go/src/pkg/_/home/dan/envdb (from $GOROOT)
    /home/dan/envdb/Godeps/_workspace/src/_/home/dan/envdb (from $GOPATH)
    /home/dan/stuff/src/_/home/dan/envdb
godep: go exit status 1
godep: exit status 1
make: *** [deps] Error 1

$ echo $GOROOT x $GOPATH
x /home/dan/stuff

Link to Kolide is broken

@mephux the main link needs to be updated.

perhaps use go-extpoints for extensions

Very cool project. I'll be thinking about how we can integrate it with other tools in the space we're working in. Anyway, I wanted to suggest (since I saw you had as a todo) a tool used for a nice way to do extensions in Go:
https://github.com/progrium/go-extpoints

Here is a blog post explaining our first major use case and a bit on the user experience of it:
http://gliderlabs.com/blog/2015/03/31/new-logspout-extensible-docker-logging/