datadog / kvexpress Goto Github PK

View Code? Open in Web Editor NEW

128.0 52.0 13.0 284 KB

Go program to move data in and out of Consul's KV store.

License: Apache License 2.0

Makefile 1.46% Go 80.03% Shell 6.79% Ruby 11.72%

kvexpress's Introduction

kvexpress

Why?

Small Go utility to:

Put data into Consul's KV store.
Pull data out of Consul's KV store and write it to a file.

Why a dedicated utility though? Can't I just do it with curl?

Yes you can - but we kept wanting to:

Make sure the file was long enough. 0-length configuration files are bad.
Load the file from some other custom templating process - not just from straight KV files.
Put the file into any location in the filesystem.
Restart/reload/stop/start daemon after writing the file.
Run some other custom command after writing the file.
Verify that the file we put into the KV was the same file that was written on the other end.
Stop the process on all nodes - in or out - if we want everything to stay as it is for the moment.

We did this at first with some custom Ruby scripts - but the pattern was apparent and could be applied to many other files as well.

This replaces our previous custom Ruby/shell scripts with a single Go binary we can use to get data in and out of Consul's KV store.

How does it work? - 1000 foot view

There's an introductory blog post available that shows how it can be used.

In

kvexpress in --key hosts --file /etc/consul-template/output/hosts.consul --length 100 --sorted=true

Check that at least --file and --key are passed along with the command. Quit if they're not present - there are no safe defaults for those flags.
Check for the existence of a stop key - if it's there - stop and exit.
Read the file into a string, and sort the string if requested.
Check if the file is long enough - if not - stop and exit.
Save the file to a .compare file - we will use this data from now on.
Check for the existence of a .last file - if it's not there - create it.
Are the .compare and .last files blank? If not - let's continue.
Compare the checksums of the .compare and .last files - if they're different - continue.
Grab the checksum from Consul and compare with the .compare file - if it's different - then let's update. This is to guard against it running on multiple server nodes that might have different .last files.
Save data, and checksum keys.
Copy .compare to .last
If --exec is passed - run that command.

Out

kvexpress out -k hosts -f /etc/hosts.consul -l 100 -e 'sudo pkill -HUP dnsmasq'

Check that at least --file and --key are passed along with the command. Quit if they're not present - there are no safe defaults for those flags.
Check for the existence of a stop key - if it's there - stop and exit.
Pull the data and checksum keys out of Consul.
If data is long enough and the checksum as computed on this side matches the checksum key - then continue.
Write the contents of data to the passed --file location.
If --exec is passed - run that command.

Where can I get it?

Build instructions are available below.

Debian packages for Ubuntu can be downloaded from packagecloud.

Additional binaries can be downloaded from the releases page.

A Chef cookbook is available as well.

How to use it with Consul watches.

Instructions are available here.

Ad-Hoc Usage with Consul exec

If you want to push a file to all nodes in your Consul cluster quickly - you can use Consul and kvexpress.

kvexpress in --prefix tmp --key config --length 5 --url https://git.io/vzz5P

This places the file and checksum in:

tmp/config/data
tmp/config/checksum

To get the file on every node:

consul exec "sudo kvexpress out --prefix tmp --key config --length 5 -f /etc/file.conf -e 'sudo restart something here'"

If you're not seeing it work the way you expect - you can always add --verbose to the kvexpress command - that will add lots of additional output.

Commands Available

A detailed list of commands is available here.

Consul KV Structure

How are keys organized in Consul's KV store to work with kvexpress?

Underneath a global prefix /kvexpress/ - each directory represents a specific file we are distributing through the KV store.

Each directory is named for the unique key and has the following keys underneath it:

data - where the configuration file is stored.
checksum - where the SHA256 of the data is stored.

For example - the hosts file is arranged like this:

/kvexpress/hosts/data
/kvexpress/hosts/checksum

There is an optional stop key - that if present - will cause all in and out processes to stop before writing anything. Allows us to freeze the automatic process if we need to.

Logging

All logs are sent to syslog and are tagged with kvexpress. To enable debug logs, please export KVEXPRESS_DEBUG=1

Build

Can be built with the standard go toolchain: go get -u -v github.com/DataDog/kvexpress

To build manually - clone the repo then: make deps && make

To run integration tests: make deps && make && make test - it will spin up an empty Consul and kill it after the run.

Because we use user.Current() - you can't cross compile this. If you want to build for Linux - you must build on Linux. Closed Issue

To install Consul - there are instructions here.

To launch an empty Consul instance: make consul

Contributing

We love pull requests from anyone. Details are available here.

Code of Conduct

This project adheres to the [Open Code of Conduct][code-of-conduct]. By participating, you are expected to honor this code. [code-of-conduct]: http://todogroup.org/opencodeofconduct/#kvexpress/[email protected]

kvexpress's People

Contributors

Stargazers

Watchers

Forkers

dindinw 40a devopsbox labtests digitaloptimizationgroup bobbyshaftoe isavcic davidd2k mattconzen isabella232 noqcks

kvexpress's Issues

Add Datadog event logging to `stop` command.

Want to know when that's happening.

Of course - without a config file that already has the APP key and API key it's less than useful.

--version

Needs that command.

Blank lines are annoying.

With the --sorted flag - if there are lots of blank lines then it keeps them all.

Need to remove them - don't want to shell out to uniq to do it.

Update Docs - they aren't accurate right now.

Add `in` command.

Replaces kvexpress.rb.

Remove file output from syslog

Not needed when we go to production.

Too much output.

Can we use `check and set` to lock during event.

Investigate.

Do we add some sort of random optional delay?

So that not everything restarts at the same time?

Or do we make it so that restarts are nice and smooth? (Probably better.)

Sticking this here to think about it.

Add dogstatsd logging.

diff for `in` command doesn't seem to work.

Will take a look at it later and continue the rest for now.

consul api panics if it doesn't find the stop key

It's likely how I've programmed it, but I'd like it to just return nil and keep on moving.

Refactor functions to `kvexpress` package.

To allow for some code reuse between the out and in commands.

Refactor Consul connection logic.

Make it smarter.

Setup the connection once per binary invocation.

Make diff output for `in` command better.

It's pretty ugly right now.

Time guage over length of kvexpress run.

Sent to dogstatsd.

Doesn't copy the file to .last if the checksum in the KV store matches.

You end up in this state:

[staging]root@i-c972d41e:/etc/consul-template/output# ls -al
total 48
drwxr-x--- 2 root root  4096 Nov  3 22:39 .
drwxr-x--- 7 root root  4096 Aug  7 16:33 ..
-rw-r--r-- 1 root root 16516 Nov  3 22:39 config_hosts
-rw-r----- 1 root root 16335 Nov  3 22:39 config_hosts.compare
-rw-r----- 1 root root    22 Nov  3 22:39 config_hosts.last

Where - the files didn't match - but the checksum inside the KV store matched:

[staging]root@i-c972d41e:/etc/consul-template/output# shasum -a 256 config_hosts.compare
deab902e0ab2f432f0ebc68035992f10a743059222cbea991f3345ba195c4dad  config_hosts.compare

So it didn't update the KV store - but it leaves the .last file in the wrong state.

make version 0.1

Be able to create a specific version for distribution.

Let's not log the token. That's a bit of a fail.

Maybe we can obfuscate it a bit - and show beginning and end?

Track the byte sizes of the config files we're putting in.

So that we can make sure we're not getting too big for Consul's KV store. The current limit is 512KB.

Strip the top 3 lines from the diff.

Isn't needed and doesn't add anything in this context.

Add some tests.

There's none at the moment.

Would be nice to have some automated ones.

Don't connect to Datadog if API and APP keys aren't set.

Right now - it's set to always connect.

May need to set the data, checksum and updated keys all at once.

To deal with possible race condition where:

We're watching the "data" key.
We update the "data" key - but not the "checksum" or the "updated" keys.
Watches fire - and they checksum against the old version.

A possible alternative solution:

May need to instead watch the "updated" key as that's updated last.

Tokens

We sometimes need to use them.

Right now there's no token awareness at all.

Something strange happening with CONSUL_TOKEN ENV variable.

Testing on staging.

Not sure if it's related to the 0.6 upgrade.

Make sure `out` command follows `stop` key addition.

That way - when we hit stop:

Nothing goes in.
Nothing comes out.

Oct 30 15:48:58 i-c972d41e kvexpress:  2015/10/30 15:48:58 in: file='compare' full_path='/etc/consul-template/output/etc/consul-template/output/config_consul_config.ini.compare'
Oct 30 15:48:58 i-c972d41e kvexpress:  2015/10/30 15:48:58 in: file='last' full_path='/etc/consul-template/output/etc/consul-template/output/config_consul_config.ini.last'

Datadog API and App keys.
Connection parameters that may never change.

Resurrect this branch:

https://github.com/darron/kvexpress/tree/darron/config-file