datadog / kvexpress Goto Github PK

If a file already exists, and it's the same stuff. Don't write it.

Maybe we can obfuscate it a bit - and show beginning and end?

Would make it easier to adjust as needed.

Add flag and exec that command.

Much easier than sshing into the web interface. Finding the token to look at the KV store and setting it there.

Investigate.

So that not everything restarts at the same time?

Or do we make it so that restarts are nice and smooth? (Probably better.)

Sticking this here to think about it.

Want to know when that's happening.

Of course - without a config file that already has the APP key and API key it's less than useful.

So that files can land with different owners / groups.

10/22/15 10:08:06.000 PM kvexpress[78395]: 2015/10/22 22:08:06 out: file_wrote='true' location='file' permissions='416'

I want them to show 0640

It's pretty ugly right now.

Needs that command.

That way we can easily do that programmatically from capistrano or consul exec

Right now I'm doing it manually which is error prone.

Sent to dogstatsd.

That way - when we hit stop:

1. Nothing goes in.
2. Nothing comes out.

Replaces kvexpress.rb.

There's none at the moment.

Would be nice to have some automated ones.

Testing on staging.

Not sure if it's related to the 0.6 upgrade.

Right now there's no docs.

Not needed when we go to production.

Too much output.

Don't want to hang waiting.

Which is quite nice - helps with all sorts of debugging - but will need to be tuned when we're in production.

Make a way to do that.

We sometimes need to use them.

Right now there's no token awareness at all.

To grab a file from a URL and place it as a file.

So that we could grab files as a replacement if necessary.

To allow for some code reuse between the out and in commands.

Would make it easier for people to start.

Like an actual command in the root rather than how I've done it?

Seems sort of hackish.

Right now - it's set to always connect.

Just to see - don't see any performance problems at the moment.

Make it smarter.

Setup the connection once per binary invocation.

With the --sorted flag - if there are lots of blank lines then it keeps them all.

Need to remove them - don't want to shell out to uniq to do it.

You end up in this state:

[staging]root@i-c972d41e:/etc/consul-template/output# ls -al
total 48
drwxr-x--- 2 root root  4096 Nov  3 22:39 .
drwxr-x--- 7 root root  4096 Aug  7 16:33 ..
-rw-r--r-- 1 root root 16516 Nov  3 22:39 config_hosts
-rw-r----- 1 root root 16335 Nov  3 22:39 config_hosts.compare
-rw-r----- 1 root root    22 Nov  3 22:39 config_hosts.last

Where - the files didn't match - but the checksum inside the KV store matched:

[staging]root@i-c972d41e:/etc/consul-template/output# shasum -a 256 config_hosts.compare
deab902e0ab2f432f0ebc68035992f10a743059222cbea991f3345ba195c4dad  config_hosts.compare

So it didn't update the KV store - but it leaves the .last file in the wrong state.

For some of the values:

1. Datadog API and App keys.
2. Connection parameters that may never change.

Resurrect this branch:

https://github.com/darron/kvexpress/tree/darron/config-file

So that we can put larger files in AND/OR use less network traffic.

So that we can make sure we're not getting too big for Consul's KV store. The current limit is 512KB.

Will take a look at it later and continue the rest for now.

Example output:

Oct 30 15:48:58 i-c972d41e kvexpress:  2015/10/30 15:48:58 in: file='compare' full_path='/etc/consul-template/output/etc/consul-template/output/config_consul_config.ini.compare'
Oct 30 15:48:58 i-c972d41e kvexpress:  2015/10/30 15:48:58 in: file='last' full_path='/etc/consul-template/output/etc/consul-template/output/config_consul_config.ini.last'

That way we can put stuff there too - would be easier.

To deal with possible race condition where:

1. We're watching the "data" key.
2. We update the "data" key - but not the "checksum" or the "updated" keys.
3. Watches fire - and they checksum against the old version.

A possible alternative solution:

May need to instead watch the "updated" key as that's updated last.

For times when you've got it stopped but still want to be able to pull stuff out.

Isn't needed and doesn't add anything in this context.

Do we want to bother to do anything with it in kvexpress?

Right now we're ignoring it.

Would this be a problem when deploying new nodes?

We "say" stop - but new node still needs the files.

Be able to create a specific version for distribution.

It's likely how I've programmed it, but I'd like it to just return nil and keep on moving.