puts that would succeed on consul & zookeeper fail on etcd about libkv HOT 5 OPEN

Thanks for opening the issue @spikecurtis

I'm actually torn between just changing the documentation and waiting for the support in etcd v3 (which will come at some point in the future, without certitude), or using hidden keys to store directory informations.

If I'm not mistaken, using hidden keys is the only way to store informations for a directory without the key being listed or events being triggered for that key. The interoperability with other client libraries is not really an issue in that case: they must point directly to /path/to/__data__ and know about the key structure anyway.

I'm not against making the change in that case because this is a bug. Even though this is a design issue in etcd that must be eventually fixed.

from libkv.

@abronan it's more than other client libraries seeing keys they wouldn't recognize.

Suppose you have an existing application that writes to etcd, and you want to use libkv to interoperate with that application. If we append __data__ to every key the user of libkv supplies, then they will be unable to read or write to the "normal" keys the existing application uses.

We can't just use this trick only for "directory" keys, since we won't know a priori whether the path is a leaf or not. So every Put and Get for the etcd backend would have the suffix.

My personal feeling is to change the documentation. At this stage, anyone serious about multiple backends better be testing them with their application. Other options involve a lot of black magic and complexity for very little benefit.

from libkv.

To my understanding, the prefix would only be used for a single unique key __data__ under each directory (and not every key).

So we would have a structure like this:

path/to/
path/to/__data__    // Holds the data for the directory, hidden to regular GET operations like List/Watch, etc.
path/to/key1
path/to/key2
path/to/key3

path/
path/__data__       // We can have a key here as well because "path/" is a directory

The process of Put for etcd and to actually hide the directory/key distinction would be something like:

Put("path/to/directory/", []byte(""), nil)
// "path/to/directory/" is a regular key

Put("path/to/directory/node1/", []byte(""), nil)
// Changes "path/to/directory/" from a key to a directory, moving the data to the child key `__data__`

Put("path/", []byte(""), nil)
// We create a new child key `__data__` under "path/" to hold the data

Now, if you try to Get("path/to/directory/"), because this is a directory, it will append __data__ to get back the value from the hidden key.

Unless I'm missing something obvious, this would be possible but this involves a lot of trickery and back and forth transformations between key and directory.

Other options involve a lot of black magic and complexity for very little benefit.

I actually agree with this. Not sure if this is a good option as this hides a lot of things that could backfire on us if people are upgrading to new versions of etcd or using other client libraries. It is definitely possible but this will change a lot of code under each call to make the distinction and handle special cases.

Is there any actual Issue or open PR in the etcd repository to track the changes in API v3? We could probably concentrate our efforts here and help making this change happen in etcd.

from libkv.

@spikecurtis After a discussion with @kelseyhightower (thanks for all the infos!), etcd API v3 will definitely drop this distinction about directories and keys. So we might just wait and document it properly until this drops and we make the appropriate changes :)

from libkv.

Base on my observation, etcd and zookeeper act like a tree tree, consul on the other hand like a hash table.

Say we Put("foot/to/new/node") to an empty backend, if the command succeed, etcd and zookeeper will have the same amount of keys(3 in this case), while consul only create 1 key.

and consul treat foo, foo/ as two different keys.

theses should be considered.

from libkv.