Comments (8)
I'm not sure this is a good pattern to watch over a directory that does not exist yet. The client code should make sure that the directory exists before watching it (or creating it if it does not exist). We have this problem in Swarm where we don't verify that the key exists before trying to watch it for the discovery (so we'll fix it in Swarm directly). A good way would be to do something like:
if !store.Exists(key) {
store.Put(key, []byte(""))
}
store.Watch(key)
I'm not entirely sure that this is the responsibility of libkv
to do this. (So even if consul works in this case, you should still make sure that the key exists no matter which one of the backend you are using)
from libkv.
@abronan It's fine with me that the client needs to ensure the directory exists before watching it. However, etcd makes a distinction between directories (which cannot have values) and other keys. I believe that in etcd, if you Put
an empty string at a key, then try to create another key beneath it in the tree this will fail. Hence, my suggestion that we need a way to create directories. Perhaps a new WriteOption
?
from libkv.
@spikecurtis You have a point!
We can adopt either your solution or the go-client has this method: SetDir
, which transforms a key into a directory. So we can assume that you can create a directory foo/bar
which first acts as a regular key with an empty value. Then if you try to put foo/bar/foo
or tree watch foo/bar
we test if bar
is a key and if this is the case we call SetDir
to transform it into a directory.
Thoughts?
from libkv.
@abronan I don't really like the idea of a WatchTree
mutating the data. It violates the principle of least surprise.
Clearly, a well designed application should have a key structure that is unambiguous about which are and are not directories. However, it's easy to mess this up. Watches being "helpful" and overwriting existing keys (possibly throwing out any data on them) could be really difficult to debug.
from libkv.
@spikecurtis I see how this could turn bad even though the key structure is something that should be well thought out in the client code so if you do a Put
and then a Watch
on a key, the client code is aware that the current key is indeed a directory. If there is a mistaken WatchTree
on a key that holds a valuable information, it is something that should probably be fixed in the client code.
We have the choice of either:
- Follow the
Consul
model to manage keys: with no directories but just a key structure with prefixes separated by/
(So no notions ofCreate
or directories that couldn't hold values) - Follow the
Zookeeper
andEtcd
model to manage keys and directories with separate operations for both notions
This is a choice and I'm probably slightly leaning towards the Consul
model (what we are doing now with no CreateDir
operations) because it's way easier to manage. Ultimately we wanted to avoid users of the library (also us as we use it in swarm) to think about directories
or nodes
creation at all, everything is just a key (even though you can still watch child keys under a prefix
as in Consul).
It's just a matter of smoothing the experience on the three existing stores (and probably more in the future) so it's going to be nearly impossible to have an optimal solution for each store. We will need to make compromises for some backends unfortunately.
We can probably open a separate issue and talk about which model makes more sense for libkv
as this is a huge deal on how we will handle more storage backends in the future.
from libkv.
@abronan if you want to open a separate issue, can you cc me on it?
For the record, I think Zookeeper works like Consul inasmuch as it doesn't distinguish between directories and "files". I.e. znodes can store values even if they are not a leaf in the tree.
I'm not sure we can offer a totally consistent API. If we follow what you call the Consul model, then
Put("/path/to/", "")
Put("/path/to/new/node", "Hello")
will succeed on Consul, but the second call will fail on etcd. Likewise
Put("/path/to/new/node", "Hello")
Put("/path", "World!")
will succeed on Consul, but will fail on etcd.
There will be similar inconsistencies if we follow the etcd model of explicitly creating directories, since etcd will fail later attempts to Put
to them, but zookeeper and consul will succeed.
Basically, what I'm saying is that either way we go, application designers are going to have an inconsistent experience. Etcd is going to fail requests that would succeed in Consul. So, designers that want their applications to work with any backend are going to have to design the keyspace with etcd-style constraints, i.e. that you can only store data at leaves in your tree.
from libkv.
@spikecurtis Sure!
So far in swarm we only create values on Leaf keys/nodes (so we don't encounter the issue you mentioned). This is kind of a mix between Consul
and etcd
. Zookeeper
is between the two with a notion of directory (node) in which you can still put a value.
This is mentioned nowhere so we need to spin up a generic documentation explaining what to expect from the library.
In your example, etcd
would not fail if the call to Put
transforms the key into a directory. I admit that this is bad because you lose the data in key: path/
. This is a sacrifice to make in order to achieve "consistency" (double quotes here) across the three existing backends.
There are still room for improvement though as this is hard to abstract backend K/V stores with such different ways to handle keys/directories/values 😄
(PS: would you mind opening a separate issue with your example? I think this is a perfect example of what we need to take care of/improve in libkv)
from libkv.
For anyone following this thread, @spikecurtis raised #20 to cover the example above (as requested by @abronan)
from libkv.
Related Issues (20)
- do libkv support etcd v3? HOT 1
- bug in WatchTree for etcd
- Add function to list all the key-values in the boltdb store HOT 1
- add option to disable quorum for gets with etcd backend HOT 1
- Lock() of Locker interface should probably take a receive-only channel
- In etcd at least, waitLock() ignores an action HOT 1
- A race is possible in etcd's Lock() HOT 1
- Implement a Kubernetes driver
- Panic in older versions of consul
- Active again? HOT 2
- Optionally disable ZooKeeper logging
- Inconsistent reference to license for docs
- When multiple etcd stores are passed to the API. Which one is given preference? HOT 1
- can not get zookeeper path data, node cannot be discovery.
- panic in List() method with boltdb backend
- zookeeper GetW function seems has a atomic error
- possible data race in AddStore()
- implemented a etcd v3 api store HOT 1
- Travis-ci: AMD64 build is failing HOT 1
- Valkeyrie: a maintained fork created by the original author of libkv HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libkv.