Currently, any JSON can be associated with a timestamp.

I think it makes more sense to restrict to integer and floats as this will allow for more efficient data handling.

Not sure if it makes sense to have a timestamp per data point or allow multiple data points to be associated with a timestamp eg.

[ (1511254543330, 1), (1511254543330, 2), (1511254543330, 3) ]

vs

1511254543330 => [1, 2, 3]

Allow a specific time-series to be removed from storage.

Need to refactor server code.

When receiving data from twitter utf8 chars are often present in the JSON this cause the store to return a bad request and the data is not saved.

see https://pastebin.com/BLxMMMk2

The above is an example of one tweet that is successfully saved and one that is rejected by the store.

The new KV API needs an endpoint to list all the keys available under a datasource.

/<DsId>/list or something similar would be very useful.

Not sure if this is a but or a "feature"

If you write data using "write at" out of order, then the data returned by /since is also out of order.

Currently all data associated with a datasource are written to the same path within the git file store. This can be further divided by another level of key (see #9) but could also internally be divided through time-based buckets/bins to further restrict the amount of data to one path. Needs investigating in terms of performance on a pi3.

Add support for writing an array of values in a single POST.

This would require a timestamp to be included as part of an individual datapoint e.g. {"timestamp": 11111111, "value": 42}. Currently we support timestamps being defined for a POST via the URI which works for single value POSTs.

If the store crashes or is restarted then the state of the Hypercat catalogue is lost.

Currently the server is configured from command line args but this does not scale well as the options increase. Should replace with a configuration file of some standard format.

It is possible to write to the store faster than the smallest resolution of the timestamp in milliseconds.

When this happens the data return by latest, earliest API calls can be out of order causing tests to fail.

The current API is path based and only operates on a single stream of time series data. This API does not scale well or support joining together and carrying out operations on data from multiple time series streams. I useful addition would be a simple query language to give this type of flexibility.

The current backend log-based storage is suited to in-order time-series writes. Do we need to support out-of-order time-series writes?

Once a store has had a reasonable amount of data written to it, there is a delay at startup.

This is fine but calls to the API during this time appear to succeed but return not data.

I'll try and reproduce out of databox so we can test this, and to convince myself that this is whats happening.

I'll update this issue with findings later.

The hypercat can be used to contain more information on the state of the server. Currently it is just used to log which data sources exist. It could contain more information such as the number of records in the store, read/write access count, details on current observers, uptime etc.

Currently observations must expire (max-age). Can use 0 to mean never expire.

At the moment you can tag a time series value but it is restricted to name and string value. It would be good to extend this to allow some kind of geotagging to then allow data points to be queried on location.

It would be useful to list what time series ids exist in a similar way to listing the keys within the key value store.

@jptmoore

I might be missing something but, are you verifying the token on the observe endpoint

https://github.com/jptmoore/zest/blob/master/src/server.re#L493

ZestDB has a docker build for HypriotOS. Since Raspbian(the official OS) is currently supporting docker, is there any chance to have a docker build for it?

For example

A GET request for /ts/foo instead of /ts/foo/latest will hang clients as will /ts/[key]/[any unsupoted string]

To repeat:

docker run -p 5555:5555 -p 5556:5556 -d --name zest --rm jptmoore/zest /app/zest/server.exe --secret-key-file example-server-key --enable-logging

run --network host -it jptmoore/zest /app/zest/client.exe --server-key 'vl6wu0A@XP?}Or/&BR#LSxn>A+}L)p44/W[wXL3<' --path '/ts/foo' --payload '{"name":"dave", "age":30}' --mode post

docker run --network host -it jptmoore/zest /app/zest/client.exe --server-key 'vl6wu0A@XP?}Or/&BR#LSxn>A+}L)p44/W[wXL3<' --path '/ts/foo' --mode get

Posting the same Item to /cat more then once, leads to multiple enteritis in the catalogue.

Posts with the same href and datasourceid in the item should overwrite the old entry.

We have /ts/[id]/latest got get the latest data

/ts/[id]/earliest would be handy for apps so that they know how much historical data is available.

Currently, the observe endpoint will allow you to watch for changes on a single key /observe/<DsId>/<keyName>.

It would be useful to be able to watch for changes on any key, using something like this:

/observe/<DsId>

It should send

{"key":"<name of key>","value":"<data stored at key>"}

Need to reformat syntax for Reason 3 (probably in a new branch)

Need to add in support for text and binary data.

Both the KV and TS store could use an extra key to allow collections of data.

e.g. post to /ts/[datasource]/[key] => /ts/lightbulb/lounge

This store type allows drivers to publish data without having it written to disk.

It's useful for larger data sources like IP cameras or high-frequency data.

Drivers can only write to the endpoint. (but no data is saved)

Apps can only observe the endpoint to get notification of new data.

Calls to /ts//last/5/min for example before a driver has written any data will cause the following crash:

server.exe: main: report_error:Opps: (Invalid_argument "index out of bounds")Raised at file "format.ml" (inlined), line 239, characters 35-52
server.exe: main: Called from file "format.ml", line 465, characters 8-33
server.exe: main: Called from file "format.ml", line 480, characters 6-24
server.exe: main:
server.exe: main: report_error:Opps: ZMQ.ZMQ_exception(0, "Operation cannot be accomplished in current state")Raised at file "map.ml", line 122, characters 10-25
server.exe: main: Called from file "src0/sexp_conv.ml", line 149, characters 10-37
server.exe: main: