xtdb / xtdb Goto Github PK

Our querying code potentially matches the criteria for when to use reducers, as laid out by Alex in this comment: https://www.reddit.com/r/Clojure/comments/5lhz7q/seq_functions_reducers_transducers_when_do_i_use/dbw3fph/.

It seems a performance gain worth chasing, to utilise the cores available.

The idea of branching off a time dimension. This is with a specific use case in mind: banks wanting to snapshot their data at business time T, but then to make corrections against this newly formed timeline.

It would be good to have a story around this in the docs - the need for, and the way Crux solves this problem (even if it doesn't directly solves it, but offers a helpful path).

Currently, not MVP.

We want to have some insight into the Kafka cluster, but avoid building too many pieces, we might be able to use something like these for example:
https://www.datadoghq.com/dashboards/kafka-dashboard/
https://www.confluent.io/product/control-center/

There should be a way to deploy a Crux cluster into the cloud.

This might involve several layers, from Docker Compose to full AWS one click deployment or Kubernetes pods.

For this to be simple to use, we're likely to need a remote REST API, see #8. Depends on #5.

For the MVP we assume that a Crux cluster also includes Kafka and Zookeeper, these components might later be possible to swap out.

Can API users make subscriptions?

Performance testing on AWS with large data sets.

https://snap.stanford.edu/index.html

The join query in the micro-bench is extremely slow. We are not leveraging the AVT index in the triple store where we could be.

Ability to take a backup of the KV store and restore it on another node.

A variant can be to seed it on start with a URL to either a directory or S3 resource to bootstrap the KV store from.
There should also be a way to trigger the backup of a running node.

We're currently want to avoid building intelligence about when to do these actions into the code itself as long as it can be scripted.

We want to support bitemporality. In our model the easiest way should be to add a transaction time at the end of the index key, as business time is the usual axis for query. When querying for transaction time, business time can default to transaction time if not given, as there can (?) be no writes in the future. When transaction time is not given, it defaults to now, or the last part of the index is simply ignored.

Needs further analysis.

Because there's only so much you can fit in a github readme.

For MVP: something simple and clean in Asciidoc, with basic authentication.

Some of this info might be indirectly be visible through a Kafka dashboard, as it depends on where the node is at in its consumption of the topics.

We want to spike Proposal A from docs/internals.md aside crux.kv, it should preferably be able to reuse most of the crux.kv-store and crux.db protocols and have crux.query work without too many changes.

This isn't necessarily about performance, but just a realistic domain and set of queries to stress the engine in different ways.

Part of the API to the KV store must be to be able to back itself up to a directory.
Another namespace will then take this, tgz it and back it up somewhere else (like s3).

The Kafka offsets are stored inside the KV store, so a replica should be able to fetch the latest backup and simply continue.

https://github.com/facebook/rocksdb/wiki/How-to-backup-RocksDB%3F

We might want to create a separate topic for transient, high volume data that can still be queries, but should not get stuck in the transaction log, and might also not be documents, but more individual measurements. Some of this high churn data could be related to existing entities, and might also be pushed onto a second, more durable topic for various retention mechanisms.

You can potentially get some of this retention "for free" by putting them into a compacted topic by key/time-granularity, like :foo/#inst "2001-01-01" which will ensure that there's only one version of :foo for this day kept.

There's a large perf gain for Rocks in re-using Iterators in queries, especially where multiple seeks might be made. For example there's a 30-40% improvement for the multiple-clauses query in the micro-bench.

We could add a new protocol fn in-context to Datasource, so that queries's can surround query execution with this.

Some thoughts:

• the name in-context can be improved upon
• Since we may one day use reducers or pmap & friends, we should probably not use thread-locals and bite the bullet with a context arg to iterate-with.

I had an initial attempt, but have decided to wait on implementing this, until a conversation is had around the above two points. It's also a straightforward change, so it's not critical at this stage (but would be critical prior to the MVP release).

We want to be able to build a JAR that starts consuming from a Kafka topic and populates the KV store.

This implies adding some basic configuration, like where the broker lives, which topic to subscribe to and where to store the KV data.

This does not include remote access to query the KV store yet, for this, see #8.

By having the documents in one (or more) separate, potentially partitioned, topic(s) which the immutable transaction log refers to, we can delete and compact a way data. This also opens up to archive older or seldom used documents in various ways.

It should be possible to leverage the embedded Kafka setup used for tests to start up a local cluster and running the main method to have an ingest node running in process, with state persisted across runs.

We want to import the N-triples from http://wiki.dbpedia.org/services-resources/ontology

This is around 8Gb unzipped, without any temporal aspects. This is still a good first start to stress the engine.

Will require some form of upsert and involves mapping RDF to meaningful EDN.

Range queries should use the underlying indexes to scan the data.

As part of seek-and-iterate, we generate small vector tuples. Rudimentary REPL testing shows a slight perf gain if we use defrecords.

Note, the perf downside of small vectors could also be due to the way we are consuming them - more investigation needed.

Note (2), profiling shows it could be that multiple calls to (.key i) on the Rocks iterator has a hit. Refactoring to a micro-type should also these tackle redundant calls.

There are various LMDB bindings for Java. We want one which works in Java 9 and later.

https://github.com/lmdbjava/lmdbjava
https://github.com/lmdbjava/lmdbjava/blob/master/src/test/java/org/lmdbjava/TutorialTest.java

https://github.com/LWJGL/lwjgl3
https://github.com/LWJGL/lwjgl3/blob/master/modules/samples/src/test/java/org/lwjgl/demo/util/lmdb/LMDBDemo.java

https://github.com/deephacks/lmdbjni

The lwjql library claims to work in Java 9, while lmdbjava seems to have issues.

Ability to really remove data as part of a transaction. Combined with Kafka compaction this can be used to implement various retention mechanisms and to delete personal data.

It should be a relatively cheap operation, and not require rebuilding the entire topics or indexes.

We want a simple crux.client namespace that allow you to both submit transactions to Kafka and query the query nodes over HTTP. This might not be the end game. See #8

There needs to be a way to query the data.

The simplest approach is to assume every participant is running their own query node as a library, and use the API directly.

Another approach is some from of remote API, either just taking Datalog over REST and returning EDN, or GraphQL (but likely not for MVP).

We want to be able to transact in a few different ways.

Delete can delete either an entire entity, or an attribute.
Upsert merges a new version of an entity with the existing one.
Put overwrites and entity with a new version.

It should preferably be possible to provide an identity attribute when transacting and have the entity id resolved if it an entity with this attribute exists.

Initially to help with timings and reporting in the micro-bench

We want to explore various retention mechanisms. For the docs this can be done similar to eviction #32, having the query node decide which versions of an entity to delete. For TTL it's for example all versions of an entity older than a specific time (compared to transaction time) for example.

Unlike eviction, retention is likely done by logic on the query node.
There are more advanced versions where you keep data for say every day or month.

Similar to RocksDB #4 we need to be able to backup and restore, see #6, the LMDB back end.

See https://github.com/facebook/rocksdb/wiki/Transactions

We would prefer the entire transaction from Kafka to be written in a single RocksDB transaction.

We want to support lookup refs or something similar for the crux.kv/id attribute, both to simplify upserts and deletes. We also want to support user defined identity attributes on the initial insert.

Recursion and traversal can only be expressed using rules in Datalog, so we need support for this.

Use reducers and transducers for Query.

Should optimise for

[e :attr]

We want to support more than one value for an attribute, similar to Datomic's cardinality many.

An extension of this is to support actual types apart from (the implicit) sets, primarily lists but maybe also maps. This might be easier done using documents and not triplets. This card does not necessarily need to go that far, as that probably requires changing the underlying model.

I.e. to fail if a query goes beyond a sensible threshold

An intrinsic part of my workflow. Worth looking into.

It should be possible to read the entire history for a set of entities.

This does not necessarily mean that we need to support queries over the history database.

There should be a way to ensure you can easily read what you just wrote.

When transacting, one could get some form of transaction context (transaction id, transaction time) back, and queries can block until the node has caught up until this point.

Currently number? maps straight to long, which isn't correct. We should consider simply encoding the basic primitives and a support a similar list to Datomic, see:
https://docs.datomic.com/on-prem/schema.html#required-schema-attributes

We could also consider defaulting to the EDN encoding for other things for simplicity?

We want the bench marks to be runnable as part of the build, potentially in CI. They should be canary style tests that can tell us if something suddenly went slow.

This new query backend will be based straight of the indexes and use worst case optimal joins.

These indexes are designed to be able to be semi-lazy, so only when sorting is necessary do they need to realise full results from an index (but usually not for the entire query). One goal is to be able to get a lazy seq of results, so one can handle much larger result sets.

The indexes have implicit support for normal unification joins and ranges, so these features will be kept from the current query capabilities.

Set based not, != should be possible to add. Clause-based not (ie. not-exists) should also be possible to implement. We also aim to support or. Support for and is implicit, but the operator might be needed for certain scenarios with nesting.

We don't aim to support or-join or not-join yet, as they are a bit quirkier both to understand and to implement.

Function predicates (other than the built-ins) can be added as a post-processing step when realising a single result, but often they should be possible to push down as an decorator on a specific unary index. Predicates operate on the ground facts, and need the entity documents to retrieve the attribute values.

This issue is closely related to #10.

Once we have #1965 and #1972. Possibly using AWS CodeDeploy.

Connect with CircleCI.

Redeployment should be manual, not automatic, as long as we're all pushing to master with wild abandon. People might be using the deployed node to do tests, it shoudn't go down every time someone fixes a typo.

Use for testing in the cloud. A simple wep-app presenting a datalog console, using an embedded Crux, connecting to a provision kafka cluster.

This is a tool for development now, that may or may not graduate into part of the actual product.

We want a way to deploy the Crux query nodes next to our Kafka cluster, see #1965.