This is currently a research project, not feature-complete in any way, and overall not ready for use in production services.
This project provides dynamic synthetization of differential dataflows at runtime. Included in this repository is the library itself, as well as a server accepting query plans serialized as JSON, via CLI, TCP, or WebSocket connections. A parser generating such plans from a Datalog-like query language is provided in Clojure(Script), but other query grammars or parsers from other host languages should be able to utilize the same query plan interface.
Assuming an up-to-date Rust environment, the server can be built and run via
cargo run --release --bin server -- <timely args> -- <server args>
The server executable accepts two sets of arguments separated by --
,
one for configuring timely dataflow
and the other for configuring the server.
OPTION | DESCRIPTION | DEFAULT
--port | port to listen at | 6262
--enable-cli | accept commands via stdin? | false
--enable-history | keep full traces | false
Logging at a specific level can be enabled by setting the RUST_LOG
environment variable to RUST_LOG=server=info
.
Read the high-level motivation for this project.
Important architectural decisions are documented in the docs/adr/ sub-directory.
Documentation for this package can be built via cargo doc --no-deps
and viewed in a browser via cargo doc --no-deps --open
. Please refer
to declarative_dataflow::plan::Plan
for documentation on the
available operators. The tests/ directory contains usage
examples.
Query plans are rather cumbersome to write manually and do not map to any interesting, higher-level semantics. Currently we provide a Datalog front end written in Clojure.