Ballista is a distributed compute platform primarily implemented in Rust, using Apache Arrow as the memory model. It is built on an architecture that allows other programming languages to be supported as first-class citizens without paying a penalty for serialization costs.

The foundational technologies in Ballista are:

• Apache Arrow memory model and compute kernels for efficient processing of data.
• Apache Arrow Flight Protocol for efficient data transfer between processes.
• Google Protocol Buffers for serializing query plans.
• Docker for packaging up executors along with user-defined code.

Ballista can be deployed in Kubernetes, or as a standalone cluster using etcd for discovery.

The following diagram highlights some of the integrations that will be possible with this unique architecture. Note that not all components shown here are available yet.

Ballista differs from Apache Spark in many ways.

• Due to the use of Rust, memory usage can be up to 100x lower than Apache Spark which means that more processing can fit on a single node, reducing the overhead of distributed compute.
• Also due to the use of Rust, there are no "cold start" overheads. The first run of a query can be up to 10x faster than Apache Spark.
• The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors in any programming language with minimal serialization overhead.
• Ballista is columnar rather than row-based, meaning that it can take advantage of vectorized processing both on the CPU (using SIMD) and on the GPU. GPU support isn't available yet but will be available in a future release.

#[tokio::main]
async fn main() -> Result<()> {

    let nyc_taxi_path = "/mnt/nyctaxi/parquet/year=2019";
    let executor_host = "localhost";
    let executor_port = 50051;

    let ctx = Context::remote(executor_host, executor_port, HashMap::new());

    let results = ctx
        .read_parquet(nyc_taxi_path, None)?
        .aggregate(vec![col("passenger_count")], vec![max(col("fare_amount"))])?
        .collect()
        .await?;

    // print the results
    pretty::print_batches(&results)?;

    Ok(())
}

Distributed execution using async Rust has now been proven and we are working towards a 0.3.0 release in August 2020 that will support the following capabilities.

• Projection
• Selection
• Hash Aggregate
• Limit

• Basic aggregate expressions (MIN, MAX, SUM, COUNT, AVG)
• Boolean expressions (AND, OR, NOT)
• Comparison expressions (==, !=, <=, <, >, >=)
• Basic math expressions (+, -, *, /, %)
• Rust user-defined functions (UDFs)
• Java user-defined functions (UDFs)

• CSV
• Parquet

After the 0.3.0 release we will start working on more complex operators, particularly joins, using the TPCH benchmarks to drive requirements. The full roadmap is available here.

The following examples should help illustrate the current capabilities of Ballista

• Distributed query execution in Rust
• TPC-H Benchmark
• Rust bindings for Apache Spark

The user guide is hosted at https://ballistacompute.org, along with the blog where news and release notes are posted.

See CONTRIBUTING.md for information on contributing to this project.