Apache Arrow Ballista Distributed Query Engine
APACHE-2.0 License
[!NOTE] The Apache DataFusion Ballista subproject is not currently actively maintained.
We encourage and welcome new contributors and maintainers to join the project. If you are passionate about distributed computing, Rust, or enhancing the performance of data processing frameworks, your contributions could make a significant impact on the future of Ballista. Whether it's fixing bugs, adding new features, or improving documentation, any level of contribution is highly valued.
To get involved, please reach out through our mailing list or join the discussion on our GitHub Issues page. Together, we can continue to advance the project and ensure that Ballista remains a valuable tool for the community.
Ballista is a distributed SQL query engine powered by the Rust implementation of Apache Arrow and Apache Arrow DataFusion.
If you are looking for documentation for a released version of Ballista, please refer to the Ballista User Guide.
Ballista implements a similar design to Apache Spark (particularly Spark SQL), but there are some key differences:
A Ballista cluster consists of one or more scheduler processes and one or more executor processes. These processes can be run as native binaries and are also available as Docker Images, which can be easily deployed with Docker Compose or Kubernetes.
The following diagram shows the interaction between clients and the scheduler for submitting jobs, and the interaction between the executor(s) and the scheduler for fetching tasks and reporting task status.
See the architecture guide for more details.
We run some simple benchmarks comparing Ballista with Apache Spark to track progress with performance optimizations. These are benchmarks derived from TPC-H and not official TPC-H benchmarks. These results are from running individual queries at scale factor 10 (10 GB) on a single node with a single executor and 24 concurrent tasks.
The tracking issue for improving these results is #339.
The easiest way to get started is to run one of the standalone or distributed examples. After that, refer to the Getting Started Guide.
Ballista supports a wide range of SQL, including CTEs, Joins, and Subqueries and can execute complex queries at scale.
Refer to the DataFusion SQL Reference for more information on supported SQL.
Ballista is maturing quickly and is now working towards being production ready. See the roadmap for more details.
Please see the Contribution Guide for information about contributing to Ballista.