Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
APACHE-2.0 License
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management. It enables data analysts and engineers to transform their data by writing simple SQL queries and YAML configuration while Optimus handles dependency management, scheduling and all other aspects of running transformation jobs at scale.
Discover why users choose Optimus as their main data transformation tool.
Optimus has two components, Optimus service that is the core orchestrator installed on server side, and a CLI binary used to interact with this service. You can install Optimus CLI using homebrew on macOS:
$ brew install raystack/tap/optimus
$ optimus --help
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for
data transformation, data modeling, pipelines, and data quality management.
Usage:
optimus [command]
Available Commands:
backup Backup a resource and its downstream
completion Generate the autocompletion script for the specified shell
extension Operate with extension
help Help about any command
init Interactively initialize Optimus client config
job Interact with schedulable Job
migration Command to do migration activity
namespace Commands that will let the user to operate on namespace
playground Play around with some Optimus features
plugin Manage plugins
project Commands that will let the user to operate on project
resource Interact with data resource
secret Manage secrets to be used in jobs
scheduler Scheduled/run job related functions
serve Starts optimus service
version Print the client version information
Flags:
-h, --help help for optimus
--no-color Disable colored output
Use "optimus [command] --help" for more information about a command.
Explore the following resources to get started with Optimus:
Optimus requires the following dependencies:
Run the following commands to compile optimus
from source
$ git clone [email protected]:raystack/optimus.git
$ cd optimus
$ make
Use the following command to run
$ ./optimus version
Optimus service can be started with
$ ./optimus serve
serve
command has few required configurations that needs to be set for it to start. Read more about it in getting started.
Optimus is currently undergoing heavy development with frequent, breaking API changes. Current major version is zero (v0.x.x) to accommodate rapid development and fast iteration while getting early feedback from users (feedback on APIs are appreciated). The public API could change without a major version update before v1.0.0 release.
Development of Optimus happens in the open on GitHub, and we are grateful to the community for contributing bugfixes and improvements. Read below to learn how you can take part in improving Optimus.
Read our contributing guide to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Optimus.
To help you get your feet wet and get you familiar with our contribution process, we have a list of good first issues that contain bugs which have a relatively limited scope. This is a great place to get started.
Optimus is Apache 2.0 licensed.