๐ง Build, run, and manage data pipelines for integrating and transforming data.
APACHE-2.0 License
magical
powersEasy developer experienceStart developing locally with a single command or launch a dev environment in your cloud using Terraform.Language of choiceWrite code in Python, SQL, or R in the same data pipeline for ultimate flexibility.Engineering best practices built-inEach step in your pipeline is a standalone file containing modular code thatโs reusable and testable with data validations. No more DAGs with spaghetti code. |
Interactive codeImmediately see results from your codeโs output with an interactive notebook UI.Data is a first-class citizenEach block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.Collaborate on cloudDevelop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment. |
Fast deployDeploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.Scaling made simpleTransform very large datasets directly in your data warehouse or through a native integration with Spark.ObservabilityOperationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI. |
Mage is an open-source data pipeline tool for transforming and integrating data.
The recommended way to install the latest version of Mage is through Docker with the following command:
docker pull mageai/mageai:latest
You can also install Mage using pip or conda, though this may cause dependency issues without the proper environment.
pip install mage-ai
conda install -c conda-forge mage-ai
Looking for help? The fastest way to get started is by checking out our documentation here.
Looking for quick examples? Open a demo project right in your browser or check out our guides.
Build and run a data pipeline with our demo app.
WARNING
The live demo is public to everyone, please donโt save anything sensitive (e.g. passwords, secrets, etc).
Click the image to play video
๐ถ | Orchestration | Schedule and manage data pipelines with observability. |
๐ | Notebook | Interactive Python, SQL, & R editor for coding data pipelines. |
๐๏ธ | Data integrations | Synchronize data from 3rd party sources to your internal destinations. |
๐ฐ | Streaming pipelines | Ingest and transform real-time data. |
โ | dbt | Build, run, and manage your dbt models with Mage. |
A sample data pipeline defined across 3 files โ
@data_loader
def load_csv_from_file():
return pd.read_csv('default_repo/titanic.csv')
@transformer
def select_columns_from_df(df, *args):
return df[['Age', 'Fare', 'Survived']]
@data_exporter
def export_titanic_data_to_disk(df) -> None:
df.to_csv('default_repo/titanic_transformed.csv')
What the data pipeline looks like in the UI โ
New? We recommend reading about blocks and learning from a hands-on tutorial.
Every user experience and technical design decision adheres to these principles.
๐ป | Easy developer experience | Open-source engine that comes with a custom notebook UI for building data pipelines. |
๐ข | Engineering best practices built-in | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |
๐ณ | Data is a first-class citizen | Designed from the ground up specifically for running data-intensive workflows. |
๐ช | Scaling is made simple | Analyze and process large data quickly for rapid iteration. |
These are the fundamental concepts that Mage uses to operate.
Project | Like a repository on GitHub; this is where you write all your code. |
Pipeline | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. |
Block | A file with code that can be executed independently or within a pipeline. |
Data product | Every block produces data after it's been executed. These are called data products in Mage. |
Trigger | A set of instructions that determine when or how a pipeline should run. |
Run | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |
Add features and instantly improve the experience for everyone.
Check out the contributing guide to set up your development environment and start building.
Individually, weโre a mage.
๐ง Mage
Magic is indistinguishable from advanced technology. A mage is someone who uses magic (aka advanced technology). Together, weโre Magers!
๐งโโ๏ธ๐ง Magers (
/หmฤjษr/
)A group of mages who help each other realize their full potential! Letโs hang out and chat together โ
For real-time news, fun memes, data engineering topics, and more, join us on โ
GitHub | |
Slack |
Check out our FAQ page to find answers to some of our most asked questions.
See the LICENSE file for licensing information.