mage-ai

๐Ÿง™ Build, run, and manage data pipelines for integrating and transforming data.

APACHE-2.0 License

Downloads
34.7K
Stars
7.3K
Committers
125
View Code on GitHub Visit Website View on X
Ecosystems: Apache Spark, Python

Bot releases are visible (Hide)

mage-ai - 0.9.71 | X-Men โŒ Latest Release

Published by wangxiaoyou1993 5 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿ“ก Add Streaming Oracle Destination

This feature adds a new custom exporter option for OracleDB allowing developers to easily export their streaming data to Oracle DB.

oracledb-exporter

@matrixstone in https://github.com/mage-ai/mage-ai/pull/4896

๐Ÿ‘ค Multi Project Platform Support

This feature adds support for each user to have their own active project within a multi project platform that keeps track of every project for every user.

@dy46 in https://github.com/mage-ai/mage-ai/pull/4865

โš™๏ธ Support Job Parameters in K8s Executor

Adds support to help manage environment variables, and control job execution and cleanup policies within the Kubernetes cluster.

k8s_executor_config:
  container_config:
    image: mageai/mageai:0.9.7
    env:
    - name: USER_CODE_PATH
      value: /home/src/k8s_project
  job:
    active_deadline_seconds: 120
    backoff_limit: 3
    ttl_seconds_after_finished: 86400

@artche in https://github.com/mage-ai/mage-ai/pull/5044

๐Ÿ”‘ Support ODIC_ROLES_MAPPING and user_roles

This feature is used to map roles obtained from an OpenID Connect (OIDC) identity provider to roles used within your application allowing for a standardized way to manage user roles across different identity providers.

@mihaivint in https://github.com/mage-ai/mage-ai/pull/5053 and n https://github.com/mage-ai/mage-ai/pull/4899

๐Ÿ’… Enhancements & Polish

๐Ÿ› Bug Fixes

โœจ New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.70...0.9.71

mage-ai - 0.9.70 | Fallout โ˜ข๏ธ

Published by wangxiaoyou1993 6 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿ’ฅ Add "Last run failed" filter to Pipeline Runs table

This feature adds a new filter option to the Pipeline Runs table, allowing users to easily view and filter pipeline runs that have failed in their most recent execution. This makes it simpler to identify and troubleshoot problematic pipelines.
pipeline runs page

by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4926

๐Ÿ› ๏ธ Add workspace update features

This feature allows users to update the workspace settings (environment variables) and update the workspace main container config from the UI. Being able to update the container config will allow users to update the workspace image as well.

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4951

๐Ÿ”„ Add upsert to MySQL, BQ, and Snowflake

Add ability to upsert data in MySQL, BigQuery, and Snowflake IO classes.
It can be achieved by specifying the following params in the export method:

            unique_conflict_method='UPDATE',
            unique_constraints=['col'],

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4769

###โš ๏ธ Support alerts and limited retries in streaming pipeline
Streaming pipeline used to be retried automatically and infinitely.
With this feature, when retry_config is set in streaming pipeline's metadata.yaml at pipeline level, only retry the streaming pipeline with limited times and send alerts when pipeline fails.
Example retry config

retry_config:
  # Number of retry times
  retries: 3
  # Initial delay (in seconds) before retry. If exponential_backoff is true,
  # the delay time is multiplied by 2 for the next retry
  delay: 5
  # Maximum time between the first attempt and the last retry
  max_delay: 60
  # Whether to use exponential backoff retry
  exponential_backoff: true

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4902

๐Ÿ”‘ Add setting to update user roles on login

This feature adds the UPDATE_ROLES_ON_LOGIN setting which will tell Mage to update the roles every time a user logs in using LDAP or OAuth (that supports roles mapping). If LDAP or OAuth returns no roles, the roles for the user will not be updated.

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4905

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

โœจ New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.68...0.9.70

mage-ai - 0.9.68 | Invincible ๐Ÿฆธโ€โ™‚๏ธ

Published by wangxiaoyou1993 7 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿ Custom python source in Streaming pipeline

This feature allows users to use custom Python source code in their Streaming pipelines, which provides more flexibility and control for advanced users who need to incorporate custom logic or functionality into their streaming pipelines. Here is the code template:

from mage_ai.streaming.sources.base_python import BasePythonSource
from typing import Callable

if 'streaming_source' not in globals():
    from mage_ai.data_preparation.decorators import streaming_source


@streaming_source
class CustomSource(BasePythonSource):
    def init_client(self):
        """
        Implement the logic of initializing the client.
        """

    def batch_read(self, handler: Callable):
        """
        Batch read the messages from the source and use handler to process the messages.
        """
        while True:
            records = []
            # Implement the logic of fetching the records
            if len(records) > 0:
                handler(records)

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4763

๐Ÿ“ฆ OracleDB exporter

Add OracleDB exporter template to standard batch pipeline.
image

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4785

๐Ÿ”ง Server logging format template overriding

This feature enables users to override the default server logging format template in Mage. This provides more flexibility in customizing the logging output, which can be useful for troubleshooting, monitoring, or integrating with external logging systems.
Here is an example setting:

export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'

by @afal007 in https://github.com/mage-ai/mage-ai/pull/4772

๐Ÿ”„ Update pipeline name before pipeline creation

Allow user to edit the pipeline name and add a description/tags before creating a new pipeline. Previously, a randomized name was always used for new pipelines, and then users had to go to the pipeline's settings to update the name.
image

by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4790

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

โœจ New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.66...0.9.68

mage-ai - 0.9.66 | Shogun ๐ŸŽŒ

Published by wangxiaoyou1993 7 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿ—‘๏ธ Delete backfills

  • Add delete button to backfills table that allows user to delete backfills individually.
  • Prevent user from setting a backfill interval unit value less than 1, which could cause issues loading the backfills.

by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4726

๐Ÿชช Support AzureDevOps using Entra ID/AD authentication

If you have an Azure DevOps organization connected to your Active Directory/Microsoft Entra ID tenant, you can enable DevOps in the version control app.

  • First, you will need to enable the Active Directory integration in order for Mage to authenticate your user.
  • Once the Active Directory integration is enabled, you simply need to add your DevOps organization as an environment variable, and you should be able to authenticate with DevOps in the version control app
    • export AZURE_DEVOPS_ORGANIZATION=<organization_name>

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4721

๐Ÿ“…ย  Scheduler status endpoint

Add /api/status endpoint to scheduler when running scheduler in standalone mode.
This endpoint can be used to check the liveness of the scheduler.

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4736

๐Ÿ” Read authentication settings from AWS secrets manager

Add support for reading settings from AWS secrets manager. The existing authentication settings have been moved to using the new Settings class.

In order to set the settings backend, the user will need to add a section to the project metadata. Here is an example to set the setting backend to AWS secrets manager:

settings_backend:
  backend_type: aws_secrets_manager
  prefix: default_repo/settings/

Read the doc for details.

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4674

๐Ÿ’… Enhancements & Polish

๐Ÿ› Bug Fixes

โœจ New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.65...0.9.66

mage-ai - 0.9.65 | Demon Slayer ๐Ÿ— โš”๏ธ

Published by wangxiaoyou1993 8 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿ”” Telegram notification

Support for setting up Telegram notification channel for alerts is now available.
Check out the doc for details

by @luthfir-rahman in https://github.com/mage-ai/mage-ai/pull/4632

๐Ÿ”„ย Runtime variables and concurrency config in Backfills

Backfills support configuring global variables override and concurrency limit is now available.

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4618

โ˜๏ธย Export raw files to S3 and GCS

Using S3 and GCS data exporter templates, you can now directly upload raw files in addition to dataframes.

    bucket_name = 'test-bucket'
    object_key = 'test_file_copy.csv'

    S3.with_config(ConfigFileLoader(config_path, config_profile)).export(
        '/home/src/test_file.csv',
        bucket_name,
        object_key,
    )
    GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
        '/home/src/test_file.csv',
        bucket_name,
        object_key,
    )

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4607

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.64...0.9.65

mage-ai - 0.9.64 | Maestro ๐ŸŽต

Published by wangxiaoyou1993 8 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿ”ญ OpenTelemetry integration

  • Added OpenTelemetry Libraries: Integrated OpenTelemetry Python libraries to instrument the application for telemetry data collection.
    • opentelemetry-api: Provides the API for tracing and metrics.
    • opentelemetry-sdk: The core SDK for OpenTelemetry in Python.
    • opentelemetry-instrumentation: General instrumentation library.
    • opentelemetry-instrumentation-sqlalchemy: Specific for instrumenting SQLAlchemy operations.
    • opentelemetry-instrumentation-tornado: For monitoring Tornado-based HTTP server activities.
  • Environment Variables: Added new environment variables (OTEL_EXPORTER_OTLP_HTTP_ENDPOINT and OTEL_EXPORTER_OTLP_ENDPOINT) for configuring the OpenTelemetry Collector endpoints.
  • Database and HTTP Server Instrumentation: Configured instrumentation for both SQLAlchemy database operations and Tornado HTTP server activities to gather traces and metrics.
  • Check out this doc for more details
    image

by @mfreeman451 in https://github.com/mage-ai/mage-ai/pull/4476

๐ŸฆŠ Add GitLab authentication to version control

Support authenticating with GitLab in version control app.
The feature can be enabled by configuring the following environment variables:

  • GITLAB_HOST
  • GITLAB_CLIENT_ID
  • GITLAB_CLIENT_SECRET

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4534

๐Ÿ”” Discord notification integration

Support setting up Discord notification channel for alerts.
Check out the doc for details

by @luthfir-rahman in https://github.com/mage-ai/mage-ai/pull/4478

๐Ÿ”‘ Add OIDC generic provider

This feature allows users to hook up their OIDC server (e.g. Keycloak ) and use it to authenticate with Mage.
It can be enabled by configuring the following environment variables:

  • OIDC_CLIENT_ID
  • OIDC_CLIENT_SECRET
  • OIDC_DISCOVERY_URL

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4563

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.63...0.9.64

mage-ai - 0.9.63 | Halo ๐Ÿ‘พ

Published by wangxiaoyou1993 8 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿฆ† MotherDuck Support

This one's for all the ducklings out there! In addition to supporting DuckDB, Mage now supports MotherDuck destinations!

By specifying a MOTHERDUCK_TOKEN and adding a md: prefix to your DuckDB database, you can read/write from/to MotherDuck locations! Check it out and get started here.

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4533

๐Ÿค– Support Thick mode in OracleDB

We like our OracleDB connections like we like our pancakes, THICK ๐Ÿฅž. By default, Mage's Oracle client runs in a "Thin" mode which connects directly to Oracle Databaseโ€” this mode does not need Oracle Client libraries. However, some additional functionality is available when they're used.

Now, you can use the "Thick" mode in Mage to connect to OracleDB using the Oracle Client libraries!

Check out our docs to get started or read more about the differences between "Thin" and "Thick" modes here.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4421

๐Ÿ—„๏ธ Show unused block files in file browser

This is one of our most requested features and we're excited to announce that it's finally here! ๐ŸŽ‰

You can now see which files are not being used in your pipeline! This is a great way to clean up your projects and remove any unnecessary files. Check out the gif below to see it in action!

Head over to our docs to learn more!

by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4449

๐Ÿค Import functionality for pipeline zips

Like to share? So do we! You can now import pipelines (via .zip files) in your Mage projects! We're optimistic that this simple improvement will make it easier to share your pipelines or even borrow pipelines from your friends!

by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4453

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.62...0.9.63

mage-ai - 0.9.62 | The Beekeeper ๐Ÿ

Published by wangxiaoyou1993 9 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿงต [Mage IO] Weaviate Integration

๐Ÿค” Building AI apps with Mage? Perfect! Now you can use Weaviate as a data source. Weaviate is an open-source, AI-native vector database that helps developers create intuitive and reliable AI applications. With Mage, you can now read from and write to Weaviate databases! Read more in our docs here.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4158

๐Ÿ” [Mage IO] Alogia Integration

Like search? Us too! That's why we've added support for Alogia as a data source in Mage. Alogia is a powerful search engine that helps you build fast and accurate search experiences. With Mage, you can now read from and write to Alogia! Read more in our docs here.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4198

๐Ÿ’ฅ Dynamic SQL Blocks

Another big improvement to dynamic blocks this week SQL Dynamic Blocks! That's right, you can now create dynamic outputs from SQL blocks in Mage. Previously, blocks had to be Python for dynamic outputs, but no more! This is a big step forward in making Mage more flexible and powerful. Give it a shot today. ๐ŸŽ‰

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4430

๐ŸŒŠ [Kafka Streaming Sources] Offset & Partitions

For our streaming aficionados, we have a few new Kafka features! You can now specify the offset and partitions for Kafka streaming sources. Offsets can be one of: Beginning, End, Int, & Timestamp. This allows users to set specific positions inside a topic to consume data.

by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4226

โ›ด๏ธ Configure Kubernetes Affinity & Tolerations

Last, but certainly not least, we've got some nitty Kubernetes configuration updates! You can now specify affinity and tolerations in your Kubernetes settings. โš“

Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on nodes and label selectors specified in pods. Node affinity allows a pod to specify an affinity (or anti-affinity) towards a group of nodes it can be placed on.

Tolerations are applied to pods and indicate that the pod can be scheduled on nodes with specific taints.

These configurations should help our Kubernetes power users make the most of Mage! ๐Ÿง™

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4407

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.60...0.9.62

mage-ai - 0.9.60 | Yusuke Urameshi

Published by wangxiaoyou1993 9 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐ŸŒŠ [Streaming] Google Cloud Storage Destination

๐ŸŽ‰ Google Cloud users rejoice! Streaming pipelines just even got betterโ€”ย Mage now supports Google Cloud Storage as a streaming destination! Check out the docs here and get started today!

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4340

๐Ÿ‘ทโ€โ™‚๏ธ Overwrite SQL types

For anyone with a data warehouse, listen up! (We assume that's most of you ๐Ÿ˜…)

You can now specify custom column types when exporting to SQL destinations. This is useful when you want to export a dataframe with a column that has a type that is not supported by the default mapping. You can read more about overwriting types here.

Here's an example of an exporter that overwrites column types for a PostgreSQL destination:

@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
    schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'VARCHAR(255)'}

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            allow_reserved_words=True,
            unique_conflict_method='UPDATE',
            unique_constraints=['col'],
            overwrite_types=overwrite_types,
        )

This feature is currently supported for PostgreSQL, Redshift, Trino, MSSQL, MySQL, Clickhouse, and BigQuery.

by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3474

๐Ÿ‘จโ€๐Ÿ’ป [Command Center] Version Control & Files

The Mage Command Center can now be used for version control commands! You can both configure git and run your favorite version control commands directly from the Mage UI. Additionally, you can manage your files via nav and Mage's new file explorer (๐Ÿง™ Arcane Library)!

As a reminder, to enable the command center, you can do so by going to Settings (the wizard icon) and click the Command Center toggle. It can be accessed via โŒ˜ + . (Mac) or Win + . (Windows).

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4273

๐Ÿ‘พ [Command Center] Terminal App

Mage LEGEND @tommydangerous is back at it againโ€” he's implemented a full terminal app into the command center. For those of you with this beta feature enabled, you'll now have a terminal at your fingertips at all times.

Check out the following video for sample usage:

https://github.com/mage-ai/mage-ai/assets/1066980/82db6eb4-ecca-49e4-aa77-f464e6a94620

As a reminder, to enable the command center, you can do so by going to Settings (the wizard icon) and click the Command Center toggle. It can be accessed via โŒ˜ + . (Mac) or Win + . (Windows).

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4365

JSON Logging

A huge shoutout to @dy46 for adding JSON logging to Mage! This will make it easier to parse logs and integrate with other logging tools. Just specify SERVER_LOGGING_FORMAT=json to change the output to something like the following:

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4336

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.59...0.9.60

mage-ai - 0.9.59 | ๐Ÿฒ ๐Ÿ‰ Year of the Dragon

Published by wangxiaoyou1993 9 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

Note: many new features this week are in beta. You can enable them by navigating to your Mage settings and toggling the beta features there.

๐ŸŽฎ Multi-project Platform [BETA]

We've reworked our support for mulitple projects with a new multi-project platform! @tommydangerous is back at it again with this huge feature release, enabling nested projects, custom code paths, cross-project triggering, a split pipeline scheduler, and much more!

https://github.com/mage-ai/mage-ai/assets/59450879/80a88de4-615c-431a-9d8f-1d9e61af523a

If you'd like to try out the multi-project platform, you can check out this repo for a sample structure. Head over to your Mage settings to enable

๐Ÿš€ Command Center [BETA]

Another new & exciting feature this weekโ€” the Mage Command Center. The command center is a floating search bar that can invoke actions like opening files & pages, perform actions within Mage, interacting with the page, and much more!

https://github.com/mage-ai/mage-ai/assets/59450879/a01a7f11-a0c8-43f1-9a44-253202659c75

Enable the Command Center in settings and give it a spin today!

๐Ÿชฃ Bitbucket Version Control

Shout out to @dy46 for continuing to crush the version control integrations!

You can now use Bitbucket as a version control provider! This is a great option for teams that use Bitbucket for their code repositories. To get started, navigate to the Mage Version Control app and select Bitbucket as your provider. You'll be prompted to authenticate with Bitbucket, and then you'll be able to select your repositories. Read more here.

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4176

โ—” Qdrant integration

Mage now supports Qdrant, an open-source vector search engine. Qdrant is a great tool for similarity search, and it can be used for a variety of use cases, including product recommendations, image search, and more. With this update, you can load/export data from Qdrant sources in your batch pipelines! Read more here.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4081

๐Ÿงฑ dbt DX v2 + dbt Upgrade [BETA]

This release contains a huge dbt overhaul ๐Ÿคฏ

Alongside a much-awaited upgrade to dbt 1.7, the dbt developer experience has been completely rebuilt. @tommydangerous has been hard at work crafting a dbt experience that is more intuitive, powerful, and flexible!

Here's a quick demo:

https://github.com/mage-ai/mage-ai/assets/59450879/c6518d08-fd32-4bbb-b271-9aa28409b41d

๐Ÿ› Bug Fixes

SQL blocks

Data integration

Trigger and scheduling

dbt

File browser

Dynamic blocks

Git

Other

๐Ÿ’… Enhancements & Polish

File browser

Raw SQL enhancements

Workspace management

Other

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.50...0.9.59

mage-ai - 0.9.50 | Wonka ๐ŸŽฉ

Published by mattppal 10 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐ŸŒŠ [Data Integration] Dremio Source

๐Ÿฅณ Dremio users, rejoice! Mage now supports Dremio as a data integration source, meaning you can now build data integration pipelines pulling from data lakes and more!

by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4072

๐Ÿƒโ€โ™‚๏ธ Manually run pipeline once in same trigger

This update, courtesy of our frontend engineer, Johnson, is a big one!

First, he added a new button to the Trigger Detail page for running pipelines once in the same trigger! ๐Ÿ‘€

Next, some quality of life improvements:

  • For @once triggers, the trigger does not need to be active status before running the pipeline once using the Run@once button in the Trigger Detail page. Many users have told us this is confusing... No more!
  • The trigger will automatically be updated to active status. However, if the trigger is NOT an @once trigger (e.g. a recurring interval or API trigger), the trigger must be set to active status before manually running the pipeline once from the Trigger Detail page.

Finally, Johnson renamed the "start/pause" trigger on Trigger Detail page to "enable/disable" to be a bit more clear and communicative. Again, we've heard this is a bit misleading, so we did something about it! ๐Ÿ—ฃ๏ธ

by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4133

[Streaming] ActiveMQ Sink

Shruti continues her epic tear of contributing magical โœจ pipelines. In this PR, she's added an ActiveMQ streaming sink to Mage. Apache ActiveMQ is an open source message broker written in Java... and now you can write data there via Mage! ๐Ÿ’ซ

by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/4141

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.48...0.9.50

mage-ai - 0.9.48 | The Boy and the Heron

Published by wangxiaoyou1993 11 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

Incremental data integration in batch pipelines

๐Ÿฅณ Data integrations in batch pipelines now support incremental replication! You can read more here to get started!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4068

[Streaming] RabbitMQ Destination

Another community PR from @shrutimantri adds support for RabbitMQ as a streaming data sink. ๐Ÿ”ฅ

Check it out today with your favorite streaming sources! You can find the configuration reference here.

by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/4041

Chroma integration

Mage now has a ChromaDB IO Class, meaning you can use data loaders and exporters in your batch pipelines to read/write from Chroma sources. You can read more about configuration here or visit Chroma's site to learn more about their vector database.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4017

Bookmark overrides

๐ŸŽŠ If you're creating a trigger on a data integration, you can now override bookmarks with your own custom values!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4073

SQL Block environment variable interpolation

For our fans of SQL blocks, you can now interpolate environment variables directly in your queries!

SELECT 
    '{{ env_var("ENV") }}' AS test
    , '{{ variables("test") }}' AS test2
    , {{ test }} AS test3

This should allow for much greater flexibility in pipelines with SQL!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4076

Additional upstream dependencies for dynamic children

Love dynamic blocks? ๐Ÿค” They dynamic children can now have additional upstream dependencies!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4104

Support caching block output in memory

Previously, pipelines with large Spark DataFrames faced out of heap space errors when persisting block outputs to disk. This PR allows the user to disable persisting output. The feature is only supported in standard batch pipeline (without dynamic blocks) for now.

cache_block_output_in_memory: true
run_pipeline_in_one_process: true

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4127

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.46...0.9.48

mage-ai - 0.9.46 | Wish ๐Ÿช„

Published by wangxiaoyou1993 11 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

โšก๏ธ Spark UI/UX 2.0 for AWS EMR

It's finally here! Mage now comes with a completely revamped custom Spark UI/UX for our AWS EMR users! This is a huge update that comes with a complete overhaul of every element possible for managing your Spark cluster!

Check out the PR for more screenshots and get started today (docs coming soon)!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3997

๐ŸŒŠ Streaming: ActiveMQ Source

๐Ÿ”ฅ @shrutimantri is on fire! In another community PR, she adds streaming support for ActiveMQ as a source! If you're an ActiveMQ user, give it a shot today!

by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/3978

๐Ÿ“ Download Files & Pipelines via the UI

Another community PR, this one from @PopaRares, allows you to download files and pipelines via the right-click menu in the Mage UI.

This will be a game changer for collabrative projects and importing/exporting data from Mage!

by @PopaRares in https://github.com/mage-ai/mage-ai/pull/3813

โœจ Streaming: NATS JetStream Source

A big shoutout to community member @mfreeman451 for adding the NATS JetStream message broker as a Streaming Source in Mage!

by @mfreeman451 in https://github.com/mage-ai/mage-ai/pull/3985

๐ŸŽ Data integration: Kafka destination

@Luishfs is back at it with another destinationโ€” this one for data integration. You can now write DI outputs to a Kafka topic. We can't wait to see what y'all cook up with this one!

by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3976

๐Ÿ›Ÿ Auto-save triggers in code

Last, but certainly not least, Mage is now able to auto-save triggers as code. That means (when enabled) you can update triggers and have them auto-save to your Mage project. WThis should help you keep track of your trigger changes across projects.

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4009

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

mage-ai - 0.9.45 | Yuji Itadori ๐Ÿ‘น

Published by wangxiaoyou1993 11 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿ” New SSO/OAuth providers

With our latest release, Mage now supports SSO/OAuth from not one, but two providersโ€” Okta & Google. Our engineers also thought ahead, laying the groundwork for supporting more providers in the future, too! Check out the docsโ€” Google & Okta.

by @dy46 in https://github.com/mage-ai/mage-ai/pull/3921

๐Ÿ”ฅ Compute management for Apache Spark blocks

Tommy is back at it with another massive PR, this one adding full support for compute management in Apache Spark blocks. For those of you who leverage Spark, this PR will allow you fine-grained control over your compute. Keep an eye out for completely revamped EMR functionality in the near future!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3883

๐Ÿค— HuggingFace AI Client

Mage now supports using different AI models for interfaces within the applicaiton, not just OpenAI! The first we've added is a HuggingFace client... You can now use HuggingFace with Mage's AI functionality! Read more about getting started here.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/3850 and https://github.com/mage-ai/mage-ai/pull/3919

๐Ÿงฑ Azure Databricks Delta Lake Destination

๐ŸŽ‰ Mage now supports Azure Databricks Delta Lake as a destination for data integration pipelines! That means you can write all of your favorite sources to the open, parquet-based storage system on Microsofts cloud infrastructure!

by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3932

๐Ÿ“Š Prometheus Metrics

Prometheus style metrics are a vendor neutral standard based on a pull model. Prometheus-enabled servers output time-series data on a route (usually /metrics), which can be scraped. Being an open standard, most monitoring tools know how to interface with Prometheus metrics (Open Telemetry support them too).

You can learn more about Prometheus here:

This PR enables the basic built-in metrics, which cover the Tornado server (http metrics) and the Python runtime. More metrics can be added in the future! Check out our docs here.

by @adelcast in https://github.com/mage-ai/mage-ai/pull/3910

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.43...0.9.45

mage-ai - 0.9.43 | Attack on Titan ๐Ÿ’ฅ

Published by wangxiaoyou1993 12 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐ŸŒณ Dependency Tree 2.0

This one is hugeโ€” a complete rewrite of our dependency tree functionality! You'll notice an improved appearance and performance in addition to the following:

  • A full right click menu
  • The ability to add blocks between nodes
  • The ability to remove blocks using the menu
  • Improved connectivity through dragging lines
  • Block groups and subgroups
  • The ability to drag and drop blocks to connect
  • The ability to drag and drop groups of blocks
  • Double click to see all connections
  • New block execution animation

This new featureset is so expansive you'll need to upgrade to check it out. See this PR for a walkthough of all the features or update to the latest version to try it out today!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3886

๐Ÿท Apache Pinot Data Loader

Pinot users rejoice!

Community member @shrutimantri has added a brand new data loader for the OLAP datastore! Update to the latest version to try it out.

by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/3898

๐Ÿ“Š Google Sheets Data Loader/Exporters

Who doesn't use Google Sheets?

Mage now supports reading and writing to individual Sheets/Tabs natively! This PR also includes some handy loader/exporter templates to make it easy.

Get started with your Sheets data in Mage today!

by @mattppal in https://github.com/mage-ai/mage-ai/pull/3828

๐Ÿงฑ Redis dependency added to Mage Helm chart

Thanks to community member @sriniarul for adding a Redis dependency to our Helm chart! Mage now supports multiple replicas. They also added a standalone scheduler and webserver option to the chart. It you use Redis you'll want to give this one a look!

by @sriniarul in https://github.com/mage-ai/helm-charts/pull/22

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.41...0.9.43

mage-ai - 0.9.41 | Halloween ๐ŸŽƒ๐Ÿ‘ป

Published by wangxiaoyou1993 12 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

Workspace Lifecycle Management

๐ŸŽ‰ Mage now provides support for managing the workspace lifecycles in Kubernetes! That means you can control how Mage is deployed, start-to-finish, with the following options:

  1. Auto-termination
  2. Pre-start scripts
  3. Post-start scripts

Read more about lifecycle management here and give it a shot today!

by @dy46 in https://github.com/mage-ai/mage-ai/pull/3836

Elasticsearch Data Integration Destination

Mage now supports writing data to Elasticsearch for all of your search & LLM needs! ๐Ÿง™๐Ÿปโ€โ™‚๏ธ

by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3756

Block Detach for Shared Pipelines

Ok, this one is hugeโ€” say you have a block in multiple pipelines, but you need to change the logic in a single block instance... That sounds tricky, right? ๐Ÿค”

Now you can with Block Detach! Simply click # Pipelines on the block, then Detatch to create a clone of the block in your current pipeline!

by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/3816

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.38...0.9.41

mage-ai - 0.9.38 | Goosebumps ๐Ÿ˜ฑ ๐ŸŽƒ

Published by wangxiaoyou1993 12 months ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿง‘โ€๐Ÿ’ป Side-by-side block view

Your Mage development workflow just got a whole lot more efficient. Starting today, you can view blocks side-by-side for twice the editing power! Check this one out to improve your DevEx and make more data magic! Simply click the "side-by-side" icon in the center of the editor to get started!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3804

๐Ÿงฑ Support for dbt-dremio

Dremio users, rejoice! You can now execute dbt models in you lakehouse thanks to support for the dbt-dremio package.

by @dy46 in https://github.com/mage-ai/mage-ai/pull/3760

๐Ÿ‘จ๐Ÿปโ€๐Ÿ’ผ Support Github Enterprise authentication

An often-requested feature, Mage now supports Github Enterprise authentication! ๐Ÿ’ฅ

by @dy46 in https://github.com/mage-ai/mage-ai/pull/3817

๐Ÿค– Support auto termination in workspace

Mage now supports auto-termination checks, which will run once every sixty seconds. This can be used to auto start/stop k8s workspaces. Configure it when creating your workspace to get started!

by @dy46 in https://github.com/mage-ai/mage-ai/pull/3721 and https://github.com/mage-ai/mage-ai/pull/3751

โฐ Show server time in app

This PR adds a new current time display in the top right of the app header. By default, the time display shows in the UTC format, but if you click on the time display, a dropdown menu shows up:

Nice! Frontend UX improvements coming in clutch!

by @anniexcheng in https://github.com/mage-ai/mage-ai/pull/3785

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

๐Ÿ˜Ž New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.35...0.9.38

mage-ai - 0.9.35 | ๐Ÿคนโ€โ™‚๏ธ Loki

Published by wangxiaoyou1993 about 1 year ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿ™‡โ€โ™‚๏ธ The Great Pipeline Unification

Perhaps not as momentus as The Second Great Unification, this unification is much more useful for data pipelines! Data Integration sources & destinations can now be added as blocks in batch pipelines! ๐Ÿคฏ

What does this mean? Using Mage, you can now perform integration (extract), transformation, and loading in the SAME pipeline using Singer sources and your favorite tools (dbt, Python, SQL)! Read more in our docs here.

This is like having Fivetran/Airbyte, dbt, and a jupyter notebook all-in-one WITH engineering best practices built-in!

๐Ÿ‘ฉ๐Ÿปโ€๐Ÿ’ป Interactions - a no-code UI for configuring data pipelines

Another huge update, Mage now let's you build templates to unlock data at scale. You can configure data pipelines that are fully customizable for stakeholders and consumers. Read more about interactions here and get started building today!

This functionality will go a long way for democratizing data pipelines and easing data workloads!

๐Ÿค“ Granular API Permissions

Mage now supports granular API permissions on ANY action. Each permission can grant read and write operations on specific resources (e.g. API endpoints). One or more roles can be assigned to 1 or multiple users.

What does this mean? You can create permissions for your team at the most granular level possible. Mage is now completely governable for ANY action. Read more here.

๐Ÿฅณ NEW MongoDB CDC Streaming Source

Mage now supports MongoDB CDC Streaming Sources! A big thanks to @emincanoguz11 for the contribution!

by @emincanoguz11 in https://github.com/mage-ai/mage-ai/pull/3716

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.34...0.9.35

mage-ai - 0.9.34 | C-3PO

Published by wangxiaoyou1993 about 1 year ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐Ÿง  Add support for dbt-synapse

This is going to "dbt" amazing! Mage now supports the dbt-synapse library, allowing dbt to be executed against the Azure Synapse Data Warehouse. You can read more on the package here. Excited to see our Azure users make use of this one!

@dy46 in https://github.com/mage-ai/mage-ai/pull/3657

โ˜๏ธ Added Google Cloud Pubsub as a sink for streaming pipeline

A big shout out to @pammusankolli, who recently added Google Cloud Pubsub as a sink for data streaming pipelines! If you're not familiar with PubSub, you can read more here. Be sure to check out the docs in Mage to build your next pipeline!

@pammusankolli in https://github.com/mage-ai/mage-ai/pull/3689

๐Ÿค“ Support HTML tags in Markdown blocks

All of our Markdown enthusiasts will appreciate this one! Previously, Mage's markdown blocks only supported images via this format:

![](https://images.photowall.com/products/57215/golden-retriever-puppy.jpg?h=699&q=85)

Now, Mage supports <img> elements with custom sizes in Markdown blocks by providing width and height attributes like so:

<img src="https://images.photowall.com/products/57215/golden-retriever-puppy.jpg?h=699&q=85" alt="drawing" width="200"/>

Here are some examples:

Nice!

@anniexcheng in https://github.com/mage-ai/mage-ai/pull/3692

๐Ÿ” Added Opensearch Destination

Mage now supports OpenSearch as a destination in data integration pipelines! ๐Ÿฅณ

@Luishfs in https://github.com/mage-ai/mage-ai/pull/3719

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.30...0.9.34

mage-ai - 0.9.30 | Cowboy Bebop ๐Ÿค ๐Ÿš€

Published by wangxiaoyou1993 about 1 year ago

What's Changed

๐ŸŽ‰ Exciting New Features

๐ŸŒŠ Streaming: Base Class Overhaul + 8 New Destinations

This. is. huge. With a complete base class re-write, every IO destination is now a streaming destination.

That means you can stream to:

And any future destinations added as an IO base. Huge shutout to @wangxiaoyou1993 on the herculean effort!

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3623

๐Ÿ‘€ Recently viewed pipelines

Some frontend polish now allows you to see your Recently Viewed pipelines from the overview pageโ€” a nice touch!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3633

๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆ Community: GCS Sensors

@pammusankolli deliver's his first contribution by adding a Google Cloud Storage sensor to check if a file exists in a given bucket! Thanks for the addโ€” this will be super useful to our Google platform users!

by @pammusankolli in https://github.com/mage-ai/mage-ai/pull/3651

๐Ÿงฉ Support syncing Git submodules

A solid improvement to our Git Sync functionality, you can now sync submodules, too! Just check the Git Sync settings to enable the feature.

by @dy46 in https://github.com/mage-ai/mage-ai/pull/3593

๐Ÿ”‚ Add always on schedule interval

On user request, we've added an @always_on interval for scheduled triggers. Always on schedules will trigger the new pipeline run as soon as the latest pipeline run is completed.

Once a pipeline run ends, regardless of whether or not it failed or succeeded, it will start a new run. Let us know if you find that valuable!

by @dy46 in https://github.com/mage-ai/mage-ai/pull/3611

๐Ÿ› Bug Fixes

๐Ÿ’… Enhancements & Polish

New Contributors

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.28...0.9.30

Package Rankings
Top 4.77% on Pypi.org
Top 26.57% on Conda-forge.org
Badges
Extracted from project README
Mage quick start demo Ask us questions on Slack Hang out on Slack
Related Projects
megabots

๐Ÿค– State-of-the-art, production ready LLM apps made mega-easy, so you don't have to build them fro...

11 Apr 2023 343
txtai

๐Ÿ’ก All-in-one open-source embeddings database for semantic search, LLM orchestration and language ...

09 Aug 2020 6,914
genaiops-promptflow-template

GenAIOps with Prompt Flow is a "GenAIOps template and guidance" to help you build LLM-infused app...

12 Oct 2023 249
runhouse

The fastest way to iterate and deploy AI workloads on your own infra. Unobtrusive, debuggable, Py...

10 May 2022 707
ydata-profiling

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

09 Jan 2016 12,108
magictables

An advanced Python library designed to simplify data preparation and transformation for data scie...

18 Jul 2024 3
energy-forecasting

๐ŸŒ€ ๐—ง๐—ต๐—ฒ ๐—™๐˜‚๐—น๐—น ๐—ฆ๐˜๐—ฎ๐—ฐ๐—ธ ๐Ÿณ-๐—ฆ๐˜๐—ฒ๐—ฝ๐˜€ ๐— ๐—Ÿ๐—ข๐—ฝ๐˜€ ๐—™๐—ฟ๐—ฎ๐—บ๐—ฒ๐˜„๐—ผ๐—ฟ๐—ธ | ๐—Ÿ๐—ฒ๐—ฎ๐—ฟ๐—ป ๐— ๐—Ÿ๐—˜ & ๐— ๐—Ÿ๐—ข๐—ฝ๐˜€ for free by designing, building and ...

04 Feb 2023 775
DocsGPT

GPT-powered chat for documentation, chat with your documents

02 Feb 2023 14,124