๐ง Build, run, and manage data pipelines for integrating and transforming data.
APACHE-2.0 License
Bot releases are hidden (Show)
This feature adds a new custom exporter option for OracleDB allowing developers to easily export their streaming data to Oracle DB.
@matrixstone in https://github.com/mage-ai/mage-ai/pull/4896
This feature adds support for each user to have their own active project within a multi project platform that keeps track of every project for every user.
@dy46 in https://github.com/mage-ai/mage-ai/pull/4865
Adds support to help manage environment variables, and control job execution and cleanup policies within the Kubernetes cluster.
k8s_executor_config:
container_config:
image: mageai/mageai:0.9.7
env:
- name: USER_CODE_PATH
value: /home/src/k8s_project
job:
active_deadline_seconds: 120
backoff_limit: 3
ttl_seconds_after_finished: 86400
@artche in https://github.com/mage-ai/mage-ai/pull/5044
This feature is used to map roles obtained from an OpenID Connect (OIDC) identity provider to roles used within your application allowing for a standardized way to manage user roles across different identity providers.
@mihaivint in https://github.com/mage-ai/mage-ai/pull/5053 and n https://github.com/mage-ai/mage-ai/pull/4899
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.70...0.9.71
Published by wangxiaoyou1993 6 months ago
This feature adds a new filter option to the Pipeline Runs table, allowing users to easily view and filter pipeline runs that have failed in their most recent execution. This makes it simpler to identify and troubleshoot problematic pipelines.
by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4926
This feature allows users to update the workspace settings (environment variables) and update the workspace main container config from the UI. Being able to update the container config will allow users to update the workspace image as well.
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4951
Add ability to upsert data in MySQL, BigQuery, and Snowflake IO classes.
It can be achieved by specifying the following params in the export method:
unique_conflict_method='UPDATE',
unique_constraints=['col'],
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4769
###โ ๏ธ Support alerts and limited retries in streaming pipeline
Streaming pipeline used to be retried automatically and infinitely.
With this feature, when retry_config
is set in streaming pipeline's metadata.yaml at pipeline level, only retry the streaming pipeline with limited times and send alerts when pipeline fails.
Example retry config
retry_config:
# Number of retry times
retries: 3
# Initial delay (in seconds) before retry. If exponential_backoff is true,
# the delay time is multiplied by 2 for the next retry
delay: 5
# Maximum time between the first attempt and the last retry
max_delay: 60
# Whether to use exponential backoff retry
exponential_backoff: true
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4902
This feature adds the UPDATE_ROLES_ON_LOGIN
setting which will tell Mage to update the roles every time a user logs in using LDAP or OAuth (that supports roles mapping). If LDAP or OAuth returns no roles, the roles for the user will not be updated.
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4905
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.68...0.9.70
Published by wangxiaoyou1993 7 months ago
This feature allows users to use custom Python source code in their Streaming pipelines, which provides more flexibility and control for advanced users who need to incorporate custom logic or functionality into their streaming pipelines. Here is the code template:
from mage_ai.streaming.sources.base_python import BasePythonSource
from typing import Callable
if 'streaming_source' not in globals():
from mage_ai.data_preparation.decorators import streaming_source
@streaming_source
class CustomSource(BasePythonSource):
def init_client(self):
"""
Implement the logic of initializing the client.
"""
def batch_read(self, handler: Callable):
"""
Batch read the messages from the source and use handler to process the messages.
"""
while True:
records = []
# Implement the logic of fetching the records
if len(records) > 0:
handler(records)
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4763
Add OracleDB exporter template to standard batch pipeline.
by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4785
This feature enables users to override the default server logging format template in Mage. This provides more flexibility in customizing the logging output, which can be useful for troubleshooting, monitoring, or integrating with external logging systems.
Here is an example setting:
export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'
by @afal007 in https://github.com/mage-ai/mage-ai/pull/4772
Allow user to edit the pipeline name and add a description/tags before creating a new pipeline. Previously, a randomized name was always used for new pipelines, and then users had to go to the pipeline's settings to update the name.
by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4790
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.66...0.9.68
Published by wangxiaoyou1993 7 months ago
by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4726
If you have an Azure DevOps organization connected to your Active Directory/Microsoft Entra ID tenant, you can enable DevOps in the version control app.
export AZURE_DEVOPS_ORGANIZATION=<organization_name>
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4721
Add /api/status
endpoint to scheduler when running scheduler in standalone mode.
This endpoint can be used to check the liveness of the scheduler.
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4736
Add support for reading settings from AWS secrets manager. The existing authentication settings have been moved to using the new Settings class.
In order to set the settings backend, the user will need to add a section to the project metadata. Here is an example to set the setting backend to AWS secrets manager:
settings_backend:
backend_type: aws_secrets_manager
prefix: default_repo/settings/
Read the doc for details.
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4674
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.65...0.9.66
Published by wangxiaoyou1993 8 months ago
Support for setting up Telegram notification channel for alerts is now available.
Check out the doc for details
by @luthfir-rahman in https://github.com/mage-ai/mage-ai/pull/4632
Backfills support configuring global variables override and concurrency limit is now available.
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4618
Using S3 and GCS data exporter templates, you can now directly upload raw files in addition to dataframes.
bucket_name = 'test-bucket'
object_key = 'test_file_copy.csv'
S3.with_config(ConfigFileLoader(config_path, config_profile)).export(
'/home/src/test_file.csv',
bucket_name,
object_key,
)
GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
'/home/src/test_file.csv',
bucket_name,
object_key,
)
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4607
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.64...0.9.65
Published by wangxiaoyou1993 8 months ago
opentelemetry-api
: Provides the API for tracing and metrics.opentelemetry-sdk
: The core SDK for OpenTelemetry in Python.opentelemetry-instrumentation
: General instrumentation library.opentelemetry-instrumentation-sqlalchemy
: Specific for instrumenting SQLAlchemy operations.opentelemetry-instrumentation-tornado
: For monitoring Tornado-based HTTP server activities.OTEL_EXPORTER_OTLP_HTTP_ENDPOINT
and OTEL_EXPORTER_OTLP_ENDPOINT
) for configuring the OpenTelemetry Collector endpoints.by @mfreeman451 in https://github.com/mage-ai/mage-ai/pull/4476
Support authenticating with GitLab in version control app.
The feature can be enabled by configuring the following environment variables:
GITLAB_HOST
GITLAB_CLIENT_ID
GITLAB_CLIENT_SECRET
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4534
Support setting up Discord notification channel for alerts.
Check out the doc for details
by @luthfir-rahman in https://github.com/mage-ai/mage-ai/pull/4478
This feature allows users to hook up their OIDC server (e.g. Keycloak ) and use it to authenticate with Mage.
It can be enabled by configuring the following environment variables:
OIDC_CLIENT_ID
OIDC_CLIENT_SECRET
OIDC_DISCOVERY_URL
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4563
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.63...0.9.64
Published by wangxiaoyou1993 8 months ago
This one's for all the ducklings out there! In addition to supporting DuckDB, Mage now supports MotherDuck destinations!
By specifying a MOTHERDUCK_TOKEN
and adding a md:
prefix to your DuckDB database, you can read/write from/to MotherDuck locations! Check it out and get started here.
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4533
Thick
mode in OracleDBWe like our OracleDB connections like we like our pancakes, THICK ๐ฅ. By default, Mage's Oracle client runs in a "Thin" mode which connects directly to Oracle Databaseโ this mode does not need Oracle Client libraries. However, some additional functionality is available when they're used.
Now, you can use the "Thick" mode in Mage to connect to OracleDB using the Oracle Client libraries!
Check out our docs to get started or read more about the differences between "Thin" and "Thick" modes here.
by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4421
This is one of our most requested features and we're excited to announce that it's finally here! ๐
You can now see which files are not being used in your pipeline! This is a great way to clean up your projects and remove any unnecessary files. Check out the gif below to see it in action!
Head over to our docs to learn more!
by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4449
Like to share? So do we! You can now import pipelines (via .zip
files) in your Mage projects! We're optimistic that this simple improvement will make it easier to share your pipelines or even borrow pipelines from your friends!
by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4453
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.62...0.9.63
Published by wangxiaoyou1993 9 months ago
๐ค Building AI apps with Mage? Perfect! Now you can use Weaviate as a data source. Weaviate is an open-source, AI-native vector database that helps developers create intuitive and reliable AI applications. With Mage, you can now read from and write to Weaviate databases! Read more in our docs here.
by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4158
Like search? Us too! That's why we've added support for Alogia as a data source in Mage. Alogia is a powerful search engine that helps you build fast and accurate search experiences. With Mage, you can now read from and write to Alogia! Read more in our docs here.
by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4198
Another big improvement to dynamic blocks this week SQL Dynamic Blocks! That's right, you can now create dynamic outputs from SQL blocks in Mage. Previously, blocks had to be Python for dynamic outputs, but no more! This is a big step forward in making Mage more flexible and powerful. Give it a shot today. ๐
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4430
For our streaming aficionados, we have a few new Kafka features! You can now specify the offset and partitions for Kafka streaming sources. Offsets can be one of: Beginning
, End
, Int
, & Timestamp
. This allows users to set specific positions inside a topic to consume data.
by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4226
Last, but certainly not least, we've got some nitty Kubernetes configuration updates! You can now specify affinity
and tolerations
in your Kubernetes settings. โ
Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on nodes and label selectors specified in pods. Node affinity allows a pod to specify an affinity (or anti-affinity) towards a group of nodes it can be placed on.
Tolerations are applied to pods and indicate that the pod can be scheduled on nodes with specific taints.
These configurations should help our Kubernetes power users make the most of Mage! ๐ง
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4407
global_vars
context in pipeline executor by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4435
write_pandas
issue by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4395
/pipelines
by @edmondwinston in https://github.com/mage-ai/mage-ai/pull/4306
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.60...0.9.62
Published by wangxiaoyou1993 9 months ago
๐ Google Cloud users rejoice! Streaming pipelines just even got betterโย Mage now supports Google Cloud Storage as a streaming destination! Check out the docs here and get started today!
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4340
For anyone with a data warehouse, listen up! (We assume that's most of you ๐ )
You can now specify custom column types when exporting to SQL destinations. This is useful when you want to export a dataframe with a column that has a type that is not supported by the default mapping. You can read more about overwriting types here.
Here's an example of an exporter that overwrites column types for a PostgreSQL destination:
@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
schema_name = 'your_schema_name' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
allow_reserved_words=True,
unique_conflict_method='UPDATE',
unique_constraints=['col'],
overwrite_types=overwrite_types,
)
This feature is currently supported for PostgreSQL, Redshift, Trino, MSSQL, MySQL, Clickhouse, and BigQuery.
by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3474
The Mage Command Center can now be used for version control commands! You can both configure git and run your favorite version control commands directly from the Mage UI. Additionally, you can manage your files via nav and Mage's new file explorer (๐ง Arcane Library)!
As a reminder, to enable the command center, you can do so by going to Settings
(the wizard icon) and click the Command Center
toggle. It can be accessed via โ + .
(Mac) or Win + .
(Windows).
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4273
Mage LEGEND @tommydangerous is back at it againโ he's implemented a full terminal app into the command center. For those of you with this beta feature enabled, you'll now have a terminal at your fingertips at all times.
Check out the following video for sample usage:
https://github.com/mage-ai/mage-ai/assets/1066980/82db6eb4-ecca-49e4-aa77-f464e6a94620
As a reminder, to enable the command center, you can do so by going to Settings
(the wizard icon) and click the Command Center
toggle. It can be accessed via โ + .
(Mac) or Win + .
(Windows).
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4365
A huge shoutout to @dy46 for adding JSON logging to Mage! This will make it easier to parse logs and integrate with other logging tools. Just specify SERVER_LOGGING_FORMAT=json
to change the output to something like the following:
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4336
custom_fields
to freshdesk source by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4354
auth_type
enum by @dy46 in https://github.com/mage-ai/mage-ai/pull/4376
google-ads
lib by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4330
emr_config
if using EMR cluster by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4379
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.59...0.9.60
Published by wangxiaoyou1993 9 months ago
Note: many new features this week are in beta. You can enable them by navigating to your Mage settings and toggling the beta features there.
We've reworked our support for mulitple projects with a new multi-project platform! @tommydangerous is back at it again with this huge feature release, enabling nested projects, custom code paths, cross-project triggering, a split pipeline scheduler, and much more!
https://github.com/mage-ai/mage-ai/assets/59450879/80a88de4-615c-431a-9d8f-1d9e61af523a
If you'd like to try out the multi-project platform, you can check out this repo for a sample structure. Head over to your Mage settings to enable
Another new & exciting feature this weekโ the Mage Command Center. The command center is a floating search bar that can invoke actions like opening files & pages, perform actions within Mage, interacting with the page, and much more!
https://github.com/mage-ai/mage-ai/assets/59450879/a01a7f11-a0c8-43f1-9a44-253202659c75
Enable the Command Center in settings and give it a spin today!
Shout out to @dy46 for continuing to crush the version control integrations!
You can now use Bitbucket as a version control provider! This is a great option for teams that use Bitbucket for their code repositories. To get started, navigate to the Mage Version Control app and select Bitbucket as your provider. You'll be prompted to authenticate with Bitbucket, and then you'll be able to select your repositories. Read more here.
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4176
Mage now supports Qdrant, an open-source vector search engine. Qdrant is a great tool for similarity search, and it can be used for a variety of use cases, including product recommendations, image search, and more. With this update, you can load/export data from Qdrant sources in your batch pipelines! Read more here.
by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4081
This release contains a huge dbt overhaul ๐คฏ
Alongside a much-awaited upgrade to dbt 1.7, the dbt developer experience has been completely rebuilt. @tommydangerous has been hard at work crafting a dbt experience that is more intuitive, powerful, and flexible!
Here's a quick demo:
https://github.com/mage-ai/mage-ai/assets/59450879/c6518d08-fd32-4bbb-b271-9aa28409b41d
query_vars
as a dict by @dy46 in https://github.com/mage-ai/mage-ai/pull/4280
TIME
type by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4275
yaml
files by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4207
requestIdleCallback
not supported on Safari by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4284
GitBranchPolicy
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4213
statistics.json
in correct execution partition folder by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4271
queue_subscription
support for NATS Jetstream streaming source by @atjhoendz in https://github.com/mage-ai/mage-ai/pull/4169
role_arn
as an optional config for AWS DynamoDB source by @robreinhold in https://github.com/mage-ai/mage-ai/pull/4256
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.50...0.9.59
Published by mattppal 10 months ago
๐ฅณ Dremio users, rejoice! Mage now supports Dremio as a data integration source, meaning you can now build data integration pipelines pulling from data lakes and more!
by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4072
This update, courtesy of our frontend engineer, Johnson, is a big one!
First, he added a new button to the Trigger Detail page for running pipelines once in the same trigger! ๐
Next, some quality of life improvements:
@once
triggers, the trigger does not need to be active
status before running the pipeline once using the Run@once
button in the Trigger Detail page. Many users have told us this is confusing... No more!active
status. However, if the trigger is NOT an @once
trigger (e.g. a recurring interval or API trigger), the trigger must be set to active
status before manually running the pipeline once from the Trigger Detail page.Finally, Johnson renamed the "start/pause" trigger on Trigger Detail page to "enable/disable" to be a bit more clear and communicative. Again, we've heard this is a bit misleading, so we did something about it! ๐ฃ๏ธ
by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4133
Shruti continues her epic tear of contributing magical โจ pipelines. In this PR, she's added an ActiveMQ streaming sink to Mage. Apache ActiveMQ is an open source message broker written in Java... and now you can write data there via Mage! ๐ซ
by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/4141
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.48...0.9.50
Published by wangxiaoyou1993 11 months ago
๐ฅณ Data integrations in batch pipelines now support incremental replication! You can read more here to get started!
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4068
Another community PR from @shrutimantri adds support for RabbitMQ as a streaming data sink. ๐ฅ
Check it out today with your favorite streaming sources! You can find the configuration reference here.
by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/4041
Mage now has a ChromaDB IO Class, meaning you can use data loaders and exporters in your batch pipelines to read/write from Chroma sources. You can read more about configuration here or visit Chroma's site to learn more about their vector database.
by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4017
๐ If you're creating a trigger on a data integration, you can now override bookmarks with your own custom values!
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4073
For our fans of SQL blocks, you can now interpolate environment variables directly in your queries!
SELECT
'{{ env_var("ENV") }}' AS test
, '{{ variables("test") }}' AS test2
, {{ test }} AS test3
This should allow for much greater flexibility in pipelines with SQL!
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4076
Love dynamic blocks? ๐ค They dynamic children can now have additional upstream dependencies!
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4104
Previously, pipelines with large Spark DataFrames faced out of heap space errors when persisting block outputs to disk. This PR allows the user to disable persisting output. The feature is only supported in standard batch pipeline (without dynamic blocks) for now.
cache_block_output_in_memory: true
run_pipeline_in_one_process: true
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4127
opentelemetry-exporter-prometheus
package version by @dy46 in https://github.com/mage-ai/mage-ai/pull/4101
message_events_json
in Postmark messages_outbound stream by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4085
column_header_format
option by @dy46 in https://github.com/mage-ai/mage-ai/pull/4118
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.46...0.9.48
Published by wangxiaoyou1993 11 months ago
It's finally here! Mage now comes with a completely revamped custom Spark UI/UX for our AWS EMR users! This is a huge update that comes with a complete overhaul of every element possible for managing your Spark cluster!
Check out the PR for more screenshots and get started today (docs coming soon)!
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3997
๐ฅ @shrutimantri is on fire! In another community PR, she adds streaming support for ActiveMQ as a source! If you're an ActiveMQ user, give it a shot today!
by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/3978
Another community PR, this one from @PopaRares, allows you to download files and pipelines via the right-click menu in the Mage UI.
This will be a game changer for collabrative projects and importing/exporting data from Mage!
by @PopaRares in https://github.com/mage-ai/mage-ai/pull/3813
A big shoutout to community member @mfreeman451 for adding the NATS JetStream message broker as a Streaming Source in Mage!
by @mfreeman451 in https://github.com/mage-ai/mage-ai/pull/3985
@Luishfs is back at it with another destinationโ this one for data integration. You can now write DI outputs to a Kafka topic. We can't wait to see what y'all cook up with this one!
by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3976
Last, but certainly not least, Mage is now able to auto-save triggers as code. That means (when enabled) you can update triggers and have them auto-save to your Mage project. WThis should help you keep track of your trigger changes across projects.
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4009
pyarrow-hotfix
to requirements.txt
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3993
REQUIRE_USER_PERMISSIONS
by @dy46 in https://github.com/mage-ai/mage-ai/pull/4007
run_pipeline_in_one_process
is true by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4046
Published by wangxiaoyou1993 11 months ago
With our latest release, Mage now supports SSO/OAuth from not one, but two providersโ Okta & Google. Our engineers also thought ahead, laying the groundwork for supporting more providers in the future, too! Check out the docsโ Google & Okta.
by @dy46 in https://github.com/mage-ai/mage-ai/pull/3921
Tommy is back at it with another massive PR, this one adding full support for compute management in Apache Spark blocks. For those of you who leverage Spark, this PR will allow you fine-grained control over your compute. Keep an eye out for completely revamped EMR functionality in the near future!
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3883
Mage now supports using different AI models for interfaces within the applicaiton, not just OpenAI! The first we've added is a HuggingFace client... You can now use HuggingFace with Mage's AI functionality! Read more about getting started here.
by @matrixstone in https://github.com/mage-ai/mage-ai/pull/3850 and https://github.com/mage-ai/mage-ai/pull/3919
๐ Mage now supports Azure Databricks Delta Lake as a destination for data integration pipelines! That means you can write all of your favorite sources to the open, parquet-based storage system on Microsofts cloud infrastructure!
by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3932
Prometheus style metrics are a vendor neutral standard based on a pull model. Prometheus-enabled servers output time-series data on a route (usually /metrics), which can be scraped. Being an open standard, most monitoring tools know how to interface with Prometheus metrics (Open Telemetry support them too).
You can learn more about Prometheus here:
This PR enables the basic built-in metrics, which cover the Tornado server (http metrics) and the Python runtime. More metrics can be added in the future! Check out our docs here.
by @adelcast in https://github.com/mage-ai/mage-ai/pull/3910
INCREMENTAL
run and updated tap by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3835
while
loop when not raising error on failure for pipeline triggered by code by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/3933
dict
column to json
if column type is string
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3969
io_config.yaml
by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/3970
duckdb
version by @dy46 in https://github.com/mage-ai/mage-ai/pull/3959
assignPublicIp
and enableExecuteCommand
in EcsConfig by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3966
api_version
to the data loader and data exporter templates by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/3967
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.43...0.9.45
Published by wangxiaoyou1993 11 months ago
This one is hugeโ a complete rewrite of our dependency tree functionality! You'll notice an improved appearance and performance in addition to the following:
This new featureset is so expansive you'll need to upgrade to check it out. See this PR for a walkthough of all the features or update to the latest version to try it out today!
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3886
Pinot users rejoice!
Community member @shrutimantri has added a brand new data loader for the OLAP datastore! Update to the latest version to try it out.
by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/3898
Who doesn't use Google Sheets?
Mage now supports reading and writing to individual Sheets/Tabs natively! This PR also includes some handy loader/exporter templates to make it easy.
Get started with your Sheets data in Mage today!
by @mattppal in https://github.com/mage-ai/mage-ai/pull/3828
Thanks to community member @sriniarul for adding a Redis dependency to our Helm chart! Mage now supports multiple replicas. They also added a standalone scheduler and webserver option to the chart. It you use Redis you'll want to give this one a look!
by @sriniarul in https://github.com/mage-ai/helm-charts/pull/22
LIST
API performance by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3903
UPDATE
API by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3909
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.41...0.9.43
Published by wangxiaoyou1993 12 months ago
๐ Mage now provides support for managing the workspace lifecycles in Kubernetes! That means you can control how Mage is deployed, start-to-finish, with the following options:
Read more about lifecycle management here and give it a shot today!
by @dy46 in https://github.com/mage-ai/mage-ai/pull/3836
Mage now supports writing data to Elasticsearch for all of your search & LLM needs! ๐ง๐ปโโ๏ธ
by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3756
Ok, this one is hugeโ say you have a block in multiple pipelines, but you need to change the logic in a single block instance... That sounds tricky, right? ๐ค
Now you can with Block Detach! Simply click # Pipelines on the block, then Detatch to create a clone of the block in your current pipeline!
by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/3816
libodbc
conflicts in Dockerfile by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3840 and https://github.com/mage-ai/mage-ai/pull/3845
pymssql
dependency by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3859
authorize_query
check by @dy46 in https://github.com/mage-ai/mage-ai/pull/3846
ctrl/cmd + click
keyboard shortcut for selecting pipeline run rows by @anniexcheng in https://github.com/mage-ai/mage-ai/pull/3843
text/csv
response type in API source by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3864
aws_session_token
support to get_aws_boto3_client
by @nyc-de in https://github.com/mage-ai/mage-ai/pull/3877
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.38...0.9.41
Published by wangxiaoyou1993 12 months ago
Your Mage development workflow just got a whole lot more efficient. Starting today, you can view blocks side-by-side for twice the editing power! Check this one out to improve your DevEx and make more data magic! Simply click the "side-by-side" icon in the center of the editor to get started!
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3804
dbt-dremio
Dremio users, rejoice! You can now execute dbt models in you lakehouse thanks to support for the dbt-dremio
package.
by @dy46 in https://github.com/mage-ai/mage-ai/pull/3760
An often-requested feature, Mage now supports Github Enterprise authentication! ๐ฅ
by @dy46 in https://github.com/mage-ai/mage-ai/pull/3817
Mage now supports auto-termination checks, which will run once every sixty seconds. This can be used to auto start/stop k8s workspaces. Configure it when creating your workspace to get started!
by @dy46 in https://github.com/mage-ai/mage-ai/pull/3721 and https://github.com/mage-ai/mage-ai/pull/3751
This PR adds a new current time display in the top right of the app header. By default, the time display shows in the UTC format, but if you click on the time display, a dropdown menu shows up:
Nice! Frontend UX improvements coming in clutch!
by @anniexcheng in https://github.com/mage-ai/mage-ai/pull/3785
DISABLE_NOTEBOOK_EDIT_ACCESS
variable by @dy46 in https://github.com/mage-ai/mage-ai/pull/3796
listdir
recursive bug by @luizarvo in https://github.com/mage-ai/mage-ai/pull/3807
IndexError
in upstream dbt block by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3808
TIMESTAMPTZ
column type for datetimes by @eofs in https://github.com/mage-ai/mage-ai/pull/3814
np.ndarray
before exporting to postgres by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3821
db_connection
session before interpolating repo config by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3824
WorkspaceConfig
default by @dy46 in https://github.com/mage-ai/mage-ai/pull/3784
mage-repo
remote by @dy46 in https://github.com/mage-ai/mage-ai/pull/3806
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.35...0.9.38
Published by wangxiaoyou1993 about 1 year ago
Perhaps not as momentus as The Second Great Unification, this unification is much more useful for data pipelines! Data Integration sources & destinations can now be added as blocks in batch pipelines! ๐คฏ
What does this mean? Using Mage, you can now perform integration (extract), transformation, and loading in the SAME pipeline using Singer sources and your favorite tools (dbt, Python, SQL)! Read more in our docs here.
This is like having Fivetran/Airbyte, dbt, and a jupyter notebook all-in-one WITH engineering best practices built-in!
Another huge update, Mage now let's you build templates to unlock data at scale. You can configure data pipelines that are fully customizable for stakeholders and consumers. Read more about interactions here and get started building today!
This functionality will go a long way for democratizing data pipelines and easing data workloads!
Mage now supports granular API permissions on ANY action. Each permission can grant read and write operations on specific resources (e.g. API endpoints). One or more roles can be assigned to 1 or multiple users.
What does this mean? You can create permissions for your team at the most granular level possible. Mage is now completely governable for ANY action. Read more here.
Mage now supports MongoDB CDC Streaming Sources! A big thanks to @emincanoguz11 for the contribution!
by @emincanoguz11 in https://github.com/mage-ai/mage-ai/pull/3716
disable_double_quotes
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3761
datetime
hover rather than UTC by @anniexcheng in https://github.com/mage-ai/mage-ai/pull/3739
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.34...0.9.35
Published by wangxiaoyou1993 about 1 year ago
dbt-synapse
This is going to "dbt" amazing! Mage now supports the dbt-synapse
library, allowing dbt to be executed against the Azure Synapse Data Warehouse. You can read more on the package here. Excited to see our Azure users make use of this one!
@dy46 in https://github.com/mage-ai/mage-ai/pull/3657
A big shout out to @pammusankolli, who recently added Google Cloud Pubsub as a sink for data streaming pipelines! If you're not familiar with PubSub, you can read more here. Be sure to check out the docs in Mage to build your next pipeline!
@pammusankolli in https://github.com/mage-ai/mage-ai/pull/3689
All of our Markdown enthusiasts will appreciate this one! Previously, Mage's markdown blocks only supported images via this format:
![](https://images.photowall.com/products/57215/golden-retriever-puppy.jpg?h=699&q=85)
Now, Mage supports <img>
elements with custom sizes in Markdown blocks by providing width
and height
attributes like so:
<img src="https://images.photowall.com/products/57215/golden-retriever-puppy.jpg?h=699&q=85" alt="drawing" width="200"/>
Here are some examples:
Nice!
@anniexcheng in https://github.com/mage-ai/mage-ai/pull/3692
Mage now supports OpenSearch as a destination in data integration pipelines! ๐ฅณ
@Luishfs in https://github.com/mage-ai/mage-ai/pull/3719
.preferences.yaml
file doesn't exist by @dy46 in https://github.com/mage-ai/mage-ai/pull/3693
UPDATE
command in raw SQL command by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3705
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.30...0.9.34
Published by wangxiaoyou1993 about 1 year ago
This. is. huge. With a complete base class re-write, every IO destination is now a streaming destination.
That means you can stream to:
And any future destinations added as an IO base. Huge shutout to @wangxiaoyou1993 on the herculean effort!
by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3623
Some frontend polish now allows you to see your Recently Viewed pipelines from the overview pageโ a nice touch!
by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3633
@pammusankolli deliver's his first contribution by adding a Google Cloud Storage sensor to check if a file exists in a given bucket! Thanks for the addโ this will be super useful to our Google platform users!
by @pammusankolli in https://github.com/mage-ai/mage-ai/pull/3651
A solid improvement to our Git Sync functionality, you can now sync submodules, too! Just check the Git Sync settings to enable the feature.
by @dy46 in https://github.com/mage-ai/mage-ai/pull/3593
On user request, we've added an @always_on
interval for scheduled triggers. Always on schedules will trigger the new pipeline run as soon as the latest pipeline run is completed.
Once a pipeline run ends, regardless of whether or not it failed or succeeded, it will start a new run. Let us know if you find that valuable!
by @dy46 in https://github.com/mage-ai/mage-ai/pull/3611
Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.28...0.9.30