ZenML 🙏: Build portable, production-ready MLOps pipelines. https://zenml.io.
APACHE-2.0 License
Bot releases are hidden (Show)
Published by schustmi almost 2 years ago
This pre-release removes the ml-pipeline-sdk
dependency and adds support for Python 3.10 as well as M1 macs.
Note: Installing and using this will migrate your ZenML metadata database
tables without any way of downgrading to previous releases. Use at your own risk.
Published by schustmi almost 2 years ago
This release comes with a brand-new Neptune integration to track your ML experiments
as well as lots of performance improvements!
The new Neptune integration
includes a Neptune experiment tracker component that allows you to track your machine learning experiments
using Neptune.
The 0.20.0 release introduced our new server but brought with it a few performance and scalability issues.
Since then we've made many improvements to it, and this release is the final and biggest boost in performance. We reduced the amount of server calls needed for almost all CLI commands and greatly improved the speed of the dashboard as well.
We've removed PyArrow as a dependency of the zenml
python package.
As a consequence of that, our NumPy and Pandas materializer no
longer read and write their artifacts using PyArrow but instead use
native formats instead. If you still want to use PyArrow to serialize
your NumPy arrays and Pandas dataframes, you'll need to install it manually
like this: pip install pyarrow
In future releases we'll get rid of other unnecessary dependencies to
further slim down the zenml
package.
The following changes introduces with this release mey require some manual
intervention to update your current installations:
Client
class, it might need to bezenml stack-component remove-attribute <COMPONENT_NAME> <ATTRIBUTE_NAME>
cleanup_step_run
method,step_failed
parameter.zenml status
by @schustmi in https://github.com/zenml-io/zenml/pull/1084
Full Changelog: https://github.com/zenml-io/zenml/compare/0.22.0...0.23.0
Published by schustmi almost 2 years ago
This release comes with a new BentoML integration as well as a reworked Airflow orchestrator. Additionally, it greatly improves the server performance as well as other small fixes and updates to our docs!
The new BentoML integration includes a BentoML model deployer component that allows you to deploy your models from any of the major machine learning frameworks on your local machine.
See example here.
The previous Airflow orchestrator was limited to running locally and had many additional unpleasant constraints that made it hard to work with. This release includes a completely rewritten, new version of the Airflow orchestrator that now relies on Docker images to run your pipelines and works both locally and with remote Airflow deployments.
See what changed in this video and check out the brand-new example here.
The following changes introduces with this release mey require some manual
intervention to update your current installations:
zenml integration install airflow
to upgrade) and Docker installedAWSSecretsManager
by @chiragjn in https://github.com/zenml-io/zenml/pull/1057
step_run_artifact
table to step_run_input_artifact
by @fa9r in https://github.com/zenml-io/zenml/pull/1075
Full Changelog: https://github.com/zenml-io/zenml/compare/0.21.1...0.22.0
Published by stefannica almost 2 years ago
This is an ad-hoc release to fix some bugs introduced the 0.21.0 release that
made the local ZenML dashboard unusable.
pip install [server]
by @strickvl in https://github.com/zenml-io/zenml/pull/1037
Full Changelog: https://github.com/zenml-io/zenml/compare/0.21.0...0.21.1
Published by stefannica almost 2 years ago
This release primarily fixes a number of bugs that were introduced as part of
the 0.20.0 ZenServer release. These significantly improve the stability when
using ZenML with the ZenML Server.
Notable fixes include:
New features and other improvements:
zenml pipeline runs export / import / migrate
CLI commands are nowThe following changes introduces with this release mey require some manual
intervention to update your current installations:
values.yaml
file has been restructured to makevalues.yaml
copies accordingly.zenml integration install azure
to update your current installation, ifFileNotFoundError
with remote path in HuggingFace Dataset materializer by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/995
zenml pipeline runs export / import / migrate
CLI commands by @fa9r in https://github.com/zenml-io/zenml/pull/977
pip install zenml[server]
to relevant examples by @dnth in https://github.com/zenml-io/zenml/pull/1027
Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.5...0.21.0
Published by fa9r almost 2 years ago
ZenML 0.20.5 fixes another series of minor bugs, significantly improves the performance of the CLI, and adds an option to specify APT packages in Docker images.
zenml==0.20.4
by @dnth in https://github.com/zenml-io/zenml/pull/987
zenml up
command by @dnth in https://github.com/zenml-io/zenml/pull/978
just_mine
flag for zenml stack list
by @strickvl in https://github.com/zenml-io/zenml/pull/979
MissingStepParameterError
exception message by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/996
core
and integration
docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/983
Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.4...0.20.5
Published by htahir1 about 2 years ago
This release fixes another series of minor bugs that were introduced in 0.20.0.
gpu_limit
condition in VertexOrchestrator
by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/963
Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.3...0.20.4
Published by stefannica about 2 years ago
This release fixes another series of minor bugs that were introduced in 0.20.0 and add full support docs for Kubeflow multi-tenancy deployments.
Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.2...0.20.3
Published by fa9r about 2 years ago
After a successful release of the new ZenML server and dashboard paradigm, we set to ironing out some bugs that slipped through.
docstring
Attribute of StepModel by @fa9r in https://github.com/zenml-io/zenml/pull/936
VertexOrchestrator
apply node selector constraint if gpu_limit > 0
by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/935
Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.1...0.20.2
Published by stefannica about 2 years ago
The ZenML 0.20.0 release brings a number of big changes to its architecture and a lot of cool new features, some of which are not backwards compatible with previous versions.
These changes are only covered briefly in the release notes. For a detailed view on what happened and how you can get the most out of the 0.20.0 release, please head over to our "ZenML 0.20.0: Our Biggest Release Yet" blog post.
Updating to ZenML 0.20.0 needs to be followed by a migration of your existing ZenML Stacks and you may also need to make changes to your current ZenML pipeline code. Please read the migration guide carefully and follow the instructions to ensure a smooth transition. The guide walks you through these changes and offers instructions on how to migrate your existing ZenML stacks and pipelines to the new version with minimal effort and disruption to your existing workloads.
If you have updated to ZenML 0.20.0 by mistake or are experiencing issues with the new version, you can always go back to the previous version by using pip install zenml==0.13.2
instead of pip install zenml
when installing ZenML manually or in your scripts.
kserve
README by @strickvl in https://github.com/zenml-io/zenml/pull/912
Click<8.0.0
now by @AlexejPenner in https://github.com/zenml-io/zenml/pull/920
Full Changelog: https://github.com/zenml-io/zenml/compare/0.13.2...0.20.0
Published by schustmi about 2 years ago
ZenML 0.13.2 comes with a new local Docker orchestrator and many other improvements and fixes:
Full Changelog: https://github.com/zenml-io/zenml/compare/0.13.1...0.13.2
Published by fa9r about 2 years ago
ZenML 0.13.1 is here and it comes with several quality of life improvements:
step_b.after(step_a)
This release also contains a lot of documentation on how to deploy
custom code (like preprocessing and postprocessing code) with our KServe and
Seldon integrations.
Full Changelog: https://github.com/zenml-io/zenml/compare/0.13.0...0.13.1
Published by schustmi about 2 years ago
ZenML version 0.13.0 is chock-full with exciting features.
Custom Code Deployment is the
continuation of the Model Deployment story that we have been working on over the last few releases. Now
it is possible to deploy custom code along with your models using Kserve or Seldon.
With Spark this release also brings distributed processing into the ZenML toolkit.
Spinning up and configuring infrastructure is a difficult part of the MLOps journey
and can easily become a barrier to entry. Using our mlops-stacks
repository, it is now possible to spin up perfectly configured infrastructure with
the corresponding ZenML stack using the ZenML CLI.
As always, we've also included various bug fixes and lots of improvements to the documentation and our examples.
This release introduces a breaking change to the CLI by adjusting the access to
the stack component specific resources for secret-managers
and
model-deployers
to be more explicitly linked to the component. Here is how:
# `zenml secret register ...` becomes
zenml secrets-manager secret register ...
# `zenml served_models list` becomes
zenml model-deployer models list
config.yaml
by @strickvl in https://github.com/zenml-io/zenml/pull/827
secret
, feature
and served-models
) by @strickvl in https://github.com/zenml-io/zenml/pull/833
Full Changelog: https://github.com/zenml-io/zenml/compare/0.12.0...0.13.0
Published by schustmi about 2 years ago
The 0.12.0 release comes with the third implementation of the ZenML Model Deployer abstraction: The KServe integration allows you to deploy any PyTorch, TensorFlow or SKLearn from within your ZenML pipelines!
We also added functionality to specify hardware resources on a step level to control the amount of memory, CPUs and GPUs that each ZenML step has access to. This is curretly limited to the Kubeflow and Vertex orchestrator but will be expanded in upcoming releases.
Additionally, we've added support for scoped secrets in our AWS, GCP and Vault Secrets Managers. These updated Secrets Managers allow you to configure a scope which determines if secrets are shared with other ZenML Secrets Managers using the same backend.
As always, we've also included various bug fixes and lots of improvements to the documentation and our examples.
Full Changelog: https://github.com/zenml-io/zenml/compare/0.11.0...0.12.0
Published by stefannica about 2 years ago
Our 0.11.0 release contains our new annotation workflow and stack component. We've been blogging about this for a few weeks, and even started maintaining our own repository of open-source annotation tools. With ZenML 0.11.0 you can bring data labeling into your MLOps pipelines and workflows as a first-class citizen. We've started our first iteration of this functionality by integrating with Label Studio, a leader in the open-source annotation tool space.
This release also includes a ton of updates to our documentation. (Seriously, go check them out! We added tens of thousands of words since the last release.) We continued the work on our data validation story from the previous release: Deepchecks is the newest data validator we support, and we updated our Evidently and whylogs integrations to include all the latest and greatest from those tools.
Beyond this, as usual we included a number of smaller bugfixes and documentation changes to cumulatively improve experience of using ZenML as a user. For a detailed look at what's changed, give our full release notes a glance.
We've been hard at work on our new stack component and integration with Label Studio, the first of our data annotation tools. 🥳
Annotators are a stack component that enables the use of data annotation as part of your ZenML stack and pipelines. You can use the associated CLI command to launch annotation, configure your datasets and get stats on how many labeled tasks you have ready for use.
Data annotation/labeling is a core part of MLOps that is frequently left out of the conversation. With this release, ZenML now supports annotation as an integrated and first-class citizen as part of the MLOps lifecycle.
Got feedback or just want to let us know how you feel? Connect with us or join us for a Community Meetup 👋 which happens every Wednesday!
We continued the work started in the previous release, adding a new integration with Deepchecks. Deepchecks is a feature-rich data validation open-source library to painlessly do data validation. Deepchecks can do a variety of data validation tasks, from data integrity checks that work with a single dataset to data+model evaluation to data drift analyses. All this can be done with minimal configuration input from the user, or customized with specialized conditions that the validation checks should perform. Check out our example if you want to see it in action!
We also updated our integrations with Great Expectations, Evidently and whylogs to support their latest releases. These tools have added lots of great new features in recent weeks and we now support most of them, all under the new data validator stack component and abstraction.
We made a significant overhaul of our documentation since the last release:
We fixed a bug that was preventing users who upgraded to 0.10.0 from pulling new examples. This now works without any problem.
The latest release includes several smaller features and updates to existing functionality:
@enable_mlflow
decorator) which will allow the creation of nested runs for each step of a ZenML pipeline in MLflow.requirements.txt
file if the file ended with a newline.The 0.11.0 release remodels the Evidently and whylogs integrations as Data Validator stack components, in an effort to converge all data profiling and validation libraries around the same abstraction. As a consequence, you now need to configure and add a Data Validator stack component to your stack if you wish to use Evidently or whylogs in your pipelines:
for Evidently:
zenml data-validator register evidently -f evidently
zenml stack update -dv evidently
for whylogs:
zenml data-validator register whylogs -f whylogs
zenml stack update -dv whylogs
In this release, we have also upgraded the Evidently and whylogs libraries to their latest and greatest versions (whylogs 1.0.6 and evidently 0.1.52). These versions introduce non-backwards compatible changes that are also reflected in the ZenML integrations:
Evidently profiles are now materialized using their original evidently.model_profile.Profile
data type and the builtin EvidentlyProfileStep
step now also returns a Profile
instance instead of the previous dictionary representation. This may impact your existing pipelines as you may have to update your steps to take in Profile
artifact instances instead of dictionaries.
the whylogs whylogs.DatasetProfile
data type was replaced by whylogs.core.DatasetProfileView
in the builtin whylogs materializer and steps. This may impact your existing pipelines as you may have to update your steps to return and take in whylogs.core.DatasetProfileView
artifact instances instead of whylogs.DatasetProfile
objects.
the whylogs library has gone through a major transformation that completely removed the session concept. As a result, the enable_whylogs
step decorator was replaced by an enable_whylabs
step decorator. You only need to use the step decorator if you wish to log your profiles to the Whylabs platform.
Please refer to the examples provided for Evidently and whylogs to learn more about how to use the new integration versions:
We received several new community contributions during this release cycle. Here's everybody who contributed towards this release:
Join our Slack to let us know if you have an
idea for a feature or something you'd like to contribute to the framework.
We have a new home for our roadmap where you can vote on your favorite upcoming
feature or propose new ideas for what the core team should work on. You can vote
without needing to log in, so please do let us know what you want us to build!
Published by stefannica over 2 years ago
The 0.10.0 release continues our streak of extending ZenML with support for new orchestrators, this time by adding the Kubernetes Native Orchestrator. This orchestrator is a lightweight alternative to other distributed orchestrators like Airflow or Kubeflow that gives our users the ability to run pipelines in any Kubernetes cluster without having to install and manage additional tools or components.
This release features another integration that we are really excited about: the popular data profiling and validation library Great Expectations is our first Data Validator, a new category of stack components that we are in the process of standardizing, that will make data quality a central feature of ZenML. The ZenML Great Expectations integration eliminates the complexity associated with configuring the store backends for Great Expectations by reusing our Artifact Store concept for that purpose and gives ZenML users immediate access to Great Expectations in both local and cloud settings.
Last but not least, the release also includes a new secrets manager implementation, courtesy of our contributor @karimhabush, that integrates ZenML with the Hashicorp Vault Server as well as a few other bug fixes and improvements.
force
flag from secrets managers' implementation by @strickvl in https://github.com/zenml-io/zenml/pull/708
Full Changelog: https://github.com/zenml-io/zenml/compare/0.9.0...0.10.0
Published by schustmi over 2 years ago
It's been a couple of weeks, so it's time for a new release! 0.9.0 brings two whole new orchestrators, one of which was contributed by a community member just one day after we unveiled new documentation for orchestrator extensibility! The release also includes a new secrets manager, a Slack integration and a bunch of other smaller changes across the codebase. (Our new orchestrators are exciting enough that they'll get their own blog posts to showcase their strengths in due course.)
Beyond this, as usual we included a number of smaller bugfixes and documentation changes to cumulatively improve experience of using ZenML as a user.
SecretExistsError
in register_secret()
method by @hectorLop in https://github.com/zenml-io/zenml/pull/648
ml-pipelines-sdk
v1.8.0 by @strickvl in https://github.com/zenml-io/zenml/pull/651
VertexOrchestrator
implementation by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/640
pre-commit
script by @strickvl in https://github.com/zenml-io/zenml/pull/481
utils
more consistent by @strickvl in https://github.com/zenml-io/zenml/pull/658
develop
by @strickvl in https://github.com/zenml-io/zenml/pull/669
config
module by @strickvl in https://github.com/zenml-io/zenml/pull/668
codespell
with pyspelling
by @strickvl in https://github.com/zenml-io/zenml/pull/663
io
and model_deployers
docstring checks by @strickvl in https://github.com/zenml-io/zenml/pull/675
zenml stack down
to use --force flag by @schustmi in https://github.com/zenml-io/zenml/pull/673
pipelines
docstring checks by @strickvl in https://github.com/zenml-io/zenml/pull/676
cli
module by @strickvl in https://github.com/zenml-io/zenml/pull/680
entrypoints
and experiment_trackers
modules by @strickvl in https://github.com/zenml-io/zenml/pull/672
AssertionError
by @ketangangal in https://github.com/zenml-io/zenml/pull/628
SlackAlerter.ask()
by @fa9r in https://github.com/zenml-io/zenml/pull/662
materializers
and metadata_stores
by @strickvl in https://github.com/zenml-io/zenml/pull/694
integrations
module(s) by @strickvl in https://github.com/zenml-io/zenml/pull/692
Full Changelog: https://github.com/zenml-io/zenml/compare/0.8.1...0.9.0
Published by htahir1 over 2 years ago
ZenML 0.8.1 is here and it comes with support for Python 3.9 🎉. It also includes major updates to our documentation, fixes some broken links in our examples and improves the zenml go
command which helps you get started with ZenML.
We did this fast turn-around to fix some bugs in 0.8.0 and get a quick release out for Python 3.9 which was a highly requested feature! #502
zenml go
by @fa9r in https://github.com/zenml-io/zenml/pull/625
served_model
format to the keras materializer by @stefannica in https://github.com/zenml-io/zenml/pull/629
CONTRIBUTING.md
by @strickvl in https://github.com/zenml-io/zenml/pull/615
zenml go
to open quickstart/notebooks. by @fa9r in https://github.com/zenml-io/zenml/pull/631
Full Changelog: https://github.com/zenml-io/zenml/compare/0.8.0...0.8.1
Published by htahir1 over 2 years ago
stack delete
confirmation prompt by @strickvl in https://github.com/zenml-io/zenml/pull/548
zenml stack export
and zenml stack import
commands by @fa9r in https://github.com/zenml-io/zenml/pull/560
rich
CLI displays full text and wraps table text by @strickvl in https://github.com/zenml-io/zenml/pull/577
-f
and --force
with -y
and --yes
by @strickvl inhttps://github.com/zenml-io/zenml/pull/566Published by schustmi over 2 years ago
PR #530 adds a new stack component to ZenMLs ever-growing list: experiment_trackers
allows users to configure your experiment tracking tools with ZenML. Examples of experiment tracking tools are Weights&Biases, mlflow, Neptune, amongst others.
Existing users might be confused, as ZenML has had MLflow and wandb support for a while now without such a component. However, this component allows uses more control over the configuration of MLflow and wandb with the new MLFlowExperimentTracker
and
WandbExperimentTracker
components. This allows these tools to work in more scenarios than the currently limiting local use-cases.
XGBoost and LightGBM are one of the most widely used boosting algorithm libraries out there. This release adds materializers for native objects for each library.
Check out both examples here and PR's #544 and #538 for more details.
A big complaint of the S3 Artifact Store integration was that it was hard to parameterize it in a way that it supports non-AWS S3 storage like minio and ceph. The latest release
made this super simple! When you want to register an S3ArtifactStore from the CLI, you can now pass in client_kwargs
, config_kwargs
or s3_additional_kwargs
as a JSON string. For example:
zenml artifact-store register my_s3_store --type=s3 --path=s3://my_bucket \
--client_kwargs='{"endpoint_url": "http://my-s3-endpoint"}'
See PR #532 for more details.
We added functionality to allow users to update stacks that already exist. This shows the basic workflow:
zenml orchestrator register local_orchestrator2 -t local
zenml stack update default -o local_orchestrator2
zenml stack describe default
zenml container-registry register local_registry --type=default --uri=localhost:5000
zenml container-registry update local --uri='somethingelse.com'
zenml container-registry rename local local2
zenml container-registry describe local2
zenml stack rename default new_default
zenml stack update new_default -c local2
zenml stack describe new_default
zenml stack remove-component -c
More details are in the CLI docs.
Users can add new stack components to a pre-existing stack, or they can modify
already-present stack components. They can also rename their stack and individual stack components.
The Seldon Core Model Deployer stack component was updated in this release to allow the configuration of ZenML secrets with credentials that authenticate Seldon to access the Artifact Store. The Seldon Core integration provides 3 different secret schemas for the 3 flavors of Artifact Store: AWS, GCP, and Azure, but custom secrets can be used as well. For more information on how to use this feature please refer to our Seldon Core deployment example.
Lastly, we had numerous other changes such as ensuring the PyTorch materializer works across all artifact stores
and the Kubeflow Metadata Store can be easily queried locally.
mypy
errors by @strickvl in https://github.com/zenml-io/zenml/pull/524
init
docstring by @strickvl in https://github.com/zenml-io/zenml/pull/527
Full Changelog: https://github.com/zenml-io/zenml/compare/0.7.2...0.7.3
Blog Post: https://blog.zenml.io/zero-seven-two-three-release/