ZenML π: Build portable, production-ready MLOps pipelines. https://zenml.io.
APACHE-2.0 License
Bot releases are visible (Hide)
Published by htahir1 over 2 years ago
0.7.2 is a minor release which quickly patches some bugs found in the last release to do with Seldon and Mlflow deployment.
This release also features initial versions of two amazing new community-led integrations: HuggingFace and Weights&Biases!
enable_wandb
decorator [see full example] -> Huge shoutout to @soumik12345 for PR #486.Continuous model deployment with MLflow has been improved with ZenML 0.7.2. A new MLflow Model Deployer Stack component is now available and needs to be part of your stack to be able to deploy models:
zenml integration install mlflow
zenml model-deployer register mlflow --type=mlflow
zenml stack register local_with_mlflow -m default -a default -o default -d mlflow
zenml stack set local_with_mlflow
The MLflow Model Deployer is yet another addition to the list of Model Deployers available in ZenML. You can read more on deploying models to production with MLflow in our [Continuous Training and Deployment documentation section](https://docs.zenml.io/features/continuous training-and-deployment) and our MLflow deployment example.
Full Changelog: https://github.com/zenml-io/zenml/compare/0.7.1...0.7.2
Published by htahir1 over 2 years ago
The release introduces the Seldon Core ZenML integration, featuring the Seldon Core Model Deployer and a Seldon Core standard model deployer step. The Model Deployer is a new type of stack component that enables you to develop continuous model deployment pipelines that train models and continuously deploy them to an external model serving tool, service or platform. You can read more on deploying models to production with Seldon Core in our Continuous Training and Deployment documentation section and our Seldon Core deployment example.
We also see two new integrations with Feast as ZenML's first feature store integration. Feature stores allow data teams to serve data via an offline store and an online low-latency store where data is kept in sync between the two. It also offers a centralized registry where features (and feature schemas) are stored for use within a team or wider organization. ZenML now supports connecting to a Redis-backed Feast feature store as a stack component integration. Check out the full example to see it in action!
0.7.1 also brings an addition to ZenML training library integrations with NeuralProphet. Check out the new example for more details, and the docs for more further detail on all new features!
pre-commit
by @strickvl in https://github.com/zenml-io/zenml/pull/490
.gitignore
by @strickvl in https://github.com/zenml-io/zenml/pull/488
Published by stefannica over 2 years ago
With ZenML 0.7.0, a lot has been revamped under the hood about how things are stored. Importantly what this means is that ZenML now has system-wide profiles that let you register stacks to share across several of your projects! If you still want to manage your stacks for each project folder individually, profiles still let you do that as well.
Most projects of any complexity will require passwords or tokens to access data and infrastructure, and for this purpose ZenML 0.7.0 introduces the Secrets Manager stack component to seamlessly pass around these values to your steps. Our AWS integration also allows you to use AWS Secrets Manager as a backend to handle all your secret persistence needs.
Finally, in addition to the new AzureML and Sagemaker Step Operators that version 0.6.3 brought, this release also adds the ability to run individual steps on GCP's Vertex AI.
Beyond this, some smaller bugfixes and documentation changes combine to make ZenML 0.7.0 a more pleasant user experience.
zenml stack describe
bug by @strickvl in https://github.com/zenml-io/zenml/pull/476
examples
folder to linting script by @strickvl in https://github.com/zenml-io/zenml/pull/482
Full Changelog: https://github.com/zenml-io/zenml/compare/0.6.3...0.7.0rc
Published by htahir1 over 2 years ago
With ZenML 0.6.3, you can now run your ZenML steps on Sagemaker and AzureML! It's normal to have certain steps that require specific hardware on which to run model training, for example, and this latest release gives you the power to switch out hardware for individual steps to support this.
We added a new Tensorboard visualisation that you can make use of when using our Kubeflow Pipelines integration. We handle the background processes needed to spin up this interactive web interface that you can use to visualise your model's performance over time.
Behind the scenes we gave our integration testing suite a massive upgrade, fixed a number of smaller bugs and made documentation updates. For a detailed look at what's changed, see below:
rich
from 11.0 to 12.0 by @strickvl in https://github.com/zenml-io/zenml/pull/458
explain
subcommand by @bcdurak in https://github.com/zenml-io/zenml/pull/460
Published by stefannica over 2 years ago
ZenML 0.6.2 brings you the ability to serve models using MLflow deployments as well as an updated CLI interface! For a real continuous deployment cycle, we know that ZenML pipelines should be able to handle everything β from pre-processing to training to serving to monitoring and then potentially re-training and re-serving. The interfaces we created in this release are the foundation on which all of this will build.
We also improved how you interact with ZenML through the CLI. Everything looks so much smarter and readable now with the popular rich
library integrated into our dependencies.
Smaller changes that you'll notice include updates to our cloud integrations and bug fixes for Windows users. For a detailed look at what's changed, see below.
matplotlib
crash when running IPython in terminal by @strickvl in https://github.com/zenml-io/zenml/pull/416
Published by htahir1 over 2 years ago
ZenML 0.6.1 is out and it's all about the cloud βοΈ! We have improved AWS integration and a brand-new Azure integration! Run your pipelines on AWS and Azure now and let us know how it went on our Slack. To learn more, check out the new documentation page we just included to guide you in deploying your pipelines to AWS, GCP and/or Azure.
Smaller changes that you'll notice include much-awaited updates and fixes, including the first iterations of scheduling pipelines and tracking more reproducibility-relevant data in the metadata store.
For a detailed look at what's changed, see below.
Full Changelog: https://github.com/zenml-io/zenml/compare/0.6.0...0.6.1
Published by htahir1 over 2 years ago
ZenML 0.6.0 is out now. We've made some big changes under the hood, but our biggest public-facing addition is our new integration to support all your data logging needs: whylogs
. Our core architecture was thoroughly reworked and is now in a much better place to support our ongoing development needs.
Smaller changes that you'll notice include extensive documentation additions, updates and fixes. For a detailed look at what's changed, see below.
Whylogs is an open source library that analyzes your data and creates statistical summaries called whylogs profiles. Whylogs profiles can be visualized locally or uploaded to the WhyLabs platform where more comprehensive analysis can be carried out.
ZenML integrates seamlessly with Whylogs and WhyLabs. This example shows how easy it is to enhance steps in an existing ML pipeline with Whylogs profiling features. Changes to the user code are minimal while ZenML takes care of all aspects related to Whylogs session initialization, profile serialization, versioning and persistence and even uploading generated profiles to Whylabs.
With our WhylogsVisualizer
, as described in the associated example notes, you can visualize Whylogs profiles generated as part of a pipeline.
We implemented some fundamental changes to the core architecture to solve some of the issues we previously had and provide a more extensible design to support quicker implementations of different stack components and integrations. The main change was to refactor the Repository
, Stack
and StackComponent
architectures. These changes had a pretty wide impact so involved changes in many files throughout the codebase, especially in the CLI which makes calls to all these pieces.
We've already seen how it helps us move faster in building integrations and we hope it helps making contributions as pain-free as possible!
As the codebase and functionality of ZenML grows, we always want to make sure our documentation is clear, up-to-date and easy to use. We made a number of changes in this release that will improve your experience in this regard:
StepContext
mkdocs
for this so you'll notice a slight visual refresh as well.As with most releases, we made a number of small but significant fixes and additions. The most import of these were that you can now access the metadata store via the step context. This enables a number of new possible workflows and pipeline patterns and we're really excited to have this in the release.
We added in a markdown parser for the zenml example info β¦
command, so now when you want to use our CLI to learn more about specific examples you will see beautifully parsed text and not markdown markup.
We improved a few of our error messages, too, like for when the return type of a step function doesnβt match the expected type, or if step is called twice. We hope this makes ZenML just that little bit easier to use.
[Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.7...0.6.0]
Published by AlexejPenner almost 3 years ago
ZenML 0.5.7 is here π― and it brings not one, but π₯TWOπ₯ brand new integrations π! ZenML now support MLFlow for tracking pipelines as experiments and Evidently for detecting drift in your ML pipelines in production!
stack up
by @stefannica in https://github.com/zenml-io/zenml/pull/299
stack up
failure prompts by @alex-zenml in https://github.com/zenml-io/zenml/pull/290
stack up
and orchestrator up
failure prompts by @stefannica in https://github.com/zenml-io/zenml/pull/297
stack up
by @stefannica in https://github.com/zenml-io/zenml/pull/299
fileio
by @alex-zenml in https://github.com/zenml-io/zenml/pull/298
Published by htahir1 almost 3 years ago
) * (
( /( ( ` )\ )
)\()) ( )\))( (()/(
((_)\ ))\ ( ((_)()\ /(_))
_((_) /((_) )\ ) (_()((_) (_))
|_ / (_)) _(_/( | \/ | | |
/ / / -_) | ' \)) | |\/| | | |__
/___| \___| |_||_| |_| |_| |____|
This release fixes some known bugs from previous releases and especially 0.5.5. Therefore, upgrading to 0.5.6 is a breaking change. You must do the following in order to proceed with this version:
cd zenml_enabled_repo
rm -rf .zen/
And then start again with ZenML init:
pip install --upgrade zenml
zenml init
zenml example run [EXAMPLE_RUN_NAME]
feature: The ability to run an example with one command. In order to run this, do zenml example pull
first and see all examples available by running zenml example list
..dockerignore
file before running pipelines on Kubeflow.describe
command group to the CLI for groups stack
, orchestrator
, artifact-store
, and metadata-store
. E.g. zenml stack describe
StepContext
to a branch now invalidates caching by default. Disable explicitly with enable_cache=True
.list
commands now mentions active component. Try zenml stack list
to check this out.zenml version
now has cooler art.Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.5...0.5.6
Published by htahir1 almost 3 years ago
ZenML 0.5.5 is jam-packed with new features to take your ML pipelines to the next level. Our three biggest new features: Kubeflow Pipelines, CLI support for our integrations and Standard Interfaces. Thatβs right, Standard Interfaces are back!
Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.4...0.5.5
Published by htahir1 almost 3 years ago
0.5.4 adds a lineage tracking integration to visualize lineage of pipeline runs! It also includes numerous bug fixes and optimizations.
Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.3...0.5.4
Published by schustmi almost 3 years ago
Version 0.5.3 adds statistics visualizations, greatly improved speed for CLI commands as well as lots of small improvements to the pipeline and step interface.
examples
command by @alex-zenml in https://github.com/zenml-io/zenml/pull/180
Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.2...0.5.3
Published by htahir1 almost 3 years ago
0.5.2 brings an improved post-execution workflow and lots of minor changes and upgrades for the developer experience when
creating pipelines. It also improves the Airflow orchestrator logic to accommodate for more real world scenarios. Check out the
low level API guide for more details!
core
and cli
modules by @alex-zenml in https://github.com/zenml-io/zenml/pull/149
Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.1...0.5.2
Published by htahir1 almost 3 years ago
0.5.1 builds on top of Slack of the 0.5.0 release with quick bug updates.
Our roadmap goes into further detail on the timeline. Vote on the next features now.
We encourage every user (old or new) to start afresh with this release. Please go over our latest docs and examples to get a hang of the new system.
Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.0...0.5.1
Published by schustmi about 3 years ago
This long-awaited ZenML release marks a seminal moment in the project's history. We present to you a complete
revamp of the internals of ZenML, with a fresh new design and API. While these changes are significant, and have been months
in the making, the original vision of ZenML has not wavered. We hope that the ZenML community finds the new
design choices easier to grasp and use, and we welcome feedback on the issues board.
0.5.0 is a complete API change from the previous versions of ZenML, and is a breaking upgrade. Fundamental
concepts have been changed, and therefore backwards compatibility is not maintained. Please use only this version
with fresh projects.
With such significant changes, we expect this release to also be breaking. Please report any bugs in the issue board, and
they should be addressed in upcoming releases.
Stack
paradigm: Easily transition from one MLOps stack to the next with a few CLI commands read more
Artifact
, Typing
, and Annotation
system, with pydantic
(and dataclasses
) support read more
pipelines_dir
: Now individual pipelines will be stored in their metadata stores, making the metadata store a single source of truth. read more
Currently, this release is bare bones. We are missing some basic features which used to be part of ZenML 0.3.8 (the previous release):
TrainingPipeline
.PreprocesserStep
, TrainerStep
, DeployerStep
etc. need to be rewritten from within the new paradigm. They shouldBackends
will evolve into a simple mechanism of transitioning individual steps into different runners.KubernetesOrchestrator
, KubeflowOrchestrator
, GCPOrchestrator
and AWSOrchestrator
are also planned.Our roadmap goes into further detail on the timeline.
We encourage every user (old or new) to start afresh with this release. Please go over our latest docs
and examples to get a hang of the new system.
Onwards and upwards to 1.0.0!
Published by hamzamaiot over 3 years ago
Published by bcdurak over 3 years ago
preparing for 0.3.7.1
Published by hamzamaiot over 3 years ago
0.3.7 is a much-needed, long-awaited, big refactor of the Datasources paradigm of ZenML. There are also bug fixes, improvements, and more!
For those upgrading from an older version of ZenML, we ask to please delete their old pipelines
dir and .zenml
folders and start afresh with a zenml init
.
If only working locally, this is as simple as:
cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/
And then another ZenML init:
pip install --upgrade zenml
cd zenml_enabled_repo
zenml init
The inner-workings of the BaseDatasource
have been modified along with the concrete implementations. Now, there is no relation between a DataStep
and a Datasource
: A Datasource
holds all the logic to version and track itself via the new commit
paradigm.
Introduced a new interface for datasources, the process
method which is responsible for ingesting data and writing to TFRecords to be consumed by later steps.
Datasource versions (snapshots) can be accessed directly via the commits
paradigm: Every commit is a new version of data.
Added JSONDatasource
and TFRecordsDatasource
.
A big thanks to our new contributer @aak7912 for the help in this release with issue #71 and PR #75.
compare_training_runs()
now takes an optional datasource
parameter to filter by datasource.Trainer
interface refined to focus on run_fn
rather than other helper functions.Published by bcdurak over 3 years ago
0.3.6 is a more inwards-facing release as part of a bigger effort to create a more flexible ZenML. As a first step, ZenML now supports arbitrary splits for all components natively, freeing us from the train/eval
split paradigm. Here is an overview of changes:
The inner-workings of the BaseTrainerStep
, BaseEvaluatorStep
and the BasePreprocesserStep
have been modified along with their respective components to work with the new split_mapping. Now, users can define arbitrary splits (not just train/eval). E.g. Doing a train/eval/test
split is possible.
Within the instance of a TrainerStep
, the user has access to input_patterns
and output_patterns
which provide the required uris with respect to their splits for the input and output(test_results) examples.
The built-in trainers are modified to work with the new changes.
A big thanks to our new super supporter @zyfzjsc988 for most of the feedback that led to bug fixes and enhancements for this release:
from zenml.utils.requirements_utils import list_integrations.
list_integrations()
view_anomalies()
breaking in the quickstart.opt-in
by default, to get rid of the unnecessary prompt at zenml init
. Users can still freely opt-out
by using the CLI:zenml config analytics opt-out
Again, the telemetry data is fully anonymized and just used to improve the product. Read more here
Published by hamzamaiot over 3 years ago
This release finally brings model agnostic automatic evaluation to ZenML! Now you can easily use TFMA with any model type to produce evaluation visualizations. This means you can now use TFMA with PyTorch or Scikit - a big win for automated sliced evaluation! It also introduces a new language for differentiation between features, raw features, labels and predictions, in addition to solving a few big bugs in the examples
directory! Read more below.
As has been the case in the last few releases, this release is yet another breaking upgrade.
For those upgrading from an older version of ZenML, we ask to please delete their old pipelines
dir and .zenml
folders and start afresh with a zenml init
.
If only working locally, this is as simple as:
cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/
And then another ZenML init:
pip install --upgrade zenml
cd zenml_enabled_repo
zenml init
Added a new interface into the trainer step called test_fn
which is utilized to produce model predictions and save them as test results
Implemented a new evaluator step called AgnosticEvaluator
which is designed to work regardless of the model type as long as you run the test_fn
in your trainer step
The first two changes allow torch trainer steps to be followed by an agnostic evaluator step, see the example here.
Proposed a new naming scheme, which is now integrated into the built-in steps, in order to make it easier to handle feature/label names.
Modified the TorchFeedForwardTrainer
to showcase how to use TensorBoard in conjunction with PyTorch
from examples.scikit.step.trainer import MyScikitTrainer
One can also do the following:
from step.trainer import MyScikitTrainer
ZenML automatically figures out the absolute path of the module based on the root of the directory.
Big shout out to @SaraKingGH in issue #55 for raising the above issues!