zenml

ZenML πŸ™: Build portable, production-ready MLOps pipelines. https://zenml.io.

APACHE-2.0 License

Downloads
44.5K
Stars
3.6K
View Code on GitHub Visit Website View on X
Ecosystems: PyTorch, Tensorflow

Bot releases are hidden (Show)

zenml - 0.7.2

Published by htahir1 over 2 years ago

0.7.2 is a minor release which quickly patches some bugs found in the last release to do with Seldon and Mlflow deployment.

This release also features initial versions of two amazing new community-led integrations: HuggingFace and Weights&Biases!

richard_socher_shoutout

  • HuggingFace models are now supported to be passed through ZenML pipelines [see full example] -> Huge shoutout to @Ankur3107 for PR #467.
  • You can now track your pipeline runs with Weights&Biases with the new enable_wandb decorator [see full example] -> Huge shoutout to @soumik12345 for PR #486.

Continuous model deployment with MLflow has been improved with ZenML 0.7.2. A new MLflow Model Deployer Stack component is now available and needs to be part of your stack to be able to deploy models:

zenml integration install mlflow
zenml model-deployer register mlflow --type=mlflow
zenml stack register local_with_mlflow -m default -a default -o default -d mlflow
zenml stack set local_with_mlflow

The MLflow Model Deployer is yet another addition to the list of Model Deployers available in ZenML. You can read more on deploying models to production with MLflow in our [Continuous Training and Deployment documentation section](https://docs.zenml.io/features/continuous training-and-deployment) and our MLflow deployment example.

What's Changed

New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.7.1...0.7.2

zenml - 0.7.1

Published by htahir1 over 2 years ago

0.7.1

Feast-Seldon

The release introduces the Seldon Core ZenML integration, featuring the Seldon Core Model Deployer and a Seldon Core standard model deployer step. The Model Deployer is a new type of stack component that enables you to develop continuous model deployment pipelines that train models and continuously deploy them to an external model serving tool, service or platform. You can read more on deploying models to production with Seldon Core in our Continuous Training and Deployment documentation section and our Seldon Core deployment example.

We also see two new integrations with Feast as ZenML's first feature store integration. Feature stores allow data teams to serve data via an offline store and an online low-latency store where data is kept in sync between the two. It also offers a centralized registry where features (and feature schemas) are stored for use within a team or wider organization. ZenML now supports connecting to a Redis-backed Feast feature store as a stack component integration. Check out the full example to see it in action!

0.7.1 also brings an addition to ZenML training library integrations with NeuralProphet. Check out the new example for more details, and the docs for more further detail on all new features!

What's Changed

zenml - 0.7.0

Published by stefannica over 2 years ago

With ZenML 0.7.0, a lot has been revamped under the hood about how things are stored. Importantly what this means is that ZenML now has system-wide profiles that let you register stacks to share across several of your projects! If you still want to manage your stacks for each project folder individually, profiles still let you do that as well.

Most projects of any complexity will require passwords or tokens to access data and infrastructure, and for this purpose ZenML 0.7.0 introduces the Secrets Manager stack component to seamlessly pass around these values to your steps. Our AWS integration also allows you to use AWS Secrets Manager as a backend to handle all your secret persistence needs.

Finally, in addition to the new AzureML and Sagemaker Step Operators that version 0.6.3 brought, this release also adds the ability to run individual steps on GCP's Vertex AI.

Beyond this, some smaller bugfixes and documentation changes combine to make ZenML 0.7.0 a more pleasant user experience.

What's Changed

New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.6.3...0.7.0rc

zenml - 0.6.3

Published by htahir1 over 2 years ago

0.6.3

ZenInTheClouds_3

With ZenML 0.6.3, you can now run your ZenML steps on Sagemaker and AzureML! It's normal to have certain steps that require specific hardware on which to run model training, for example, and this latest release gives you the power to switch out hardware for individual steps to support this.

We added a new Tensorboard visualisation that you can make use of when using our Kubeflow Pipelines integration. We handle the background processes needed to spin up this interactive web interface that you can use to visualise your model's performance over time.

Behind the scenes we gave our integration testing suite a massive upgrade, fixed a number of smaller bugs and made documentation updates. For a detailed look at what's changed, see below:

What's Changed

New Contributors

zenml - 0.6.2

Published by stefannica over 2 years ago

ZenML0-6-2

ZenML 0.6.2 brings you the ability to serve models using MLflow deployments as well as an updated CLI interface! For a real continuous deployment cycle, we know that ZenML pipelines should be able to handle everything β€” from pre-processing to training to serving to monitoring and then potentially re-training and re-serving. The interfaces we created in this release are the foundation on which all of this will build.

We also improved how you interact with ZenML through the CLI. Everything looks so much smarter and readable now with the popular rich library integrated into our dependencies.

Smaller changes that you'll notice include updates to our cloud integrations and bug fixes for Windows users. For a detailed look at what's changed, see below.

What's Changed

zenml - 0.6.1

Published by htahir1 over 2 years ago

0.6.1

ZenML 0.6.1 is out and it's all about the cloud ☁️! We have improved AWS integration and a brand-new Azure integration! Run your pipelines on AWS and Azure now and let us know how it went on our Slack. To learn more, check out the new documentation page we just included to guide you in deploying your pipelines to AWS, GCP and/or Azure.

Smaller changes that you'll notice include much-awaited updates and fixes, including the first iterations of scheduling pipelines and tracking more reproducibility-relevant data in the metadata store.

For a detailed look at what's changed, see below.

What's Changed

Full Changelog: https://github.com/zenml-io/zenml/compare/0.6.0...0.6.1

zenml - 0.6.0

Published by htahir1 over 2 years ago

ZenML 0.6.0 is out now. We've made some big changes under the hood, but our biggest public-facing addition is our new integration to support all your data logging needs: whylogs. Our core architecture was thoroughly reworked and is now in a much better place to support our ongoing development needs.

Smaller changes that you'll notice include extensive documentation additions, updates and fixes. For a detailed look at what's changed, see below.

πŸ“Š Whylogs logging

Whylogs is an open source library that analyzes your data and creates statistical summaries called whylogs profiles. Whylogs profiles can be visualized locally or uploaded to the WhyLabs platform where more comprehensive analysis can be carried out.

ZenML integrates seamlessly with Whylogs and WhyLabs. This example shows how easy it is to enhance steps in an existing ML pipeline with Whylogs profiling features. Changes to the user code are minimal while ZenML takes care of all aspects related to Whylogs session initialization, profile serialization, versioning and persistence and even uploading generated profiles to Whylabs.

Example of the visualizations you can make from Whylogs profiles

With our WhylogsVisualizer, as described in the associated example notes, you can visualize Whylogs profiles generated as part of a pipeline.

β›© New Core Architecture

We implemented some fundamental changes to the core architecture to solve some of the issues we previously had and provide a more extensible design to support quicker implementations of different stack components and integrations. The main change was to refactor the Repository, Stack and StackComponent architectures. These changes had a pretty wide impact so involved changes in many files throughout the codebase, especially in the CLI which makes calls to all these pieces.

We've already seen how it helps us move faster in building integrations and we hope it helps making contributions as pain-free as possible!

πŸ—’ Documentation and Example Updates

As the codebase and functionality of ZenML grows, we always want to make sure our documentation is clear, up-to-date and easy to use. We made a number of changes in this release that will improve your experience in this regard:

  • added a number of new explainers on key ZenML concepts and how to use them in your code, notably on how to create a custom materializer and how to fetch historic pipeline runs using the StepContext
  • fixed a number of typos and broken links
  • added versioning to our API documentation so you can choose to view the reference appropriate to the version that you're using. We now use mkdocs for this so you'll notice a slight visual refresh as well.
  • added new examples highlighting specific use cases and integrations:
    • how to create a custom materializer (example)
    • how to fetch historical pipeline runs (example)
    • how to use standard interfaces for common ML patterns (example)
    • whylogs logging (example)

βž• Other updates, additions and fixes

As with most releases, we made a number of small but significant fixes and additions. The most import of these were that you can now access the metadata store via the step context. This enables a number of new possible workflows and pipeline patterns and we're really excited to have this in the release.

We added in a markdown parser for the zenml example info … command, so now when you want to use our CLI to learn more about specific examples you will see beautifully parsed text and not markdown markup.

We improved a few of our error messages, too, like for when the return type of a step function doesn’t match the expected type, or if step is called twice. We hope this makes ZenML just that little bit easier to use.

New Contributors

[Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.7...0.6.0]

zenml - 0.5.7

Published by AlexejPenner almost 3 years ago

0.5.7

ZenML 0.5.7 is here πŸ’― and it brings not one, but πŸ”₯TWOπŸ”₯ brand new integrations πŸš€! ZenML now support MLFlow for tracking pipelines as experiments and Evidently for detecting drift in your ML pipelines in production!

New Features

  • Introducing the MLFLow Tracking Integration, a first step towards our complete MLFlow Integration as described in the #115 poll. Full example found here.

Mlflow

evidently

Bugfixes

What's Changed

New Contributors

zenml - 0.5.6

Published by htahir1 almost 3 years ago 

    )                    *      (     
 ( /(                  (  `     )\ )  
 )\())    (            )\))(   (()/(  
((_)\    ))\    (     ((_)()\   /(_)) 
 _((_)  /((_)   )\ )  (_()((_) (_))   
|_  /  (_))    _(_/(  |  \/  | | |    
 / /   / -_)  | ' \)) | |\/| | | |__  
/___|  \___|  |_||_|  |_|  |_| |____|

This release fixes some known bugs from previous releases and especially 0.5.5. Therefore, upgrading to 0.5.6 is a breaking change. You must do the following in order to proceed with this version:

cd zenml_enabled_repo
rm -rf .zen/

And then start again with ZenML init:

pip install --upgrade zenml
zenml init

New Features

  • Added zenml example run [EXAMPLE_RUN_NAME] feature: The ability to run an example with one command. In order to run this, do zenml example pull first and see all examples available by running zenml example list.
  • Added ability to specify a .dockerignore file before running pipelines on Kubeflow.
  • Kubeflow Orchestrator is now leaner and faster.
  • Added the describe command group to the CLI for groups stack, orchestrator, artifact-store, and metadata-store. E.g. zenml stack describe

Bug fixes and minor improvements

  • Adding StepContext to a branch now invalidates caching by default. Disable explicitly with enable_cache=True.
  • Docs updated to reflect minor changes in CLI commands.
  • CLI list commands now mentions active component. Try zenml stack list to check this out.
  • zenml version now has cooler art.

What's Changed

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.5...0.5.6

zenml - 0.5.5

Published by htahir1 almost 3 years ago

ZenML 0.5.5 is jam-packed with new features to take your ML pipelines to the next level. Our three biggest new features: Kubeflow Pipelines, CLI support for our integrations and Standard Interfaces. That’s right, Standard Interfaces are back!

What's Changed

New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.4...0.5.5

zenml - 0.5.4

Published by htahir1 almost 3 years ago

0.5.4 adds a lineage tracking integration to visualize lineage of pipeline runs! It also includes numerous bug fixes and optimizations.

What's Changed

New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.3...0.5.4

zenml - 0.5.3

Published by schustmi almost 3 years ago

Version 0.5.3 adds statistics visualizations, greatly improved speed for CLI commands as well as lots of small improvements to the pipeline and step interface.

What's Changed

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.2...0.5.3

zenml - 0.5.2

Published by htahir1 almost 3 years ago

0.5.2

0.5.2 brings an improved post-execution workflow and lots of minor changes and upgrades for the developer experience when
creating pipelines. It also improves the Airflow orchestrator logic to accommodate for more real world scenarios. Check out the
low level API guide for more details!

What's Changed

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.1...0.5.2

zenml - 0.5.1

Published by htahir1 almost 3 years ago

0.5.1

0.5.1 builds on top of Slack of the 0.5.0 release with quick bug updates.

Overview

  • Pipeline can now be run via a YAML file. #132
  • CLI now let's you pull directly from GitHub examples folder. πŸ”₯ Amazing @alex-zenml with #141!
  • ZenML now has full mypy compliance. πŸŽ‰ Thanks @schustmi for #140!
  • Numerous bugs and performance improvements. #136, @bcdurak great job with #142
  • Airflow orchestrator now bootstraps airflow locally and spins it up before running pipelines.
  • Added new docs with a low level API guide. #143

Our roadmap goes into further detail on the timeline. Vote on the next features now.

We encourage every user (old or new) to start afresh with this release. Please go over our latest docs and examples to get a hang of the new system.

Auto-generated docs:

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.0...0.5.1

zenml - 0.5.0

Published by schustmi about 3 years ago

0.5.0

This long-awaited ZenML release marks a seminal moment in the project's history. We present to you a complete
revamp of the internals of ZenML, with a fresh new design and API. While these changes are significant, and have been months
in the making, the original vision of ZenML has not wavered. We hope that the ZenML community finds the new
design choices easier to grasp and use, and we welcome feedback on the issues board.

Warning

0.5.0 is a complete API change from the previous versions of ZenML, and is a breaking upgrade. Fundamental
concepts have been changed, and therefore backwards compatibility is not maintained. Please use only this version
with fresh projects.

With such significant changes, we expect this release to also be breaking. Please report any bugs in the issue board, and
they should be addressed in upcoming releases.

Overview

  • Introducing a new functional API for creating pipelines and steps. This is now the default mechanism for building ZenML pipelines. read more
  • Steps now use Materializers to handle artifact serialization/deserialization between steps. This is a powerful change, and will be expanded upon in the future. read more
  • Introducing the new Stack paradigm: Easily transition from one MLOps stack to the next with a few CLI commands read more
  • Introducing a new Artifact, Typing, and Annotation system, with pydantic (and dataclasses) support read more
  • Deprecating the pipelines_dir: Now individual pipelines will be stored in their metadata stores, making the metadata store a single source of truth. read more
  • Deprecating the YAML config file: ZenML no longer natively compiles to an intermediate YAML-based representation. Instead, it compiles and deploys directly into the selected orchestrator's
    representation. While we do plan to support running pipelines directly through YAML in the future, it will no longer be
    the default route through which pipelines are run. read more about orchestrators here

Technical Improvements

  • A completely new system design, please refer to the docs.
  • Better type hints and docstrings.
  • Auto-completion support.
  • Numerous performance improvements and bug fixes, including a smaller dependency footprint.

What to expect in the next weeks and the new ZenML

Currently, this release is bare bones. We are missing some basic features which used to be part of ZenML 0.3.8 (the previous release):

  • Standard interfaces for TrainingPipeline.
  • Individual step interfaces like PreprocesserStep, TrainerStep, DeployerStep etc. need to be rewritten from within the new paradigm. They should
    be included in the non-RC version of this release.
  • A proper production setup with an orchestrator like Airflow.
  • A post-execution workflow to analyze and inspect pipeline runs.
  • The concept of Backends will evolve into a simple mechanism of transitioning individual steps into different runners.
  • Support for KubernetesOrchestrator, KubeflowOrchestrator, GCPOrchestrator and AWSOrchestrator are also planned.
  • Dependency management including Docker support is planned.

Our roadmap goes into further detail on the timeline.

We encourage every user (old or new) to start afresh with this release. Please go over our latest docs
and examples to get a hang of the new system.

Onwards and upwards to 1.0.0!

zenml - 0.3.8

Published by hamzamaiot over 3 years ago

zenml - 0.3.7.1rc5

Published by bcdurak over 3 years ago

preparing for 0.3.7.1

zenml - 0.3.7

Published by hamzamaiot over 3 years ago

0.3.7

0.3.7 is a much-needed, long-awaited, big refactor of the Datasources paradigm of ZenML. There are also bug fixes, improvements, and more!

For those upgrading from an older version of ZenML, we ask to please delete their old pipelines dir and .zenml folders and start afresh with a zenml init.

If only working locally, this is as simple as:

cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/

And then another ZenML init:

pip install --upgrade zenml
cd zenml_enabled_repo
zenml init

New Features

  • The inner-workings of the BaseDatasource have been modified along with the concrete implementations. Now, there is no relation between a DataStep and a Datasource: A Datasource holds all the logic to version and track itself via the new commit paradigm.

  • Introduced a new interface for datasources, the process method which is responsible for ingesting data and writing to TFRecords to be consumed by later steps.

  • Datasource versions (snapshots) can be accessed directly via the commits paradigm: Every commit is a new version of data.

  • Added JSONDatasource and TFRecordsDatasource.

Bug Fixes + Refactor

A big thanks to our new contributer @aak7912 for the help in this release with issue #71 and PR #75.

  • Added an example for regression.
  • compare_training_runs() now takes an optional datasource parameter to filter by datasource.
  • Trainer interface refined to focus on run_fn rather than other helper functions.
  • New docs released with a streamlined vision and coherent storyline: https://docs.zenml.io
  • Got rid of unnecessary Torch dependency with base ZenML version.
zenml - 0.3.6

Published by bcdurak over 3 years ago

0.3.6

0.3.6 is a more inwards-facing release as part of a bigger effort to create a more flexible ZenML. As a first step, ZenML now supports arbitrary splits for all components natively, freeing us from the train/eval split paradigm. Here is an overview of changes:

New Features

  • The inner-workings of the BaseTrainerStep, BaseEvaluatorStep and the BasePreprocesserStep have been modified along with their respective components to work with the new split_mapping. Now, users can define arbitrary splits (not just train/eval). E.g. Doing a train/eval/test split is possible.

  • Within the instance of a TrainerStep, the user has access to input_patterns and output_patterns which provide the required uris with respect to their splits for the input and output(test_results) examples.

  • The built-in trainers are modified to work with the new changes.

Bug Fixes + Refactor

A big thanks to our new super supporter @zyfzjsc988 for most of the feedback that led to bug fixes and enhancements for this release:

  • #63: Now one can specify which ports ZenML opens its add-on applications.
  • #64 Now there is a way to list integrations with the following code:
from zenml.utils.requirements_utils import list_integrations.
list_integrations()
  • Fixed #61: view_anomalies() breaking in the quickstart.
  • Analytics is now opt-in by default, to get rid of the unnecessary prompt at zenml init. Users can still freely opt-out by using the CLI:
zenml config analytics opt-out

Again, the telemetry data is fully anonymized and just used to improve the product. Read more here

zenml - 0.3.5

Published by hamzamaiot over 3 years ago

This release finally brings model agnostic automatic evaluation to ZenML! Now you can easily use TFMA with any model type to produce evaluation visualizations. This means you can now use TFMA with PyTorch or Scikit - a big win for automated sliced evaluation! It also introduces a new language for differentiation between features, raw features, labels and predictions, in addition to solving a few big bugs in the examples directory! Read more below.

As has been the case in the last few releases, this release is yet another breaking upgrade.

For those upgrading from an older version of ZenML, we ask to please delete their old pipelines dir and .zenml folders and start afresh with a zenml init.

If only working locally, this is as simple as:

cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/

And then another ZenML init:

pip install --upgrade zenml
cd zenml_enabled_repo
zenml init

New Features

  • Added a new interface into the trainer step called test_fn which is utilized to produce model predictions and save them as test results

  • Implemented a new evaluator step called AgnosticEvaluator which is designed to work regardless of the model type as long as you run the test_fn in your trainer step

  • The first two changes allow torch trainer steps to be followed by an agnostic evaluator step, see the example here.

  • Proposed a new naming scheme, which is now integrated into the built-in steps, in order to make it easier to handle feature/label names.

  • Modified the TorchFeedForwardTrainer to showcase how to use TensorBoard in conjunction with PyTorch

Bug Fixes + Refactor

  • Refactored how ZenML treats relative imports for custom steps. Now, rather than doing absolute imports like:
from examples.scikit.step.trainer import MyScikitTrainer

One can also do the following:

from step.trainer import MyScikitTrainer

ZenML automatically figures out the absolute path of the module based on the root of the directory.

Big shout out to @SaraKingGH in issue #55 for raising the above issues!

Package Rankings
Top 2.72% on Pypi.org
Badges
Extracted from project README
PyPi PyPi PyPi Contributors License Star this project Open In Colab Introductory Youtube Video Star this project
Related Projects
mlreef

The collaboration workspace for Machine Learning

20 Jul 2020 1,442
ml-workspace

πŸ›  All-in-one web-based IDE specialized for machine learning and data science.

27 May 2019 3,406