prefect

Releases some bug fixes and minor enhancements. Check out the release notes for more info.

This release includes some important bug fixes and minor enhancements. Check out the release notes for details.

There are some exciting features in this release:

• Add support for deploying flows stored in Docker images — #6574
• Add support for deploying flows stored on GitHub — #6598
• Add file system block for reading directories from GitHub — #6517
• Add a context manager to disable the flow and task run loggers for testing — #6575

The list of changes is way to big to fit here! Check out the full release notes.

Exciting New Features 🎉

• Added automatic detection of static arguments to Task.map in #6513

Fixes

• Updated deployment flow run retry settings with runtime values in #6489
• Updated media queries for flow-run-filter in #6484
• Added empirical_policy to flow run update route in #6486
• Updated flow run policy retry settings to be nullable in #6488
• Disallowed extra attribute initialization on Deployment objects in #6505
• Updated deployment build to raise an informative error if two infrastructure configs are provided in #6504
• Fixed calling async subflows from sync parents in #6514

Fixes an issue where deployment flow run's would fail if they did not specify a value for parameters that have a default value on the flow function. A few minor fixes are included as well, see the release notes for more details.

See the release notes for details about what's new in this release.

Simplified deployments

The deployment experience has been refined to remove extraneous artifacts and make configuration even easier. In particular:

• prefect deployment build no longer generates a manifest.json file. Instead, all of the relevant information is written to the deployment.yaml file.
• Values in the deployment.yaml file are more atomic and explicit
• Local file system blocks are no longer saved automatically
• Infrastructure block values can now be overwritten with the new infra_overrides field

Start custom flow runs from the UI

Now, from the deployment page, in addition to triggering an immediate flow run with default parameter arguments, you can also create a custom run. A custom run enables you to configure the run's parameter arguments, start time, name, and more, all while otherwise using the same deployment configuration. The deployment itself will be unchanged and continue to generate runs on its regular schedule.

Improvements and bug fixes

• Made timeout errors messages on state changes more intuitive
• Added debug level logs for task run rehydration
• Added basic CLI functionality to inspect Blocks; more to come
• Added support for filtering on state name to prefect flow-run ls
• Refined autogenerated database migration output

New Contributors

• @hugolmn made their first contribution in https://github.com/PrefectHQ/prefect/pull/6275

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.0.3...2.0.4

2.0.3

This release contains a number of bug fixes and documentation improvements.

Introducing prefect-dbt

We've released prefect-dbt - a collection of Prefect integrations for working with dbt in your Prefect flows. This collection has been built as part of a partnership with dbt Labs to ensure that it follows best practices for working with dbt.

Improvements and bug fixes

• Azure storage blocks can use .prefectignore
• Resolved bugs and improved interface in the Orion client.
• Resolved a bug in Azure storage blocks that would cause uploads to get stuck.
• Resolved a bug where calling a flow in a separate thread would raise an exception.
• Resolved issues with loading flows from a deployment.
• Corrected some erroneous type annotations.
• Better handling of database errors during state transition validation.
• Better sanitization of labels for Kubernetes Jobs.
• Fixes --manifest-only flag of prefect deployment build command to ensure that using this flag, the manifest gets generated, but the upload to a storage location is skipped.
• Added support for multiple YAML deployment paths to the prefect deployment apply command.

Enhancements

• Allow custom job name for Kubernetes flow runsto be set in job template - #5819

Task Library

• Adds container parameter to ReadNamespacedPodLogs - #5979
• Add PowershellTask for running commands with Microsoft Powershell - #6035
• Move azure-mgmt-datafactory requirement to 'extras', instead of 'orchestration_extras' - #6018
• Fix default initialization of polling_interval in azure.datafactory.PipelineRun - #6021

Contributors

• Alvaro Duran Tovar
• Chinmay Bhat

2.0.2

This release implements a number of improvements and bug fixes in response to continued engagement by members of our community. Thanks, as always, to all who submitted ideas on how to make Prefect 2 even better.

Introducing .prefectignore files

.prefectignore files allow users to omit certain files or directories from their deployments. Similar to other .ignore files, the syntax supports pattern matching, so an entry of *.pyc will ensure all .pyc files are ignored by the deployment call when uploading to remote storage. Prefect provides a default .prefectignore file, but users can customize it to their needs.

Improvements and bug fixes

• Users can now submit bug reports and feature enhancements using our issue templates.
• Block deletion is now more performant.
• Inconsistencies in UI button copy have been removed.
• Error messaging is clearer in the deployment build CLI command.
• Resolved timeout errors that occurred when using async task functions inside synchronous flows.

The response to Prefect 2 has been overwhelming in the best way possible. Thank you to the many community members who tried it out and gave us feedback! Thanks in particular to the students at this week's Prefect Associate Certification Course (PACC) in San Jose for their thoughtful recommendations. This release is a compilation of enhancements and fixes that make for a more resilient, performant, and refined Prefect experience.

Improvements and bug fixes

• Schedules set via the API or UI are now preserved when building deployments from the CLI
• JSON types are now coerced to none, following Javascript convention and supporting standards compatibility
• The prefect deployment execute command has been removed to avoid confusion between running a flow locally from a Python script and running it by an agent using prefect deployment run
• This repository now includes templates for pull requests and issues to make bug reports and community contributions easier
• The scheduler and flow-run-notifications LoopServices have been made more resilient
• Log inserts have been made more performant through smaller log batches
• Local file system blocks created from the UI now point to the right base_path
• Support for unmapped values to Task.map has been added as requested by Club42 members
• The deployment build command now supports an optional output flag to customize the name of the deployment.yaml file, to better support projects with multiple flows

We're thrilled to announce that, with this release, Prefect 2.0 has exited its public beta! Hopefully, this release comes as no surprise. It is the culmination of nearly a year of building in public and incorporating your feedback. Prefect 2.0 is now the default version of the open source prefect framework provided upon installation. We will continue enhancing Prefect 2.0 rapidly, but future breaking changes will be less frequent and more notice will be provided.

Prefect 2.0 documentation is now hosted at docs.prefect.io. Prefect 1.0 documentation is now hosted at docs-v1.prefect.io.

Upgrading from Prefect 1.0

Flows written with Prefect 1.0 will require modifications to run with Prefect 2.0. If you're using Prefect 1.0, please see our guidance on Discourse for explicitly pinning your Prefect version in your package manager and Docker, so that you can make the transition to Prefect 2.0 when the time is right for you. See our migration page to learn more about upgrading.

Upgrading from earlier versions of Prefect 2.0

We have shipped a lot of breaking changes to Prefect 2.0 over the past week. Most importantly, recent changes to deployments required that schedules for all previously created deployments be turned off. You can learn more about the changes via the deployments concept documentation, the tutorial, or the discourse guide.

Changelog

Fixes

• Fix bug where S3Result could fail to deserialize if pickled with mismatched Prefect versions - #5978

Task library

• Fix sftp-task imports - #5971
• Fix definition of Databricks GitSource at task initialization - #5974
• Add container parameter to ReadNamespacedPodLogs - #5979

Contributors

• Bral Bral
• Edmondo Porcu
• Alvaro Duran Tovar

Changelog

Enhancements

• Add support for ExtraArgs on S3 result upload - #5887
• Add configuration options for the client's rate limit backoff - #5823

Fixes

• Allow untracked flows to be run during a tracked flow run - #5782
• Fix bug with infinite loop when parsing DST cron schedules - #5957
• Fix issue where complex Python dependencies could break Docker storage builds - #5860
• Fix issue where Git storage could not be serialized without a repo - #5877
• Fix issues with grpcio builds on the Prefect base image with Python 3.10 - #5832

Task library

• Instantiate task kwargs during DbtCloudRunTask.__init__ - #5831
• Add boto_kwargs support to S3List - #5907
• Fix DatabricksSubmitMultitaskRun inputs such as access_control_list - #5836
• Add single_user_name to Databricks NewCluster - #5903
• Allow extra arguments to Databricks NewCluster - #5949
• Add git_source argument to DatabricksSubmitMultitaskRun - #5958
• Add tasks for the Toloka API - #5865
• Add tasks for Azure datafactory - #5921
• Update the DbtShellTask return type to match ShellTask - #5872
• Fix handling for Airbyte schedule keys - #5878

Contributors

• Bryan "Beege" Berry
• Cat Zhang
• EmilRex
• Max Firman
• Nate Nowack
• Nico Neumann
• Robert Phamle
• Vladislav Moiseev
• edmondo1984

Changes

Enhancements

• Add inference of Docker network mode for "host" and "none" networks - #5748
• Add Python 3.10 support - #5770
• Raise error on task initialiation when positional-only parameters are present in function signature - #5789
• Add flag to prevent printing ASCII welcome message 5619
• Allow the Prefect client to retry connections for HTTP targets - #5825

Task Library

• Adds SFTP server tasks SftpUpload and SftpDownload #1234
• Configure logging output for AirbyteConnectionTask - #5794
• Make artifacts optional in StartFlowRun - #5795
• Use json instead of dict for DatabricksSubmitMultitaskRun - #5728
• Fix defect in serialization of Great Expectation's results in LocalResult - #5724
• Add an optional data_security_mode to Databricks cluster configuration. - #5778

Fixes

• Fix bug where Prefect signals in tasks were not re-raised by the process-based timeout handler - #5804
• Update flow builds to be deterministic when upstream and downstream slug are same - #5785

Contributors

• David McGuire
• Jaakko Lappalainen
• Jason Bertman
• Joël Luijmes
• Karthikeyan Singaravelan
• Mate Hricz
• Nico Neumann
• Robert Phamle

Changes

Enhancements

• Add ability to set a max_duration timeout in wait_for_flow_run task - #5669
• Add pipe support for EdgeAnnotation types, e.g. map - #5674
• Add 'gs' as a valid filesystem schema for reading specifications - #5705
• Add REPL mode for CLI - #5615

Fixes

• Fix bug where settings the backend to "server" would not prevent client from requesting secrets from the API - #5637
• Fix docker-in-docker issue in DockerAgent on Windows - #5657
• Fix graphviz syntax error when visualizing a flow with a task which is a mapped lambda - #5662
• Allow prefect run parameters to include equals ("=") signs - #5716

Task library

• Add HightouchRunSync task - #5672
• Fix DbtCloudRunJob task failing with nested input for additional_args - #5706"
• Fix Databricks new cluster API params: autoscale and policy_id - #5681

Contributors

• Alessandro Lollo
• Anders Segerberg
• Ben Ayers-Glassey
• Dominick Olivito
• Karthikeyan Singaravelan
• Mahmoud Lababidi
• limx0
• oscarwyatt
• satoshiking

Changes

Features

• Add retry_on to allow tasks to retry on a subset of exception types - #5634

Enhancements

• Add ability to add capacity provider for ECS flow runs - #4356
• Add support for default values to DateTimeParameter - #5519
• Calling flow.run within a flow definition context will raise a RuntimeError - #5588
• Add support for service principal and managed identities for storage on Azure - #5612

Task Library

• The azureml-sdk dependency has been moved from the azure extra into azureml - #5632
• Add task to create materializations with Transform - #5518
• Add create_bucket to GCSCopy - #5618

Fixes

• Fix issue where the FlowRunView could fail to initialize when the backend has no state data - #5554
• Fix issue where adaptive Dask clusters failed to replace workers - #5549
• Fix issue where logging in to Cloud via the CLI could fail - #5643

Contributors

• Alessandro Lollo
• Emre Akgün
• Josh Wang
• Panagiotis Simakis
• eedokl

Changes

Thanks to our many contributors!

Features

• Add .pipe operator to prefect.Task for functional chaining - #5507
• Add Kubernetes authentication support to VaultSecret - #5412

Enhancement

• Allow tasks to consume self as an argument - #5508
• Improve the default idempotency key for create_flow_run task when mapping during a local flow run - #5443

Fixes

• Fix the broken URL displayed in entrypoint.sh - #5490
• Fix zombie processes created by Hasura container during prefect server start - #5476

Task Library

• Add Airbyte configuration export task - #5410
• Update Glob task to accept a string path - #5499
• Fix pod logging while using RunNamespacedJob - #5514
• Add include_generated_sql option to CubeJSQueryTask - #5471

Contributors

• Alessandro Lollo
• Andrew Klimovski
• Brett Polivka
• Jamie Dick
• Michael Milton
• Paul Gierz
• VincentAntoine
• pseudoyim

🎉

See the latest documentation and our release blog post.

Highlights

• Authentication with tokens has been removed; use API keys instead. - #4643
• Python 3.6 is no longer supported; use Python 3.7+ instead. - #5136
• Flow Environments have been removed; use RunConfigs instead. - #5072, docs
• We have a new Discourse community to encourage lasting discussions.

Breaking Changes

• The AWS Fargate agent has been removed; use the ECS agent instead. - #3812
• DockerAgent(docker_interface=...) will now raise an exception if passed. - #4446
• Agents will no longer check for authentication at the prefect.cloud.agent.auth_token config key. - #5140

• Executors can no longer be imported from prefect.engine.executors; use prefect.executors instead. - #3798
• Parameter is not importable from prefect.core.tasks anymore; use prefect.Parameter instead.
• Exceptions are no longer importable from prefect.utilities.exceptions; use prefect.exceptions instead. - #4664
• Client.login_to_tenant has been renamed to Client.switch_tenant.

• The prefect register flow command has been removed; use prefect register instead. - #4256
• The prefect run flow command has been removed; use prefect run instead. - #4463
• Authentication token CLI commands create-token, revoke-token, list-tokens have been removed; use API keys instead. - #4643
• prefect auth login no longer accepts authentication tokens. - #5140
• prefect auth purge-tokens has been added to delete the Prefect-managed tokens directory. - #5140

• The log_to_cloud setting is now ignored; use send_flow_run_logs instead. - #4487

Enhancements

• Update LocalDaskExecutor to use new Python futures feature. - #5046
• Add a State.__sizeof__ implementation to include the size of its result for better scheduling. - #5304
• Allow the cancellation event check to be disabled in the DaskExecutor. - #5443
• Update Flow.visualize() to allow change in orientation. - #5472
• Allow ECS task definition role ARNs to override ECS agent defaults. - #5366

Task Library

• Add DatabricksGetJobID to retreive Databricks job IDs with a given name. - #5438
• Add AWSParametersManager task to retrieve value from AWS Systems Manager Parameter Store. - #5439
• Update SpacyNLP task to support spacy version >= 3.0. - #5358
• Add exclude parameter to SpacyNLP task. - #5402
• Update the AWSSecretsManager task to parse non key-value type secrets. - #5451
• Update the DatabricksRunNow task to use the Databricks 2.1 jobs API. - #5395
• Add ge_checkpoint and checkpoint_kwargs parameters to RunGreatExpectationsValidation to allow runtime configuration of checkpoint runs. - #5404
• Add support for overwriting existing blobs when using Azure BlobStorageUpload task. - #5437
• Add Neo4jRunCypherQueryTask task for running Cypher queries against Neo4j databases. - #5418
• Add DatabricksSubmitMultitaskRun task to run Databricks jobs with multiple Databricks tasks. - #5395

Fixes

• Add support to prefect.flatten for non-iterable upstreams, including exceptions and signals. - #4084
• While building Docker images for storage, rm=True is used as default, which deletes intermediate containers. - #5384
• Use __all__ to declare Prefect's public API for Pyright. - #5293
• Fix usage of sys.getsizeof to restore support for PyPy. - #5390
• Fix issues with log size estimates from #5316. - #5390

Contributors

• Alessandro Lollo
• Aneesh Makala
• Connor Martin
• Gebing
• Julio Faracco
• Kevin Mullins
• Mathijs Miermans
• Oliver Mannion
• Raymond Yu

See the list of changes in the changelog.