Changes

Enhancements

• Ensure that maximum log payload sizes are not exceeded - #5316

Server

• Upgrade Hasura to v2.1.1 which includes support for Apple M1 - #5335

Fixes

• Fix bug where logout was required before logging in with a new key if the new key does not have access to the old tenant - #5355

Task Library

• Fix bug where the Airbyte sync job failure would not be reflected in the task state - #5362

Changes

Enhancements

• Allow passing timedeltas to create_flow_run to schedule subflows at runtime - #5303
• Upgrade Prefect Server Hasura image to 2.0.9 - #5173
• Allow client retries on failed requests to Prefect Server - #5292

Task Library

• Add authentication parameter for Snowflake query tasks - #5173
• Add Mixpanel tasks - #5276
• Add Zendesk Tickets Incremental Export task - #5278
• Add Cube.js Query task - #5280
• Add Monte Carlo lineage tasks - #5256
• Add Firebolt task - #5265
• Add custom domain support to dbt Cloud tasks for enterprise customers - #5273
• Fix response key in Airbyte task health check - #5314
• Allow all Postgres task parameters to be configured at runtime - #4377
• Fix AirbyteConnectionTask requiring optional parameters - #5260
• Allow StepActivate task to receive runtime parameters - #5231

Fixes

• Fix bug where null run_config field caused deserialization errors in backend views - #1234

Contributors

• Adam Brusselback
• Ahmed Ezzat
• Alessandro Lollo
• Connor Martin
• Dennis Hinnenkamp
• Gaylord Cherencey
• Henning Holgersen
• Mathijs Miermans
• Michał Zawadzki
• Raghav Sharma

Changes

Released on December 22, 2021.

Enhancements

• Allow passing kwargs to Merge task constructor via merge() function - #5233
• Allow passing proxies to slack_notifier - #5237

Fixes

• Update RunGreatExpectationsValidation task to work with latest version of great_expectations - #5172
• Allow unsetting kubernetes imagePullSecrets with an empty string - #5001
• Improve agent handling of kubernetes jobs for flow runs that have been deleted - #5190
• Remove beta1 from kubernetes agent template - #5194
• Documentation improvements - #5220, #5232, #5288

Contributors

• Connor Martin
• Farley Farley
• Vincent Chéry

Changelog

Released on November 30, 2021.

Enhancements

• Add end_time to FlowRunView.get_logs - #5138
• Update watch_flow_run to stream logs immediately instead of waiting for flow run state changes - #5138
• Allow setting container ports for DockerRun - #5130
• Clarify ECSRun documentation, especially the ambiguities in setting IAM roles - #5110
• Fix deprecated usage of marshmallow.fields.Dict in RRule schedules - #4540, #4903

Fixes

• Fix connection to local server instances when using DockerAgent on linux - #5182

Task Library

• Add support for triggering Airbyte connection sync jobs using AirbyteConnectionTask - #5078
• Add artifact publishing to DbtCloudRunJob task - #5135
• Add support for running data quality checks on Spark DataFrames using soda-spark - #4901

Contributors

• Alessandro Lollo
• Nicholas Hemley
• Shahil Mawjee

This hotfix release fixes an issue where the kubernetes agent would attempt to load a secret value and fail if it was not present.

See the PR for details.

Don't miss all the exciting changes from 0.15.8 released today as well.

Features

• Add support for rich iCal style scheduling via RRules - #4901
• Add Google Cloud Vertex agent and run configuration - #4989

Enhancements

• Allow Azure flow storage to overwrite existing blobs - #5103
• Provide option to specify a dockerignore when using Docker storage - #4980
• Add keep-alive connections for kubernetes client API connections - #5066
• Add idempotency_key to create_flow_run task - #5125
• Add raise_final_state to wait_for_flow_run task to reflect child flow run state - #5129

Task Library

• Bump maximum google-cloud-bigquery version to support 2.x - #5084
• Add Glob task for collecting files in directories - #5077
• Add DbtCloudRunJob task for triggering dbt cloud run jobs - #5085
• Added Kafka Tasks entry to website docs - #5094

Fixes

• Update the FlowView to be more robust to serialized flow changes in the backend - #5116

Deprecations

• Move artifacts functions to prefect.backend.artifacts - #5117

Server

This release includes a Prefect Server update that updates an upstream dependency to fix a security vulnerability. See the release changelog for more details.

Contributors

• Alessandro Lollo
• Bradley Axen
• Damien Ramunno-Johnson
• Jonas Miederer
• Josh Wang
• Nitay Joffe
• Timo S.
• Brett Naul

Enhancements

• Add flatten support to apply_map - #4996
• Add dask performance report to DaskExecutor - #5032
• Update git storage repo parameter to be optional if specifying git_clone_url_secret_name - #5033
• Add task_run_name to prefect.context - #5055

Fixes

• Reduce rate limit related failures with the ECS agent - #5059

Task Library

• Add data parameter to SQLiteQuery task - #4981
• Allow EmailTask to use insecure internal SMTP servers with smtp_type="INSECURE" - #5012
• Fix Databricks run_id mutation during task runs - #4958
• Add manual setting to FivetranSyncTask allowing retention of Fivetan scheduling -#5065

Contributors

• Daniel Saxton
• Emre Akgün
• Jessica Smith

Enhancements

• Improve setting the Azure storage connection string - #4955
• Allow disabling retries for a task with max_retries=0 when retries are globally configured - #4971

Task Library

• Allow Exasol Tasks to handle Prefect Secrets directly - #4436
• Adding Census Syncs to the task library - #4935

Fixes

• Fix bug where LocalDaskExecutor did not respond to a PrefectSignal - #4924
• Fix PostgresFetch with headers for one row - #4968
• Fix bug where apply_map could create acyclic flows - #4970

Contributors

• Donny Flynn
• Noah Holm
• Timo S.

Released on September 2, 2021.

Features

• Python 3.9 docker images are now published - #4896

Enhancements

• Add --expose flag to prefect server cli to make the Core server and UI listen to all interfaces - #4821
• Pass existing/local environment variables to agentless flow runs - #4917
• Add --idempotency-key to prefect run - #4928
• Add support for larger flow registration calls - #4930
• Ignore schedules by default for CLI flow runs and add flag to run based on schedule for local only runs #4817

Task Library

• Feature: Added SnowflakeQueryFromFile task #3744
• Enhancement: Log boto exceptions encountered in the in AWS BatchSubmit task - #4771
• Breaking: Legacy Dremio authentication has been updated to the new pattern in DremioFetch - #4872
• Fix: Use runtime arguments over init arguments instead of ignoring them for MySQL Tasks - #4907

Fixes

• Adjust log limits to match backend logic for better UX - #4900
• Fix use of marshmallow.fields.Dict to use keys as a kwarg rather than key. - #4903
• API server settings are passed correctly to task workers when using Prefect Server - #4914
• Do not attempt attempt to set host_gateway if using an unsupported Docker Engine version - #4809
• Ignore jobs without a flow_run_id label in KubernetesAgent.manage_jobs - #4934

Breaking Changes

• Services run by prefect server cli are now local by default (listen to localhost instead of 0.0.0.0); use --expose if you want to connect from a remote location - #4821
• The changes in flow registration require Prefect Server 2021.09.02. Prefect Server will need to be upgraded before flows can be registered from this version - #4930

Contributors

• Deepyaman Datta
• Henri Hannetel
• Johnny Tirado
• Kathryn Klarich
• Tenzin Choedak
• Vincent Xue

Changelog

0.15.4

Released on August 17, 2021.

Docs

• Add a getting started section with a quick start guide for both core and orchestration sections - #4734

Enhancements

• Expose Snowflake cursor type to SnowflakeQuery task arguments #4786
• Add ability to use threaded flow heartbeats - #4844
• Improve behavior when API rate limits are encountered - #4852
• Allow custom git clone url for Git storage - #4870
• Add on_worker_status_changed callback to the DaskExecutor - #4874
• Add --agent-config-id to prefect agent <kubernetes|local> install - #4876

Task Library

• Add new prometheus task to push to gateway - #4623

Fixes

• Fix binding of named volumes to flow containers with Docker agent - #4800
• Fix ImportError typo in dropbox module - #4855
• Fix default safe char for gitlab storage repo path - #4828

Contributors

• Austen Bouza
• David Zucker
• Jacob Dawang
• Tom Forbes
• Mat

Enhancements

• Add new evaluation_parameters parameter to RunGreatExpectationsValidation task - #4798

Fixes

• Fix create_flow_run compatibility with auth tokens - #4801
• Fix auto-quoting for strings that begin with numeric characters - #4802

Contributors

• Michal Zawadzki

Enhancements

• Allow CLI registration of flows without starting their schedule prefect register --no-schedule - #4752
• Add host_config to DockerRun to expose deeper settings for Docker flow runs - #4733
• Enable loading additional repository files with Git storage - #4767
• Update flow run heartbeats to be robust to exceptions - #4736
• Allow prefect build/register paths to contain globs for recursion - #4761

Fixes

• Fix duplicate task runs in FlowRunView.get_all_task_runs - #4774
• Fix zombie processes from exited heartbeats - #4733
• Missing auth_file directory is created when saving credentials - #4792

Task Library

• Add VaultSecret task for retrieving secrets from private Vault instances - #4656

Contributors

• Gabriel Montañola
• Nelson Griffiths
• Pawel Janowski
• Yueh Han Huang
• Chris Ottinger

Enhancements

• Add documentation for querying role and membership info - #4721
• Checkpoint task results when SUCCESS is raised - #4744

Fixes

• Fix loading of PREFECT__CLOUD__API_KEY environment variable when starting agents - #4751
• Fix bug where the tenant could not be inferred during flow runs while using token auth - #4758
• Fix bug where an agent using token auth could clear the tenant from disk - #4759

0.15.0

Released on July 1, 2021.

Features

• Add objects for inspecting flows, flow runs, and task runs without writing queries - #4426
• Rehaul prefect run CLI for executing flows locally and with agents - #4463
• Add flow run tasks to simplify flow run result passing - #4563
• Add agentless execution for flow runs - #4589
• Add prefect auth create-key to create API keys - #4643
• Add prefect auth list-keys to list API key metadata - #4643
• Add prefect auth revoke-key to revoke an API key - #4643
• Add prefect auth status command to see the state of your authentication - #4643

Enhancements

• Improve flow run documentation with new dedicated section - #4492
• Update Client to support API keys - #4643
• Add API key support to prefect auth login/logout/switch-tenants - #4643
• API keys can be configured in the Prefect config - #4643
• Change SendGrid SendEmail task to use secret value - #4669
• Add TenantView for retrieval of tenant information - #4676
• Add custom rbac documentation - #4696
• Exit with non-zero status on flow run failure when watched - #4709
• Display return code on local agent flow process failure - #4715

Task Library

• Add KafkaBatchConsume and KafkaBatchProduce tasks - #4533

Fixes

• Fix cleanup issue with Git storage on Windows - #4665
• Pass API keys as tokens for compatibility when creating flow run environments - #4683
• Fix missing event timestamp attribute errors in K8s agent - #4693
• Fix backwards compatibility for flows without a terminal_state_handler - #4695
• Raise a better exception when a task run result type is not set in TaskRunView.get_result() - #4708

Deprecations

• Deprecate prefect auth create-token - #4643
• Deprecate prefect auth list-tokens - #4643
• Deprecate prefect auth revoke-token - #4643
• Deprecate setting auth tokens in the Prefect config - #4643
• prefect.utilities.exceptions has been deprecated in favor of prefect.exceptions - #4664

Breaking Changes

• Remove deprecated prefect.environment.storage module
• Remove deprecated DockerAgent kwarg network
• Remove deprecated kubernetes.ResourceManager class
• Remove deprecated prefect agent start/install <agent-type> commands
• Remove deprecated prefect agent local start flag --storage-labels
• Remove deprecated DroboxDownload task kwarg access_token_secret
• Remove deprecated GCS... tasks kwarg encryption_key_secret
• Remove deprecated prefect.tasks.google module
• Remove deprecated prefect.tasks.secret.Secret class
• Remove deprecated Scheduler serializers for Prefect <0.6.0
• Remove deprecated RunGreatExpectionsCheckpoint task
• Remove deprecated OneTimeSchedule and UnionSchedule classes
• Remove deprecated flow run tasks ending in Task
• Remove deprecated prefect.utilities.tasks.unmapped; moved to prefect.utilities.edges.unmapped
• Prefect state signals now inherit from BaseException to prevent accidental capture - #4664
• TaskTimeoutError has been replaced with TaskTimeoutSignal - #4664
• VersionLockError has been replaced with VersionLockMismatchSignal - #4664

Contributors

• Stéphan Taljaard
• Tenzin Choedak

0.14.22

Released on June 15, 2021.

Enhancements

• Use functools.update_wrapper for FunctionTask - #4608
• Add ability to merge reference tasks when combining two flows - #4644
• Add client side check for key value size - #4655
• Ensure stack traces are included in logs during task run exceptions - #4657
• Add poll_interval parameter to StartFlowRun to define the polling interval when waiting for the flow run to finish - #4641
• Add ability to set task timeouts with timedelta objects - #4619

Fixes

• Add ssh documentation - #4539

Contributors

• Pawel Janowski, Recursion
• Peter Roelants

Features

• Add interface for backend key-value metadata store - #4499

Enhancements

• Keep intermediate docker layers when using Docker Storage to improve caching - #4584

Fixes

• Fix possible race condition in LocalResult directory creation - #4587
• Use absolute paths when registering flows in prefect register - #4593
• Propagate storage labels (e.g. hostname label on Local storage) when registering flows with prefect register - #4593
• Fix small-flow parallelism issues with multiprocess LocalDaskExecutor - #4602
• Cleanly handle unpicklable exceptions in tasks - #4605

Contributors

• Tom Forbes

Changelog

0.14.20

Released on May 25, 2021.

Enhancements

• Refactor Agent base class for readability - #4341
• Display the agent config id on agent startup if set - #4524
• Add debug logs during agent auth verification - #4547
• Sending logs to Cloud can be globally disabled via config in addition to the agent flag - #4487

Task Library

• Enable sending attachments with emails in the EmailTask - #4457
• Add Google Cloud Platform GCPSecret task - #4561

Fixes

• Fix import_object handling of submodules that are not attributes - #4513
• Fix DockerStorage building with python slim image - #4523
• Gracefully handle events with missing timestamps in K8s agent - #4544
• Fix bug where agent uses originally scheduled start time instead of latest state time - #4568

Deprecations

• logging.log_to_cloud has been deprecated in favor of cloud.send_flow_run_logs - #4487

Contributors

• Stéphan Taljaard
• Thomas Heyenbrock

Changelog

0.14.19

Released on May 11, 2021 as a hotfix for 0.14.18

Fixes

• Fix docker container name error while using docker agents - #4511

Changelog

0.14.18

Released on May 11, 2021.

Enhancements

• Add an image_pull_policy kwarg to KubernetesRun - #4462
• Allow prefect build/register --module to accept full import path to flow - #4468
• Add hello world flow to prefect module - #4470
• Set docker container names to flow run names when running with the docker agent - #4485

Task Library

• Add basic implementation of SendGrid to Task Library - #4450
• Log link to flow run in StartFlowRun - #4458
• Allow passing in io.BytesIO to GCSUpload - #4482

Fixes

• Fix logging errors within BigQuery task - #4419
• Remove unnecessary docker interface detection in Docker agent - #4446
• Upgrade kubernetes package requirement upper limit - #4452
• Fix Prefect server startup check for custom server port - #4501

Contributors

• Brett Naul
• Joël Luijmes
• Stéphan Taljaard
• Zach Schumacher

Features

• Add git storage - #4418

Enhancements

• Add test coverage for threaded LocalDaskExecutor timeouts - #4217
• Add environment variable support to UniversalRunConfig - #4383
• Adds column name fetching to PostgresFetch task - #4414
• Allow external Postgres with prefect server start command - #4424

Fixes

• Pass reference tasks states instead of task states to terminal_state_handler - #4409
• Check for AWS_RETRY_MODE variable before setting default in ECSAgent - #4417
• Fixed bug from Flow.copy() not copying the slugs dictionary - #4435
• Fix compatibility with PyGithub >= 1.55 - #4440

Contributors

• Ben Fogelson
• Gabriel Montañola