Changelog

April 14, 2021

Released on April 14, 2021.

Enhancements

• Added support for Bitbucket cloud into Bitbucket storage class #4318
• Update docs for API keys - #4313
• Add option to disable the deletion of finished Prefect jobs in the Kubernetes agent - #4351
• Improve messaging when a flow is skipped during registration - #4373
• Display a more helpful error when calling a task outside a flow context - #4374
• Lower required docker-compose version and add note to docs - #4396
• Increase healthcheck intervals for Prefect Server - #4396
• Allow the ShellTask to be used on win32 - #4397

Fixes

• Fix ShellTask docstring - #4360
• Fix incorrect unused task tracking - #4368
• Add error handling to timeouts that fail during result pickling or passing - #4384

Contributors

• Jonathan Wright

Names are hard and stressful.

Changelog

April 05, 2021

Released on April 5, 2021.

Enhancements

• Add terminal flow state handler override - #4198
• When manually set, prefect.context.date will be used to determine dependent values - #4295
• prefect.context.date will be cast to a DateTime object if given a parsable string - #4295
• Expand logging for DaskExecutor, including the cluster dashboard address (if available) - #4321
• Add ability to stream ShellTask logs with level INFO - #4322
• Add architecture diagram to docs - #4187
• Speed up flow validation logic - #4347
• Pin dask upper package versions - #4350

Task Library

• Add first basic implementation of Soda SQL Scan task to run DQ tests using soda-sql library
• Add new task KubernetesSecret - #4307

Fixes

• Fix DatabricksRunNow task attribute override behavior - #4309
• Use default flow labels when triggering flow runs from CLI - #4316
• Improve ECS agent error messages, and fix bug that prevented using ECS agent on Fargate with ECR - #4325
• Support imports from local directory when registering/building flows via CLI - #4332
• Speedup flushing of logs to cloud/server on flow run shutdown, avoiding lost logs on platforms that SIGKILL the process after a short period - #4334

Contributors

• Alessandro Lollo
• David Zucker
• Greg Lu
• James Lamb
• Sean Talia

Changelog

Released on March 25, 2021.

Task Library

• Ensures Snowflake Query Task output is serializable [#3744] (https://github.com/PrefectHQ/prefect/issues/3744)

Fixes

• Always load task run info prior to running a task - #4296

Changelog

Released on March 24, 2021.

Features

• Add new improved prefect register CLI command. This command supports registering multiple flows at once (through multiple file paths or directories), and also includes a new --watch flag for watching files and automatically re-registering upon changes. - #4256
• New prefect build CLI command for building flows. Artifacts produced by this command can then be used by prefect register to register flows without requiring the source. - #4282

Enhancements

• Use explicit exception chaining #3306
• Add as_bytes option to S3Download task - #4238
• Improve error message when a loaded flow doesn't match the version stored in Prefect Cloud/Server - #4259
• Support setting flow run labels from cli - #4266
• Support setting default image in --job-template/--task-definition in Kubernetes/ECS agents - #4270

Task Library

• Adds logging of cell outputs to Jupyter task - [#4265] (https://github.com/PrefectHQ/prefect/issues/4265)
• Add user and password as runtime parameters to Exasol tasks - #4268

Fixes

• Fix bug where sometimes the global prefect.context wouldn't be respected during a flow run - #4287

Deprecations

• Deprecate the old prefect register flow CLI command in favor of prefect register - #4256
• Deprecate user and password arguments to Exasol task constructors in favor of runtime parameters - #4268

Contributors

• Abid Ahmad
• Alexandr N. Zamaraev
• Jacob Hayes
• Timo S.

Changelog

0.14.12

Released on March 10, 2021.

Enhancements

• Upgrade hasura to 1.3.3 in Prefect Server - #4126
• Add --docker-client-timeout flag to docker agent, for configuring the timeout for all docker API requests - #4232
• Make --slug flag optional in prefect server create-tenant - #4240
• Move command, environment, cpu, memory, execution_role_arn, and task_role_arn configuration for ECS tasks from definition time to run time in the ECS agent - #4211
• Register (and deregister) a new task definition for every flow run in ECS agent - #4211

Task Library

• Adds new filesystem Copy and Remove tasks - #4202

Fixes

• Don't forward nout to mapped tasks - #4206
• Fix Task signature generation in the presence of with variadic kwargs - #4235
• Ensure Flow is serializable using pickle - #4209

Contributors

• Ben Fogelson
• Marwan S.
• Timo S.

Changelog

0.14.11

Released on March 3, 2021.

Features

• Add command prefect server config to output configured docker-compose yaml - #4176

Enhancements

• Add project_name and project_id to context during Cloud/Server flow runs - #4083
• Better error message when flow not found in file - #4182
• Improve generated names for GetItem tasks - #4183
• Add base_url option to GitHub storage - #4194

Task Library

• Add filehandling tasks Move, Unzip, Zip - #4131
• Add msg_plain, email_to_cc, and email_to_bcc options to EmailTask - #4157
• Add jar_params option to DatabricksRunNow task - #4157

Fixes

• Make task slug generation robust to modifying existing task names - #4189
• Forward client_options to S3Result from S3 storage - #4195

Contributors

• David Zucker
• Jacob Hayes
• Joël Luijmes
• Timo S.
• Yogi Patel

Changelog

0.14.10

Released on February 23, 2021.

Fixes

• Dynamically import compression libraries for CompressedSerializer - #4150
• Support passing environment variables containing = through agent CLI --env flag - #4160

Changelog

0.14.9

Released on February 16, 2021.

Enhancements

• Add CompressedSerializer class - #4063
• Allow Client timeout seconds to be configurable through configuration - #4118

Task Library

• Update FilterTask to allow logging the filtered output via a function - #4121
• Add FivetranSyncTask, to manage your Fivetran connector sync process - #4116

Fixes

• Reduce the number of boto3 clients created across Prefect when interacting with AWS services - #4115

Deprecations

• Deprecate the use_session argument in all AWS-related components - #4115

Contributors

• Amanda Wee
• Andrew Hannigan
• Craig Wright
• Nick Acosta
• Timo S.

Changelog

0.14.8

Released on February 11, 2021.

Enhancements

• Add option to provide version group ID to prefect run flow CLI command - #4100

Fixes

• Fix bug in agent healthcheck route that was introduced in 0.14.7 - #4109

Changelog

0.14.7

Released on February 10, 2021.

Enhancements

• Support multiple docker networks with Docker Agent - #3986
• Add healthchecks to prefect server - #4041
• Raise custom TimeoutError for task timeouts to allow for more granular user control - #4091
• Add access_token_secret to GitHub, GitLab, and Bitbucket storage, making the Prefect secret containing an access token to these services configurable - #4059
• Add --skip-if-flow-metadata-unchanged to prefect register flow CLI command that avoids bumping flow version if flow metadata has not changed - #4061
• Add --skip-if-exists to prefect create project CLI command that safely skips if the project has already been created - #4061
• Add new Module storage class, for referencing flows importable from a Python module - #4073
• Drop resource limits from manifest generated using prefect agent kubernetes install - #4077

Task Library

• Add new tasks for communication with an Exasol database - #4044

Fixes

• Fix task decorator chaining by using inspect.unwrap instead of __wrap__ - #4053
• Forward Prefect backend type to deployed flow runs, ensuring backend-specific logic functions properly - #4076
• Patch around bug in dask's multiprocessing scheduler introduced in Dask 2021.02.0 - #4089

Deprecations

• Docker agent network kwarg deprecated in favor of networks - #3986

Breaking Changes

• Remove unused Storage.get_env_runner method - #4059
• Remove private utilities in prefect.utilities.git - #4059

Contributors

• Alex P.
• Marwan S.
• Peter Roelants
• Timo S.

Changelog

0.14.6

Released on February 2, 2021.

Enhancements

• Add option to provide flow ID to run flow CLI command - #4021
• Flow name and project are no longer required options when calling run flow CLI command - #4021

Task Library

• Add GCSBlobExists which checks for the existence of an object in a given GCS bucket - #4025
• Use boto3 session in S3Upload and S3Download tasks, to ensure thread-safe execution - #3981

Fixes

• Fix issue with fixed duration Paused states not resuming properly - #4031

Contributors

• Gregory Roche

Changelog

0.14.5

Released on January 26, 2021.

Enhancements

• S3 storage now logs ETag, LastModified timestamp, and VersionId (if present) when loading a flow - #3995
• GitHub storage now logs the commit sha used when loading a flow - #3998
• GitHub storage now loads from a repo's default branch, allowing default branch names other than 'master' - #3998
• Improve error message when Secrets are missing with Server - #4003
• Better error message when passing parameters to StartFlowRun constructor - #4008
• Add warning if user-defined class shadows an attribute used by the base class - #4011
• Add support for EXTRA_PIP_PACKAGES environment variable in prefecthq/prefect images, simplifying installation of dependencies during development - #4013
• Add execution role arn parameter to ecs run config and agent - #4015

Task Library

• Add ConnectGetNamespacedPodExec task which runs an exec command in provided pod container - #3991
• Ensure connection secrets can be passed to Databricks tasks at runtime - #4001

Fixes

• Fix Agent registration possibly skipping on server connection issues - #3972
• GCSUpload task now explicitely fails when ran on non-supported types - #3978
• Make logging to Prefect cloud more robust in the presence of errors or process shutdown - #3989
• Handle setting state for missing flow runs in Kubernetes agent resource management - #4006

Contributors

• Loïc Macherel
• Thomas Baldwin

Changelog

0.14.4

Released on January 19, 2021.

Enhancements

• Retry on additional status codes - #3959
• Rerun secret tasks on flow-run restart - #3977

Task Library

• Stream log output from Kubernetes RunNamespacedJob - #3715
• Add ReadNamespacedPodLogs which reads or streams logs from Kubernetes pod - #3715
• Add SQL Server task to query SQL Server databases - #3958
• Add chunking to GCP storage tasks - #3968

Fixes

• Properly handle NotImplementedError exceptions raised by a result's serializer - #3964
• Fix support for storing multiple flows in a single script in storage - #3969
• Fix regression in apply_map which prevented use in case/resource_manager blocks - #3975

Contributors

• Joël Luijmes
• Peyton Runyan
• wangjoshuah

Changelog

0.14.3

Released on January 13, 2021.

Enhancements

• Better errors/warnings when flow fails to load in execution environment - #3940

Task Library

• Add an Asana task to add tasks to an asana project - #3935

Fixes

• Fix prefect server start failure when given a custom graphql host port - #3933
• Fix Kubernetes Agent attempting to report container info for failed pods when no container statuses are found - #3941
• Avoid race condition when creating task run artifacts for mapped tasks - #3953
• Propogate agent labels info to k8s flow runs, to match other agent behavior - #3954

Changelog

Released on January 6, 2021.

Features

• Support for specifying run_config for an individual flow run - #3903

Enhancements

• Allow the usage of a profile_name on get_boto_client - #3916
• Support executing Prefect agents/flows without having the prefect CLI on path - #3918

Task Library

• Add support for specifying a run_config in StartFlowRun - #3903
• Task to add Trello card for task library - #3910

Fixes

• Remove unused description field on Task serializer - #3917
• Fix edge case in apply_map that resulted in cycles in the Flow graph - #3920
• Support storing multiple local flows with the same name when using Local storage - #3923
• Fix bug in prefect.context contextmanager that resulted in context fields reverting to their initially configured values - #3924

Contributors

• Albert Franzi
• Allyson Lubimir

Changelog

0.14.1

Released on December 29, 2020.

Enhancements

• Make setup method optional for resource_manager tasks - #3869
• Add labels to all containers managed by the docker agent - #3893
• Add prefect server stop command for stopping the server - #3899
• Add --detach to prefect server start for running the server in the background - #3899

Fixes

• Add support for google-cloud-storage < 1.31.0 - #3875
• Fix use of imagePullSecrets/serviceAccountName in k8s agent - #3884
• Fix read_bytes_from_path to work properly with S3 - #3885
• Change default idempotency_key in StartFlowRun to use task_run_id instead of flow_run_id - #3892

Changelog

0.14.0

Released on December 16, 2020.

Features

• New flow run configuration system based on RunConfig types, see here for more info

Enhancements

• Kubernetes Agent now reports events for pending pods created by prefect jobs - #3783
• Using --rbac for Kubernetes Agent install command now includes the events resource - #3783
• Add orchestration-based dependencies to the prefecthq/prefect Docker image - #3804
• Add a slimmed down prefecthq/prefect:core Docker image that only contains base dependencies - #3804
• Docker storage now installs all orchestration-based dependencies when using default image - #3804
• Add warning on flow registration if flow.executor is set but the flow is using the legacy flow.environment configuration system - #3808
• Echoing prefect config file as JSON to be able to parse it with jq in the terminal - #3818
• Produce artifact for RunGreatExpectationsValidation even if validation fails - #3829
• execute flow-run command now sends flow run log in the case of an error - #3832
• Changed name of logs raised by the Kubernetes Agent if they stem from infrastructure events - #3832
• Add tini to the official Prefect docker images - #3839
• Remove task run level heartbeats for performance - #3842

Task Library

• Fix mising job_id in DatabricksRunNow task initialization - #3793

Fixes

• Fix Azure result byte decoding of blob data - #3846
• Prefect kubernetes agent no longer relies on existence of any fields in configured Kubernetes Job Template - #3805
• Accept old envvar style configuration for Kubernetes agent for --service-account-name/--image-pull-secrets options - #3814
• Pass as_user=False when using client.get_cloud_url in StartFlowRun - #3850
• Fix AWS boto3 utility passing duplicate kwargs to client initialization - #3857

Deprecations

• Storage classes have been moved from prefect.environments.storage to prefect.storage, the old import paths have been deprecated accordingly - #3796
• Executor classes have been moved from prefect.engine.executors to prefect.executors, the old import paths have been deprecated accordingly - #3798
• Deprecated use of storage_labels boolean kwarg on local agent - #3800
• Deprecated use of --storage-labels option from agent start CLI command - #3800
• Deprecates all Environment classes - users should transition to setting flow.run_config instead of flow.environment - #3811
• Deprecate the Fargate Agent in favor of the ECS Agent - #3812

Breaking Changes

• Using in-cluster installs of the Kubernetes Agent now requires RBAC for the events resource - #3783
• Removed setting of default labels on storage objects and the local agent - #3800
• Remove deprecated RemoteEnvironment/RemoteDaskEnvironment - #3802
• Remove deprecated executor_kwargs argument to KubernetesJobEnvironment/FargateTaskEnvironment - #3802
• Remove deprecated prefect run cloud/prefect run server CLI commands - #3803
• Remove deprecated prefect execute cloud-flow CLI command - #3803
• Stop building the prefecthq/prefect:all_extras image and switch flow deployment default to using prefecthq/prefect:{core_version} - #3804
• Flows now use RunConfig based deployments by default - legacy Environment based deployments are now opt-in only - #3806
• Remove deprecated prefect.contrib module - #3813
• Remove all references to result handlers and safe results - #3838
• Remove option to enable deprecated Kubernetes resource manager in agent install CLI command - #3840

Contributors

• Christian Werner
• Erich Oliveira
• Jacob Hayes
• Pedro Martins

Changelog

0.13.19

Released on December 8, 2020.

Enhancements

• Use explicit exception chaining - #3306
• Support Bitbucket as storage option - #3711
• Surface pod failures and container errors in jobs deployed with the Kubernetes Agent - 3747
• Support timeout option in GCS tasks - #3732
• Added storage option for AWS CodeCommit - #3733
• Add the image used for a flow-run to the flow run environment as prefect.context.image - #3746
• Add UniversalRun run-config that works with all agents - #3750
• Support flows that have no run-config or environment - #3750
• Allow Docker storage environment vars to be used in commands - #3755
• Add service_account_name and image_pull_secrets options to KubernetesRun and KubernetesAgent - #3778
• Add a new Client function delete_project - #3728

Task Library

• Add task to fetch data from Dremio - #3734
• Add RunGreatExpectationsValidation task - #3753
• Add the option to post markdown artifacts from the RunGreatExpectationsValidation task - #3753

Fixes

• Fix state attempting to read result from absent upstream result - #3618
• Replace deprecated download_as_string method with download_as_bytes method - #3741
• Fix default image whenever working on a non-tagged commit - #3748
• Fix type-casting for task timeout defaults loaded from config - #3761
• Fix the ref default on GitHub storage - #3764
• Fix rare cancellation bug when running with external Dask cluster - #3770

Deprecations

• Deprecated the RunGreatExpectationsCheckpoint task in favor of RunGreatExpectationsValidation - #3766

Contributors

• BluePoof
• Faris ALSaleem
• Jonathan Owen
• Klemen Strojan
• Phillip Choi
• Sam Bail
• Takayuki Hirayama

Changelog

0.13.18

Released on November 30, 2020.

Enhancements

• Display formatted graphql errors on client request failure - #3632
• Refactor Core Client API calls for performance - #3730

Task Library

• Refactor execute query code for PostgresExecute, PostgresExecuteMany, and PostgresFetch tasks - #3714
• Fix PicklingError in BigQueryLoadFile and BigQueryLoadGoogleCloudStorage - #3724
• Allow custom exporter for ExecuteNotebook task - #3725
• Properly forward location parameter in bigquery tasks - #3726
• Support passing helper_script to ShellTask/DBTShellTask at runtime - #3729

Fixes

• Fix bug with docker storage throwing exception while trying to display output - #3717

Contributors

• Amanda Wee
• Panagiotis Simakis
• Swier Heeres
• Takayuki Hirayama

Changelog

0.13.17

Released on November 24, 2020.

Features

• Improved support for Tasks returning multiple results - #3697

Enhancements

• Allow chaining of Task imperative dependency calls - #3696
• Add task_definition_arn to ECSRun run-config - #3681
• Rerun resource_manager tasks when restarting flows from failed - #3689
• Raise nice warning if user passes Task instance to Task constructor, rather than when calling the Task (or using Task.map/Task.set_dependencies) - #3691
• Always use tenant slug in output of Client get_cloud_url function - #3692

Task Library

• Add enhancement to StartFlowRun task to create link artifact for started flow run - #3692
• Add a new postgres task PostgresExecuteMany - #3703
• Add debug logging for Docker tasks PullImage and BuildImage - #3672
• ShellTask returns output on failure - #3649
• ShellTask allows streaming of output independently of the number of lines returned - #3649

Fixes

• Make serialized_hash handle unordered task sets correctly - #3682
• Docker storage build error logs were not always displayed - #3693
• Fix automatic quoting of Docker storage environment variable values - #3694
• Use exist_ok flag in os.makedirs to avoid race condition in local storage class - #3679

Contributors

• Faris ALSaleem
• R Max Espinoza
• Takayuki Hirayama