Changelog

0.9.6

Released on Mar 4, 2020.

Features

• Add new diagnostics utility to assist in troubleshooting issues - #2062
• Add a jira_notification state handler to create jira tickets for failed tasks or flows - #1861
• Add support for Python 3.8 - #2080

Enhancements

• Add PIN 15 (skip refactor) - #2070
• Update docs and docstrings related to Result Handlers - #1792
• Add volume option to Docker Agent - #2013
• DaskKubernetesEnvironment now elevates important autoscaling logs as well as possible Kubernetes issues - #2089
• Add optional scheduler_logs kwarg to theDaskKubernetesEnvironment - #2089
• Add ERROR log if heartbeat process dies - #2097
• Enable stdout logging from inside a task with the kwarg log_stdout=True - #2092
• Direct links to Cloud flows and flow runs now shown on creation time - #2109
• Update docs related to using Context - #2077

Task Library

• Fix expanding of V1DeleteOptions kwargs for Kubernetes tasks - #2083

Fixes

• Fix extra_loggers config variable not being able to be set via environment variable - #2089
• Fix environments not passing down their extra_loggers to any created infrastructure - #2089
• Don't mutate data when serializing or deserializing - #2098

Deprecations

• None

Breaking Changes

• None

Contributors

• Romain Thalineau

Changelog

0.9.5

Released on Feb 21, 2020.

Features

• None

Enhancements

• Better exception for unsubscriptable mapping arguments - #1821
• Add DaskGateway tip to docs - #1959
• Upload package to PyPI on tag push to master - #2030
• Improve package import time - #2046

Task Library

• Fix V1DeleteOptions call for Kubernetes tasks - #2050
• Add kwargs to V1DeleteOptions for Kubernetes tasks - #2051

Fixes

• Ensure microseconds are respected on start_date provided to CronClock - #2031
• Fix duplicate Client connections when using --logs flag from run cloud CLI command - #2056

Deprecations

• None

Breaking Changes

• None

Contributors

• Romain Thalineau

Changelog

0.9.4

Released on Feb 14, 2020.

Features

• None

Enhancements

• Add incremental tutorial - #1953
• Improve error handling for unsupported callables - #1993
• Accept additional boto3 client parameters in S3 storage - #2000
• Add optional version_group_id kwarg to create_flow_run for a stable API for flow runs - #1987
• Add extra_loggers logging configuration for non-Prefect logs in stdout and cloud - #2010

Task Library

• None

Fixes

• Ensure ifelse casts its condition to bool prior to evaluation - #1991
• Do not perform ast.literal_eval on cpu and memory task_definition kwargs for Fargate Agent - #2010
• Fix new agent processing with Threadpool causing problem for Fargate Agent with task revisions enabled - #2022

Deprecations

• None

Breaking Changes

• Remove Airflow Tasks - #1992

Contributors

• Giorgio Pellero
• Braun Reyes

Changelog

0.9.3

Released on Feb 05, 2020.

Features

• None

Enhancements

• Improve heartbeat functionality to be robust across platforms - #1973
• Run storage healthchecks on other options besides Docker - 1963
• Cloud logger now attempts to elevate logger errors to flow run logs - #1961
• Attach Flow and Task attributes to LogRecords - #1938

Task Library

• None

Fixes

• Fix uncaught Fargate Agent kwarg parse SyntaxError from literal_eval - #1968
• Fix FargateTaskEnvironment passing empty auth token to run task - #1976
• Fix imagePullSecrets not being automatically passed to jobs created by Kubernetes Agent - #1982

Deprecations

• None

Breaking Changes

• Remove cancellation hooks - #1973

Contributors

• None

Changelog

0.9.2

Released on Jan 30, 2020.

Features

• Allow for parameter defaults to vary based on clock - #1946

Enhancements

• More graceful handling of Agents competing for work - #1956

Task Library

• None

Fixes

• Eliminated possible duplicate flow run issue in all agents - #1956

Deprecations

• None

Breaking Changes

• None

Contributors

• None

Changelog

0.9.1

Released on Jan 24, 2020.

Features

• None

Enhancements

• Docker daemon reconnect attempts + exit on heartbeat failure -#1918
• More responsive agent shutdown - #1921
• Background all agent flow deployment attempts - #1928
• Add show_flow_logs to Docker agent #1929
• Add per-task checkpointing opt-out - #1933
• The Task 'checkpoint' kwarg will no longer be deprecated to allow opt-out - #1933

Task Library

• None

Fixes

• Fix the Fargate Agent not parsing kwargs as literals - #1926
• Fix issue with result handler default persisting from initialization - #1936

Deprecations

• None

Breaking Changes

• None

Contributors

• None

Changelog

For a high-level description of the new features, please check out our blog post on this release

0.9.0

Released on Jan 15, 2020.

Features

• Added the ability to leverage native ECS task definition revisions for flow versions in Fargate agent - #1870
• Added the ability to pull in kwargs per flow version from S3 on flow submission in Fargate agent - #1870
• Add sensible default result handlers to non-Docker storage options - #1888

Enhancements

• Allow for task looping to beyond Python's maximum recursion depth - #1862
• Prevent duplication of stdout logs from multiple instantiated agents - #1866
• Allow intervals less than 60 seconds in IntervalClocks - #1880
• Introduce new Secret.exists method for checking whether a Secret is available - #1882
• Introduce new -e CLI options on agent start commands to allow passing environment variables to flow runs - #1878
• Stop persisting None when calling result handlers - #1894
• Change Cancelled state to indicate Finished instead of Failed - #1903
• All States now store cached_inputs for easier recovery from failure - #1898
• Always checkpoint tasks which have result handlers - #1898

Task Library

• Remove implicit requirement that Google Tasks use Prefect Cloud Secrets - #1882

Fixes

• Enforce provision of max_retries if specifying retry_delay for a Task - #1875
• Fix issue with reduce tasks in flow.visualize() - #1793

Deprecations

• The checkpointing kwarg will be removed from Tasks as it is now a default behavior - #1898

Breaking Changes

• Remove default value for aws_credentials_secret on all S3 hooks - #1886
• Remove config.engine.result_handler section of Prefect config - #1888
• Remove default value for credentials_secret on GCSResultHandler - #1888
• Remove default value for azure_credentials_secret on AzureResultHandler - #1888

Contributors

• Daryll Strauss
• Braun Reyes

Changelog

0.8.1

Released on Dec 17, 2019

Features

• None

Enhancements

• Enhanced treatment of nested and ordered constant values - #1829
• Add on_datetime, on_date, and at_time filters - #1837
• Add --latest flag for Kubernetes Agent install CLI command - #1842
• Add --no-cloud-logs flag for all agents to optionally opt-out of logs being sent to Prefect Cloud - #1843
• Agents mark Flow Runs as Failed if a deployment error occurs - #1848
• Submitted states from Agents include deployment identifier information - #1848
• Update heartbeats to respect Cloud flow settings - #1851
• Add flow run name to prefect.context - #1855
• Add --namespace option for Kubernetes Agent start CLI command - #1859
• Add Prefect job resource configuration for Kubernetes Agent - #1859

Task Library

• None

Fixes

• Fix Agent deployment errors interrupting full list of found Flow Runs - #1848
• Fix issue with a single bad log preventing all logs from being sent to Cloud - #1845
• Fix Kubernetes Agent passing empty default namespace - #1839

Deprecations

• None

Breaking Changes

• None

Contributors

• None

Changelog

0.8.0

Released on Dec 11, 2019

Features

• Universal Deploy: Added new Local Agent to run Flows from Local Storage, Azure Storage, S3 Storage, and GCS Storage - #1819
• Added Azure Blob Storage for Flows - #1831
• Added GCS Storage for Flows - #1809
• Added S3 Storage for Flows - #1753

Enhancements

• Add convenience parents() and children() classmethods to all State objects for navigating the hierarchy - #1784
• Add --rbac flag to prefect agent install for Kubernetes Agent - #1822
• Add flow_run_name to the context - #1815
• Add prefect agent install option to output supervisord.conf file for Local Agent - #1819
• Add convenience parents() and children() classmethods to all State objects for navigating the hierarchy - #1784
• Add new not_all_skipped trigger and set it as the default for merge tasks - #1768

Task Library

• Azure Blob tasks now use newer BlockBlobService with connection string authentication - #1831

Fixes

• Fix issue with flow.visualize() for mapped tasks which are skipped - #1765
• Fix issue with timeouts only being softly enforced - #1145, #1686
• Log agent errors using write_run_logs instead of the deprecated write_run_log - #1791
• Fix issue with flow.update() not transferring constants - #1785

Deprecations

• flow.deploy is deprecated in favor of flow.register - #1819

Breaking Changes

• Default Flow storage is now Local instead of Docker - #1819
• Docker based LocalAgent is renamed DockerAgent - #1819
• prefect agent start now defaults to new LocalAgent - #1819

Changelog

Released on Nov 26, 2019

Features

• Add graceful cancellation hooks to Flow and Task runners - #1758

Enhancements

• Add option to specify a run name for cloud run CLI command - #1756
• Add work_stealing option to DaskKubernetesEnvironment - #1760
• Improve heartbeat thread management - #1770
• Add unique scheduler Job name to DaskKubernetesEnvironment - #1772
• Add informative error when trying to map with the LocalDaskExecutor using processes - #1777

Task Library

• None

Fixes

• Fix issue with heartbeat thread deadlocking dask execution when using a worker_client - #1750
• Fix issue with Environments not calling run_flow on Environment stored on Flow object - #1752
• Fix issue with Docker build context when providing custom docker files - #1762

Deprecations

• None

Breaking Changes

• None

Contributors

• None

0.7.2

Released on Nov 15, 2019.

Features

• Allow users to provide a custom version group ID for controlling Cloud versioning - #1665
• Stop autogenerating constant tasks - #1730

Enhancements

• Raise an informative error when context objects are pickled - #1710
• Add an option to pass in run_name to a flow run to override the auto-generated names when calling create_flow_run #1661
• Add informative logs in the event that a heartbeat thread dies - #1721
• Loosen Job spec requirements for KubernetesJobEnvironment - #1713
• Loosen containerDefinitions requirements for FargateTaskEnvironment - #1713
• Local Docker agent proactively fails flow runs if image cannot be pulled - #1395
• Add graceful keyboard interrupt shutdown for all agents - #1731
• agent start CLI command now allows for Agent kwargs - #1737
• Add users to specify a custom Dockerfile for Docker storage - #1738
• Expose labels kwarg in flow.deploy for convenient labeling of Flows - #1742

Task Library

• None

Fixes

• FargateTaskEnvironment now uses provided family for task definition naming - #1713
• Fix executor initialization missing self in KubernetesJobEnvironment - #1713
• Fix identifier_label not being generated on each run for Kubernetes based environments - #1718
• Fix issue where users could not override their user config path when deploying Docker to Cloud - #1719
• Respect order of inputs in merge - #1736

Deprecations

• None

Breaking Changes

• None

Contributors

• Brett Naul

0.7.1

Released on Nov 5, 2019

Features

• None

Enhancements

• Add a save/load interface to Flows - #1685, #1695
• Add option to specify aws_session_token for the FargateTaskEnvironment - #1688
• Add EnvVarSecrets for loading sensitive information from environment variables - #1683
• Add an informative version header to all Cloud client requests - #1690
• Auto-label Flow environments when using Local storage - #1696
• Batch upload logs to Cloud in a background thread for improved performance - #1691
• Include agent labels within each flow's configuration environment - #1671

Task Library

• None

Fixes

• Fix Fargate Agent access defaults and environment variable support - #1687
• Removed default python version for docker builds - #1705
• Attempt to install prefect in any docker image (if it is not already installed) - #1704
• Kubernetes Agent deployment yaml now respects new prefecthq/prefect image tagging convention - #1707

Deprecations

• None

Breaking Changes

• None

Contributors

• None

0.7.0

Released on October 29th, 2019.

Features

• Flow Affinity: Environments and Agents now support labeling for execution specification - #1651
• Add new Secret Tasks for a pluggable and reusable Secrets API - #1346, #1587

Enhancements

• Add the ability to delete task tag limits using the client - #1622
• Adds an "Ask for help" button with a link to the prefect.io support page - #1637
• Reduces the size of the prefecthq/prefect Docker image by ~400MB, which is now the base Docker image used in Flows - #1648
• Add a new healthcheck for environment dependencies - #1653
• Add default 30 second timeout to Client requests - #1672

Task Library

• Add new Secret Tasks for a pluggable and reusable Secrets API - #1346, #1587
• Add support for directly passing credentials to task library tasks, instead of passing secret names - #1667

Fixes

• Fix defaults for unspecified ARNs in the Fargate Agent - #1634
• Fix ShellTask return value on empty stdout - #1632
• Fix issue with some Cloud Secrets not being converted from strings - #1655
• Fix issue with Agent logging config setting not working - #1657
• Fix issue with SnowflakeQuery tasks not working - #1663

Deprecations

• Tasks that accepted the name of a secret (often credentials_secret) will raise a deprecation warning - #1667

Breaking Changes

• Fargate Agent now takes in all boto3 camel case arguments instead of specific snake case options - #1649
• kubernetes is no longer installed by default in deployed flow images - #1653
• Tasks that accepted the name of a secret (often credentials_secret) no longer have a default value for that argument, as it has been deprecated - #1667

Contributors

• Tobias Schmidt

Changelog

0.6.7

Released on Oct 16, 2019

Features

• Environments now allow for optional on_start and on_exit callbacks - #1610

Enhancements

• Raise more informative error when calling flow.visualize() if Graphviz executable not installed - #1602
• Allow authentication to Azure Blob Storage with SAS token - #1600
• Additional debug logs to Docker Container and Docker Image tasks - #920
• Changes to Fargate agent to support temporary credentials and IAM role based credentials within AWS compute such as a container or ec2 instance. #1607
• Local Secrets set through environment variable now retain their casing - #1601
• Agents can accept an optional name for logging and debugging - #1612
• Added AWS configuration options for Fargate Agent (task_role_arn, execution_role_arn) - #1614
• Change EmailTask to accept SMTP server settings as well as an email_from kwarg - #1619

Task Library

• Add return_all kwarg to ShellTask for optionally returning all lines of stdout - #1598
• Add CosmosDBCreateItem, CosmosDBReadItems, CosmosDBQueryItems and for interacting with data stored on Azure Cosmos DB - #1617

Fixes

• Fix issue with running local Flow without a schedule containing cached tasks - #1599
• Remove blank string for task_run_id in k8s resource manager - #1604
• Fix issue with merge task not working for pandas dataframes and numpy arrays - #1609

Deprecations

• None

Breaking Changes

• Local Secrets set through environment variable now retain their casing - #1601

Contributors

• Mark McDonald
• Sherman K

0.6.6

Released on October 3, 2019

Features

• Added KubernetesJobEnvironment - #1548
• Add ability to enforce Task concurrency limits by tag in Prefect Cloud - #1570
• Added FargateTaskEnvironment - #1592

Enhancements

• Allow the Client to more gracefully handle failed login attempts on initialization - #1535
• Replace DotDict with box.Box - #1518
• Store cached_inputs on Failed states and call their result handlers if they were provided - #1557
• raise_on_exception no longer raises for Prefect Signals, as these are typically intentional / for control flow - #1562
• run cloud CLI command takes in optional --parameters as a file path pointing to a JSON file - #1582
• Always consider Constant tasks successful and unpack them immediately instead of submitting them for execution - #1527

Task Library

• Add BlobStorageDownload and BlobStorageUpload for interacting with data stored on Azure Blob Storage - #1538
• Loosen Kubernetes Tasks' requirement of an API secret key - #1559
• Add tasks for working in Azure Machine Learning Serviec with Datastores and Datasets - #1590

Fixes

• Fix issue with certain Pause / Resume / Retry pipelines retrying indefinitely - #1177
• Kubernetes Agent deployment YAML generation defaults to local Prefect version - #1573
• Fix issue with custom result handlers not working when called in cached_inputs - #1585

Deprecations

• None

Breaking Changes

• None

Contributors

• Fredrik Sannholm

Changelog

Released on September 19th, 2019.

Features

• Added Fargate agent - #1521
• Custom user-written environments can be deployed to Prefect Cloud - #1534, #1537

Enhancements

• Allow for Agents to correctly run in environments with differently calibrated clocks - #1402
• Refactor RemoteEnvironment to utilize the get_flow storage interface - #1476
• Ensure Task logger is available in context throughout every pipeline step of the run - #1509
• Skip Docker registry pushing and pulling on empty registry_url attribute - #1525
• Agents now log platform errors to flow runs which cannot deploy - #1528
• Updating ShellTask to work more like Airflow Bash Operator for streaming logs and returning values - #1451
• Agents now have a verbose/debug logging option for granular output - #1532
• DaskKubernetesEnvironment now allows for custom scheduler and worker specs - #1543, #1537

Task Library

• None

Fixes

• Fix map error by removing imagePullSecrets from Kubernetes Agent install if not provided - #1524
• Fix issue with two INFO logs not being associated with the Task Run in Cloud - #1526
• execute CLI command can now load custom environments off of the flow object - #1534

Deprecations

• None

Breaking Changes

• Update ShellTask to return only the last line of stdout, as a string - #1451

Contributors

• braunreyes

Changelog

Released on September 10, 2019

Features

• Improve Windows compatibility for local development and deploying to Prefect Cloud - #1441, #1456, #1465, #1466

Enhancements

• Add OS platform check to Local Agent for running on Windows machines - #1441
• Add --base-url argument for Docker daemons to agent start CLI command - #1441
• Add environment labels for organizing / tagging different Flow execution environments - #1438
• Use -U option when installing prefect in Docker containers to override base image version - #1461
• Remove restriction that prevented DotDict classes from having keys that shadowed dict methods - #1462
• Added livenessProbe to Kubernetes Agent - #1474
• Ensure external Dask Clusters do not require Prefect Cloud environment variables to run Cloud flows - #1481

Task Library

• None

Fixes

• Fix incorrect import in DaskKubernetesEnvironment job template - #1458
• Raise error on Agents started without an appropriate API token - #1459
• Fix bug when calling as_nested_dict on DotDicts with an items key - #1462
• Fix --resource-manager flag on agent install invalidating imagePullSecrets - #1469
• Fix issue with user-written result handlers in Prefect Cloud preventing some states from being set - #1480

Deprecations

• None

Breaking Changes

• None

Contributors

• Joe Schmid
• Brett Naul

0.6.3

Released August 30, 2019

Fixes

• Fix issue with reduced mapped tasks not respecting retries - #1436

Changelog

0.6.2

Released August 30, 2019

Features

• Added Local, Kubernetes, and Nomad agents - #1341
• Add the ability for Tasks to sequentially loop - #1356

Enhancements

• Adds a copy to clipboard button for codeblocks - #213
• Updates Vuepress to v1.0.3 - #770
• Introduce configurable default for storage class on Flows - #1044
• Allow for min and max workers to be specified in DaskKubernetesEnvironment - #1338
• Use task and flow names for corresponding logger names for better organization - #1355
• Paused states subclass Scheduled and can have predefined expirations - #1375
• Introduce new Flow health checks prior to Cloud deployment - #1372
• Improve logging functionality to include tracebacks - #1374
• Improve CLI user experience while working with Cloud - #1384
• Users can now create projects from the CLI - #1388
• Add a health check to confirm that serialized flows are valid prior to Cloud deploy - #1397
• Add task_slug, flow_id, and flow_run_id to context - #1405
• Support persistent scheduled_start_time for scheduled flow runs when run locally with flow.run() - #1418, #1429
• Add task_args to Task.map - #1390
• Add auth flows for USER-scoped Cloud API tokens - #1423
• Add AzureResultHandler for handling results to / from Azure Blob storage containers - #1421
• Add new configurable LocalDaskExecutor - #1336
• Add CLI commands for working with Prefect Cloud auth - #1431

Task Library

• Add new SnowflakeQuery task for using snowflake data warehouse - #1113

Fixes

• Fix issue with Docker storage not respecting user-provided image names - #1335
• Fix issue with local retries in Cloud not always running in-process - #1348

Deprecations

• Rename SynchronousExecutor as LocalDaskExecutor - #1434

Breaking Changes

• Rename CloudEnvironment to DaskKubernetesEnvironment - #1250
• Remove unused queue method from all executors - #1434

Contributors

• Alex Kravetz

Changelog

0.6.1

Released August 8, 2019

Features

• Introduce new flows.checkpointing configuration setting for checkpointing Tasks in local execution - #1283
• Introduce new, flexible Schedule objects - #1320

Enhancements

• Allow passing of custom headers in Client calls - #1255
• Autogenerate informative names and tags for Docker images built for Flow storage - #1237
• Allow mixed-case configuration keys (environment variables are interpolated as lowercase) - #1288
• Ensure state handler errors are logged informatively - #1326

Task Library

• Add BigQueryLoadGoogleCloudStorage task for loading data into BigQuery from Google Cloud Storage #1317

Fixes

• Fix issue with logs not always arriving in long-standing Dask clusters - #1244
• Fix issue with BuildImage docker task not actually running to completion - #1243
• Fix run --logs CLI command not exiting on flow run finished state - #1319

Deprecations

• OneTimeSchedule and UnionSchedule are deprecated, but remain callable as convenience functions - #1320
• Old-style schedules can be deserialized as new-style schedules, but support will eventually be dropped - #1320

Breaking Changes

• prefect.Client.graphql() and prefect.Client.post() now use an explicit keyword, not **kwargs, for variables or parameters - #1259
• auth add CLI command replaced with auth login - #1319