prefect

Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines

APACHE-2.0 License

Downloads
8.4M
Stars
14.5K
Committers
148
View Code on GitHub Visit Website
Ecosystems: Python

Bot releases are visible (Hide)

prefect - Release 2.18.3

Published by pleek91 6 months ago

Experimental

Engine

Fixes

All changes: https://github.com/PrefectHQ/prefect/compare/2.18.2...2.18.3

prefect - Release 2.18.2

Published by serinamarie 6 months ago

๐Ÿ’ก Providing a deployment name to flow.serve is now optional

When running flow.serve, you can now omit the deployment name. If you do not provide a deployment name, the deployment name will default to the name of the flow. This change makes it easier to run flows without needing to specify a deployment name each time:

@flow
def etl_flow():
    pass

if __name__ == "__main__":
  etl_flow.serve()

results in:

Your flow 'etl-flow' is being served and polling for scheduled runs!

To trigger a run for this flow, use the following command:

        $ prefect deployment run 'etl-flow/etl-flow'

๐Ÿ› โœจ We've also released a few important fixes to our deployment parameter form when creating a run in the Prefect UI! ๐Ÿง‘โ€๐ŸŽจ

๐Ÿš€ This release also includes a number of other fixes and in-flight feature work. See the release notes for more details!

prefect - Release 2.18.1

Published by bunchesofdonald 6 months ago

Release 2.18.1

Fixes

Experimental

Events and Automations

Engine

Documentation

Prefect UI Library

Integrations

Prefect CGP

New Contributors

prefect - Release 2.18.0

Published by abrookins 6 months ago

Breaking Changes

Enhancements

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.17.0...2.18.0

See the release notes for more!

prefect - Release 2.17.1

Published by desertaxle 6 months ago

Fixes

All changes: https://github.com/PrefectHQ/prefect/compare/2.17.0...2.17.1

prefect - Release 2.17.0

Published by desertaxle 6 months ago

๐Ÿงฎ Manage Prefect variables via the Python SDK

Prefect variables are useful for storing and reusing data and configuration between and across workflows; and previously you could only create and update variables via the Prefect UI. With this release, you can now get and set Prefect variables directly in your Python code with the new Variable.set and Variable.get methods!

For an example of reading and writing variable values in Python see the following example:

from prefect.variables import Variable

# set a variable
variable = Variable.set(name="the_answer", value="42")

# get a variable
answer = Variable.get('the_answer')
print(answer.value)
# 42

# get a variable with a default value
answer = Variable.get('not_the_answer', default='42')
print(answer.value)
# 42

# update a variable
answer = Variable.set(name="the_answer", value="43", overwrite=True)
print(answer.value)
#43

Refer to the docs for more information and see the PR for implementation details: https://github.com/PrefectHQ/prefect/pull/12596

Other Enhancements ๐ŸŒŸ

... and numerous ๐Ÿ› fixes!

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.9...2.17.0

See the release notes for more!

prefect - Release 2.16.9

Published by serinamarie 7 months ago

This release includes a number of enhancements and in-flight feature work.

๐Ÿ› โœจ One such enhancement helps streamline our CLI by adding a -jv/--job-variable option to prefect deploy, on par with the option available in prefect deployment run.

๐Ÿ”„๐Ÿ”— In terms of enhancing existing Prefect concepts, we've removed a constraint that prevented tasks from being called from other tasks. For example, this allows you to call tasks within tasks within a flow.

๐Ÿ—ฟ ๐Ÿ“‰ We no longer create artifacts for unpersisted results, which should prevent an influx of entries to the artifact table. Retried flows without persisted results will now have an error message stating that the "State data is missing" rather than referencing an "unpersisted result".

See the release notes for more details!

prefect - Release 2.16.8

Published by bunchesofdonald 7 months ago

Fixes

prefect - Release 2.16.7

Published by zzstoatzz 7 months ago

๐Ÿ’ป Introducing prefect shell for observing CLI commands

You can now observe CLI commands as a Prefect flow. For example:

ยป prefect shell watch "curl http://wttr.in/Chicago?format=3"
17:32:39.562 | INFO | prefect.engine - Created flow run 'powerful-mushroom' for flow 'Shell Command'
17:32:40.171 | INFO | Flow run 'powerful-mushroom' - Chicago: โ›…๏ธ +50ยฐF
17:32:40.315 | INFO | Flow run 'powerful-mushroom' - Finished in state Completed()

See these docs to learn how to:

  • run a shell command as a Prefect flow on-demand with watch
  • schedule a shell command as a recurring Prefect flow using serve

See the PR for implementation details: https://github.com/PrefectHQ/prefect/pull/11998

Other Enhancements ๐ŸŒŸ

... and numerous ๐Ÿ› fixes!

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.6...2.16.7

See the release notes for more!

prefect - Release 2.16.6

Published by zzstoatzz 7 months ago

Fix new behavior in typer 0.10.0 that broke the prefect CLI

See the following pull request for implementation details:

All changes: https://github.com/PrefectHQ/prefect/compare/2.16.5...2.16.6

prefect - Release 2.16.5

Published by abrookins 7 months ago

Multi-select deletion of flow runs

It is now easier to bulk select and delete flow runs through the UI. Listings of filterable and selectable flow runs (e.g. on the flow runs, flow, and deployment pages) now include a top-level checkbox for (de)selecting all currently filtered flow runs for bulk deletion.

image

See the following pull request for implementation details:

Visualize state changes and artifacts in the UI

Additionally, the flow run graph UI enhancements for visualizing state changes and artifacts added in 2.16.4 are now enabled by default. See the release notes in 2.16.4 for more details!

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.4...2.16.5

See the release notes for more!

prefect - Release 2.16.4

Published by zhen0 7 months ago

Release 2.16.4

Flow Run Graph updates

The Flow Run Graph has been updated to display additional layers of information! Interactive and real-time state changes and artifacts are now visible in context on the graph.

Agents

A year ago, we released workers as a replacement for agents. Workers significantly enhance the experience of deploying flows and simplify the specification of each flow's infrastructure and runtime environment.

With this release we are adding a six month (September 14) deprecation warning to agents and related concepts. Please note that:

  • Deprecation will not impact or break any work running with agents and agent-related concepts - although we encourage users to upgrade to workers because they provide a better deployment experience, you can continue to use existing agents and related concepts after deprecation
  • After September 14, Prefect Cloud users will not be able to create new agent work pools or infrastructure blocks
  • After September 14, new minor versions of the Prefect Python package will not include agents

Like agents, workers support creating deployments through the Prefect CLI and through Python, but require different syntax. For more information, please refer to the Upgrade from Agents to Workers guide.

Enhancements

Fixes

Experimental

Deprecations

Typing

Documentation

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.3...2.16.4

prefect - Release 2.16.3

Published by serinamarie 8 months ago

This release includes enhanced deployment parameters in the Prefect UI. Weโ€™ve refined the deployment parameters UI to significantly enhance default value handling and list management. ๐Ÿ› ๏ธ This release introduces support for tuple-type parameters and a revamped list UI, freeing users from the constraints of JSON for list inputs. ๐Ÿ”„ You now have the flexibility to utilize JSON or Prefect variables for any parameter value, enhancing the versatility of deployment configurations. ๐Ÿ’ช Moreover, weโ€™ve synchronized validation across the UI and deployment schemas, ensuring that user inputs are consistently checked against the defined parameter requirements, which simplifies the deployment process and minimizes configuration errors. โœ… These improvements are aimed at providing a more efficient and user-friendly interface for managing deployment parameters. ๐ŸŒŸ

This release also includes many other enhancements and fixes, including a new on_running state change hook ๐ŸŽฃ, two memory leak fixes ๐Ÿ”ง, and much more.

See the release notes for details! ๐Ÿ“œ๐Ÿš€

prefect - Release 2.16.2

Published by zzstoatzz 8 months ago

โ›‘๏ธ Fixing a logging bug introduced by 2.16.1

In 2.16.1, a regression related to logging non-str objects (when a PREFECT_API_KEY was set) was introduced. The fix is included in 2.16.2.

See https://github.com/PrefectHQ/prefect/pull/12151 for implementation details.

๐ŸŽฏ Use a module path entrypoint when using .serve or .deploy 

from prefect_dbt.cloud.jobs import trigger_dbt_cloud_job_run_and_wait_for_completion

from prefect.deployments import EntrypointType

if __name__ == "__main__":
    trigger_dbt_cloud_job_run_and_wait_for_completion.deploy(
        name="demo-deploy-from-module",
        work_pool_name="olympic",
        entrypoint_type=EntrypointType.MODULE_PATH,
        job_variables={"env": { "EXTRA_PIP_PACKAGES": "prefect-dbt" } }
    )

See https://github.com/PrefectHQ/prefect/pull/12134 for implementation details.

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.1...2.16.2

See the release notes for more!

prefect - Release 2.16.1

Published by zzstoatzz 8 months ago

Enhanced multiple schedule support ๐Ÿ“…

prefect.yaml now supports specifying multiple schedules via the schedules key.

This allows you to define multiple schedules for a single deployment, and each schedule can have its own cron, interval, or rrule configuration:

 ...
 schedules:
    - cron: "0 0 * * *"
      active: false
    - interval: 3600
      active: true
    - rrule: "FREQ=YEARLY"
      active: true

In addition you can specify multiple schedules via arguments to prefect deploy:

prefect deploy ... --cron '4 * * * *' --cron '1 * * * *' --rrule 'FREQ=DAILY'

We've also added support for multiple schedules to flow.serve, flow.deploy and prefect.runner.serve. You can provide multiple schedules by passing a list to the cron, interval, or rrule arguments:

import datetime
import random

from prefect import flow


@flow
def trees():
    tree = random.choice(["๐ŸŒณ", "๐ŸŒด", "๐ŸŒฒ", "๐ŸŒต"])
    print(f"Here's a happy little tree: {tree}")

if __name__ == "__main__":
    trees.serve(
        name="trees",
        interval=[3600, 7200, 14400],
    )

This will create a deployment with three schedules, one that runs every hour, one that runs every two hours, and one that runs every four hours. For more advanced cases, use the schedules argument.

    trees.serve(
        name="trees",
        schedules=[
            IntervalSchedule(interval=datetime.timedelta(minutes=30)),
            {"schedule": RRuleSchedule(rrule="FREQ=YEARLY"), "active": True},
            MinimalDeploymentSchedule(schedule=CronSchedule(cron="0 0 * * *"), active=False),
        ]
    )

Dive into these new scheduling capabilities today and streamline your workflows like never before.

For implementation details, see the following pull request:
- https://github.com/PrefectHQ/prefect/pull/12107

New Contributors ๐ŸŒŸ

  • Shoutout to @jrbourbeau for their first contribution!

See the release notes for more details!

prefect - Release 2.16.0

Published by abrookins 8 months ago

๐Ÿ•ข Deployments now support multiple schedules ๐Ÿ•

With todayโ€™s release, weโ€™re excited to roll out initial support for using multiple schedules with Deployments! You can now use multiple schedules in the following ways:

  • Specifying schedules in a Deployment YAML file
  • Creating Python-based Deployments with the Deployment class
  • New CLI commands: prefect deployment schedule <create, delete, pause, resume, ls, clear>
  • New UI components aware of multiple schedules

Coming soon, weโ€™ll round out support for multiple schedules in other areas, such as:

  • When running a flow with flow.serve() and flow.deploy()
  • When using prefect deploy

The easiest way to get started with multiple schedules is to try out the new CLI commands:

$ prefect deployment schedule ls happy-flow/my-deployment
                       Deployment Schedules
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ ID                                   โ”ƒ Schedule        โ”ƒ Active โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ c7d3ddc4-9a5a-4dec-bd59-eed282ae55d5 โ”‚ cron: 0 0 1 * 1 โ”‚ True   โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

$ prefect deployment schedule create happy-flow/my-deployment --interval 60
Created deployment schedule!

$ prefect deployment schedule ls happy-flow/my-deployment
                         Deployment Schedules
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ ID                                   โ”ƒ Schedule           โ”ƒ Active โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ 3638ed58-cab2-4462-a680-2f92fcf6c797 โ”‚ interval: 0:01:00s โ”‚ True   โ”‚
โ”‚ c7d3ddc4-9a5a-4dec-bd59-eed282ae55d5 โ”‚ cron: 0 0 1 * 1    โ”‚ True   โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Enhancements

Fixes

Documentation

Experimental

Contributors

  • @NodeJSmith

Full release notes: https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md

prefect - 2.15.0

Published by desertaxle 8 months ago

๐Ÿ”ง Task runs now execute on the main thread

We are excited to announce that task runs are now executed on the main thread!

When feasible, task runs are now executed on the main thread instead of a worker thread. Previously, all task runs were run in a new worker thread. This allows objects to be passed to and from tasks without worrying about thread safety unless you have opted into concurrency. For example, an HTTP client or database connection can be shared between a flow and its tasks now (unless synchronous concurrency is used). Some asynchronous and sequential use cases may see performance improvements.

Consider the following example:

import sqlite3
from prefect import flow, task

db = sqlite3.connect("threads.db")

try:
    db.execute("CREATE TABLE fellowship(name)")
except sqlite3.OperationalError:
    pass
else:
    db.commit()

db.execute("DELETE FROM fellowship")
db.commit()

cur = db.cursor()


@task
def my_task(name: str):
    global db, cur

    cur.execute('INSERT INTO fellowship VALUES (?)', (name,))

    db.commit()


@flow
def my_flow():
    global db, cur

    for name in ["Frodo", "Gandalf", "Gimli", "Aragorn", "Legolas", "Boromir", "Samwise", "Pippin", "Merry"]:
        my_task(name)

    print(cur.execute("SELECT * FROM fellowship").fetchall())

    db.close()


if __name__ == "__main__":
    my_flow()

In previous versions of Prefect, running this example would result in an error like this:

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 7977619456 and this is thread id 6243151872.

But now, with task runs executing on the main thread, this example will run without error! We're excited this change makes Prefect even more intuitive and flexible!

See the following pull request for implementation details:
- https://github.com/PrefectHQ/prefect/pull/11930

๐Ÿ”ญ Monitor deployment runs triggered via the CLI

You can monitor the status of a flow run created from a deployment via the CLI. This is useful for observing a flow run's progress without navigating to the UI.

To monitor a flow run started from a deployment, use the --watch option with prefect deployment run:

prefect deployment run --watch <slugified-flow-name>/<slugified-deployment-name>

See the following pull request for implementation details:
- https://github.com/PrefectHQ/prefect/pull/11702

Enhancements

Fixes

Experimental

Documentation

Contributors

  • @zanieb

All changes: https://github.com/PrefectHQ/prefect/compare/2.14.21...2.15.0

prefect - 2.14.21

Published by zangell44 8 months ago

Release 2.14.21

Introducing work queue status

We're excited to unveil the new status indicators for work queues in Prefect's UI, enhancing your ability to oversee and control flow run execution within our hybrid work pools.

Work queues will now display one of three distinct statuses:

  • Ready - one or more online workers are actively polling the work queue
  • Not Ready - no online workers are polling the work queue, signaling a need for intervention
  • Paused - the work queue is intentionally paused, preventing execution

With the introduction of work queue status, you'll notice the absence of deprecated work queue health indicators in the UI.

See the documentation on work queue status for more information.

For now, this is an experimental feature, and can be enabled by running:

prefect config set PREFECT_EXPERIMENTAL_ENABLE_WORK_QUEUE_STATUS=True

See the following pull request for implementation details:
- https://github.com/PrefectHQ/prefect/pull/11829

Fixes

Documentation

Experimental

All changes: https://github.com/PrefectHQ/prefect/compare/2.14.20...2.14.21

prefect - Release 2.14.20

Published by collincchoy 9 months ago

Hotfix

All changes: https://github.com/PrefectHQ/prefect/compare/2.14.19...2.14.20

prefect - 2.14.19

Published by bunchesofdonald 9 months ago

Dynamic descriptions for paused and suspended flow runs

You can now include dynamic, markdown-formatted descriptions when pausing or suspending a flow run for human input. This description will be shown in the Prefect UI alongside the form when a user is resuming the flow run, enabling developers to give context and instructions to users when they need to provide input.

from datetime import datetime
from prefect import flow, pause_flow_run, get_run_logger
from prefect.input import RunInput

class UserInput(RunInput):
    name: str
    age: int

@flow
async def greet_user():
    logger = get_run_logger()
    current_date = datetime.now().strftime("%B %d, %Y")

    description_md = f"""
**Welcome to the User Greeting Flow!**
Today's Date: {current_date}

Please enter your details below:
- **Name**: What should we call you?
- **Age**: Just a number, nothing more.
"""

    user_input = await pause_flow_run(
        wait_for_input=UserInput.with_initial_data(
            description=description_md, name="anonymous"
        )
    )

    if user_input.name == "anonymous":
        logger.info("Hello, stranger!")
    else:
        logger.info(f"Hello, {user_input.name}!")

See the following PR for implementation details:

Enhancements

Fixes

Documentation

Experimental

New Contributors

Package Rankings
Top 0.45% on Pypi.org
Top 4.06% on Proxy.golang.org
Top 4.91% on Conda-forge.org
Related Projects
prefect-tutorial

Python Prefect tutorial

01 Jan 2022 9
runhouse

The fastest way to iterate and deploy AI workloads on your own infra. Unobtrusive, debuggable, Py...

10 May 2022 707
fastapi_cloudrun_pubsub

Python API, queues and workers on GCP: FastAPI + CloudRun + PubSub

22 Aug 2023 3
devops-resources

DevOps resources - Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes...

07 Jan 2019 8,538
echodataflow

Orchestrated sonar data processing workflow

21 Jan 2022 4