prefect

Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines

APACHE-2.0 License

Downloads
8.4M
Stars
14.5K
Committers
148
View Code on GitHub Visit Website
Ecosystems: Python

Bot releases are hidden (Show)

prefect - Release 3.0.0rc2

Published by zangell44 4 months ago

We're excited to announce the release candidate of Prefect 3.0. It's the most flexible, powerful, fastest version of Prefect yet. Prefect 3.0 includes several exciting new features. Install it by running pip install prefect==3.0.0rc2 and check out the docs here.

Run tasks independently of flows

You can now run and serve tasks outside of flows and inside of other tasks.

from prefect import task

@task
def my_background_task(name: str):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    my_background_task.delay("ford")

Transactional semantics

Use rollback and commit hooks to facilitate idempotent python code.

from prefect import flow, task
from prefect.transactions import transaction
@task
def first_task():
    print('first')

@first_task.on_rollback
def roll(txn):
    print('rolling back')

@task
def second_task():
    raise RuntimeError("oopsie")

@flow
def txn_flow():
    with transaction():
        first_task()
        second_task()
if __name__ == "__main__":
    txn_flow()

Open source Events and Automations

Trigger actions, such as sending notifications, pausing schedules, starting flow runs and more in response to Prefect events.

More flexible variables and new artifact types

Variables can now be any JSON compatible type including dicts, lists, and integers. Progress and Image artifacts make it easy to add visual annotations to your flow run graph.

Faster and richer CLI

Improved CLI speed and several added commands and conveniences.

Updated navigation, styling, and interaction design

The new Runs page displays both flow and task run information, and an improved sidebar and switcher makes navigating Prefect simpler than ever.

Changes since 3.0.0rc1

Enhancements

Fixes

Documentation

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc1...3.0.0rc2

prefect - 2.19.4

Published by desertaxle 5 months ago

Fixes

Documentation

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.3...2.19.4

prefect - Release 3.0.0rc1

Published by zangell44 5 months ago

Release 3.0.0rc1

We're excited to announce the release candidate of Prefect 3.0. It's the most flexible, powerful, fastest version of Prefect yet. Prefect 3.0 includes several exciting new features. Install it by running pip install prefect==3.0.0rc1 and check out the docs here.

Run tasks independently of flows

You can now run and serve tasks outside of flows and inside of other tasks.

from prefect import task

@task
def my_background_task(name: str):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    my_background_task.delay("ford")

Transactional semantics

Use rollback and commit hooks to facilitate idempotent python code.

from prefect import flow, task
from prefect.transactions import transaction
@task
def first_task():
    print('first')

@first_task.on_rollback
def roll(txn):
    print('rolling back')

@task
def second_task():
    raise RuntimeError("oopsie")

@flow
def txn_flow():
    with transaction():
        first_task()
        second_task()
if __name__ == "__main__":
    txn_flow()

Open source Events and Automations

Trigger actions, such as sending notifications, pausing schedules, starting flow runs and more in response to Prefect events.

More flexible variables and new artifact types

Variables can now be any JSON compatible type including dicts, lists, and integers. Progress and Image artifacts make it easy to add visual annotations to your flow run graph.

Faster and richer CLI

Improved CLI speed and several added commands and conveniences.

Updated navigation, styling, and interaction design

The new Runs page displays both flow and task run information, and an improved sidebar and switcher makes navigating Prefect simpler than ever.

Enhancements

Fixes

Experimental

Documentation

Integrations

Breaking Changes

Contributors

  • @bsignoret

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.0...3.0.0rc1

See release notes for details.

prefect - Release 2.19.3

Published by abrookins 5 months ago

New method for generating parameter schemas without dependencies

prefect deploy now works even when dependencies are missing from the current environment. This can speed up deployment via CI by removing the need to install dependencies before deploying your flows.

See the following pull requests for implementation details:

Enhancements

Fixes

Integrations

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.2...2.19.3

prefect - Release 2.19.2

Published by serinamarie 5 months ago

This release includes a few bug fixes, ensuring:

prefect - 2.19.1

Published by desertaxle 5 months ago

Release 2.19.1

Enhancements to prefect-dbt for running dbt-core commands

Introducing prefect-dbt summary artifacts! With summary artifacts, you get a view of all of the runs that succeeded, failed, or were skipped as well as where the failed models live in your dbt project and why they failed. This exposes information about each dbt node in a succinct format that teams can use for development, debugging, insights, and so much more.

These changes are available in prefect-dbt version 0.5.0!

See the following pull requests for implementation details:

Enhancements

Experimental

Events and Automations

Documentation

Integrations

Integration library releases

This release was accompanied by the following integration library releases:

  • prefect-aws - 0.4.17
  • prefect-azure - 0.3.10
  • prefect-bitbucket - 0.2.5
  • prefect-dask - 0.2.9
  • prefect-databricks - 0.2.7
  • prefect-dbt - 0.5.0
  • prefect-docker - 0.5.2
  • prefect-email - 0.3.5
  • prefect-gcp - 0.5.12
  • prefect-github - 0.2.5
  • prefect-gitlab - 0.2.5
  • prefect-kubernetes - 0.3.10
  • prefect-ray - 0.3.6
  • prefect-shell - 0.2.5
  • prefect-slack - 0.2.6
  • prefect-snowflake - 0.27.6
  • prefect-sqlalchemy - 0.4.3

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.0...2.19.1

prefect - Release 2.19.0

Published by serinamarie 5 months ago

✨ This release includes a number of enhancements and fixes!

Support for major infrastructure and distributed task integrations

As prefect-dask and other integrations have been added to the prefect codebase, this release adds these integrations as extra requirements of the prefect package, making it easier to install support for everything in your Prefect stack:

pip install 'prefect[dask]'

We loved this community contribution so much, we did it for all our first-party integrations:

pip install 'prefect[aws,kubernetes,dask,dbt,sqlalchemy,slack]'

You can see the full list of Prefect's extra requirements in our setup.py.

Support for timeout seconds in global concurrency context manager

You may want to fail immediately if a global concurrency slot is unavailable. Rather than block and wait, you can now specify a timeout_seconds argument in the global concurrency context manager and catch a TimeoutError if a slot is not available within the specified time.

@flow
def fail_immediately_flow():
    try:
        with concurrency("there-can-be-only-one", occupy=1, timeout_seconds=0.1):
            do_something_resource_intensive()
    except TimeoutError:
        return Cancelled(message="Another flow run is already running")

Manage global concurrency limits via the CLI

Global concurrency limits let you control how many operations can run simultaneously-- now you can create, read, edit, and delete global concurrency limits via the Prefect CLI!

To create a new concurrency limit, use the prefect gcl create command. You must specify a --limit argument, and can optionally specify a --slot-decay-per-second and --disable argument.

prefect gcl create my-concurrency-limit --limit 5 --slot-decay-per-second 1.0

You can inspect the details of a concurrency limit using the prefect gcl inspect command:

prefect gcl inspect my-concurrency-limit

To update a concurrency limit, use the prefect gcl update command. You can update the --limit, --slot-decay-per-second, --enable, and --disable arguments:

prefect gcl update my-concurrency-limit --limit 10

We also have many more bug fixes and in-flight work! See the release notes for details!

prefect - Release 2.18.3

Published by pleek91 6 months ago

Experimental

Engine

Fixes

All changes: https://github.com/PrefectHQ/prefect/compare/2.18.2...2.18.3

prefect - Release 2.18.2

Published by serinamarie 6 months ago

💡 Providing a deployment name to flow.serve is now optional

When running flow.serve, you can now omit the deployment name. If you do not provide a deployment name, the deployment name will default to the name of the flow. This change makes it easier to run flows without needing to specify a deployment name each time:

@flow
def etl_flow():
    pass

if __name__ == "__main__":
  etl_flow.serve()

results in:

Your flow 'etl-flow' is being served and polling for scheduled runs!

To trigger a run for this flow, use the following command:

        $ prefect deployment run 'etl-flow/etl-flow'

🛠✨ We've also released a few important fixes to our deployment parameter form when creating a run in the Prefect UI! 🧑‍🎨

🚀 This release also includes a number of other fixes and in-flight feature work. See the release notes for more details!

prefect - Release 2.18.1

Published by bunchesofdonald 6 months ago

Release 2.18.1

Fixes

Experimental

Events and Automations

Engine

Documentation

Prefect UI Library

Integrations

Prefect CGP

New Contributors

prefect - Release 2.18.0

Published by abrookins 6 months ago

Breaking Changes

Enhancements

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.17.0...2.18.0

See the release notes for more!

prefect - Release 2.17.1

Published by desertaxle 6 months ago

Fixes

All changes: https://github.com/PrefectHQ/prefect/compare/2.17.0...2.17.1

prefect - Release 2.17.0

Published by desertaxle 6 months ago

🧮 Manage Prefect variables via the Python SDK

Prefect variables are useful for storing and reusing data and configuration between and across workflows; and previously you could only create and update variables via the Prefect UI. With this release, you can now get and set Prefect variables directly in your Python code with the new Variable.set and Variable.get methods!

For an example of reading and writing variable values in Python see the following example:

from prefect.variables import Variable

# set a variable
variable = Variable.set(name="the_answer", value="42")

# get a variable
answer = Variable.get('the_answer')
print(answer.value)
# 42

# get a variable with a default value
answer = Variable.get('not_the_answer', default='42')
print(answer.value)
# 42

# update a variable
answer = Variable.set(name="the_answer", value="43", overwrite=True)
print(answer.value)
#43

Refer to the docs for more information and see the PR for implementation details: https://github.com/PrefectHQ/prefect/pull/12596

Other Enhancements 🌟

... and numerous 🐛 fixes!

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.9...2.17.0

See the release notes for more!

prefect - Release 2.16.9

Published by serinamarie 7 months ago

This release includes a number of enhancements and in-flight feature work.

🛠✨ One such enhancement helps streamline our CLI by adding a -jv/--job-variable option to prefect deploy, on par with the option available in prefect deployment run.

🔄🔗 In terms of enhancing existing Prefect concepts, we've removed a constraint that prevented tasks from being called from other tasks. For example, this allows you to call tasks within tasks within a flow.

🗿 📉 We no longer create artifacts for unpersisted results, which should prevent an influx of entries to the artifact table. Retried flows without persisted results will now have an error message stating that the "State data is missing" rather than referencing an "unpersisted result".

See the release notes for more details!

prefect - Release 2.16.8

Published by bunchesofdonald 7 months ago

Fixes

prefect - Release 2.16.7

Published by zzstoatzz 7 months ago

💻 Introducing prefect shell for observing CLI commands

You can now observe CLI commands as a Prefect flow. For example:

» prefect shell watch "curl http://wttr.in/Chicago?format=3"
17:32:39.562 | INFO | prefect.engine - Created flow run 'powerful-mushroom' for flow 'Shell Command'
17:32:40.171 | INFO | Flow run 'powerful-mushroom' - Chicago: ⛅️ +50°F
17:32:40.315 | INFO | Flow run 'powerful-mushroom' - Finished in state Completed()

See these docs to learn how to:

  • run a shell command as a Prefect flow on-demand with watch
  • schedule a shell command as a recurring Prefect flow using serve

See the PR for implementation details: https://github.com/PrefectHQ/prefect/pull/11998

Other Enhancements 🌟

... and numerous 🐛 fixes!

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.6...2.16.7

See the release notes for more!

prefect - Release 2.16.6

Published by zzstoatzz 7 months ago

Fix new behavior in typer 0.10.0 that broke the prefect CLI

See the following pull request for implementation details:

All changes: https://github.com/PrefectHQ/prefect/compare/2.16.5...2.16.6

prefect - Release 2.16.5

Published by abrookins 7 months ago

Multi-select deletion of flow runs

It is now easier to bulk select and delete flow runs through the UI. Listings of filterable and selectable flow runs (e.g. on the flow runs, flow, and deployment pages) now include a top-level checkbox for (de)selecting all currently filtered flow runs for bulk deletion.

image

See the following pull request for implementation details:

Visualize state changes and artifacts in the UI

Additionally, the flow run graph UI enhancements for visualizing state changes and artifacts added in 2.16.4 are now enabled by default. See the release notes in 2.16.4 for more details!

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.4...2.16.5

See the release notes for more!

prefect - Release 2.16.4

Published by zhen0 7 months ago

Release 2.16.4

Flow Run Graph updates

The Flow Run Graph has been updated to display additional layers of information! Interactive and real-time state changes and artifacts are now visible in context on the graph.

Agents

A year ago, we released workers as a replacement for agents. Workers significantly enhance the experience of deploying flows and simplify the specification of each flow's infrastructure and runtime environment.

With this release we are adding a six month (September 14) deprecation warning to agents and related concepts. Please note that:

  • Deprecation will not impact or break any work running with agents and agent-related concepts - although we encourage users to upgrade to workers because they provide a better deployment experience, you can continue to use existing agents and related concepts after deprecation
  • After September 14, Prefect Cloud users will not be able to create new agent work pools or infrastructure blocks
  • After September 14, new minor versions of the Prefect Python package will not include agents

Like agents, workers support creating deployments through the Prefect CLI and through Python, but require different syntax. For more information, please refer to the Upgrade from Agents to Workers guide.

Enhancements

Fixes

Experimental

Deprecations

Typing

Documentation

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.3...2.16.4

prefect - Release 2.16.3

Published by serinamarie 8 months ago

This release includes enhanced deployment parameters in the Prefect UI. We’ve refined the deployment parameters UI to significantly enhance default value handling and list management. 🛠️ This release introduces support for tuple-type parameters and a revamped list UI, freeing users from the constraints of JSON for list inputs. 🔄 You now have the flexibility to utilize JSON or Prefect variables for any parameter value, enhancing the versatility of deployment configurations. 💪 Moreover, we’ve synchronized validation across the UI and deployment schemas, ensuring that user inputs are consistently checked against the defined parameter requirements, which simplifies the deployment process and minimizes configuration errors. ✅ These improvements are aimed at providing a more efficient and user-friendly interface for managing deployment parameters. 🌟

This release also includes many other enhancements and fixes, including a new on_running state change hook 🎣, two memory leak fixes 🔧, and much more.

See the release notes for details! 📜🚀

Package Rankings
Top 0.45% on Pypi.org
Top 4.06% on Proxy.golang.org
Top 4.91% on Conda-forge.org
Related Projects
echodataflow

Orchestrated sonar data processing workflow

21 Jan 2022 4
prefect-tutorial

Python Prefect tutorial

01 Jan 2022 9
fastapi_cloudrun_pubsub

Python API, queues and workers on GCP: FastAPI + CloudRun + PubSub

22 Aug 2023 3
devops-resources

DevOps resources - Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes...

07 Jan 2019 8,538
runhouse

The fastest way to iterate and deploy AI workloads on your own infra. Unobtrusive, debuggable, Py...

10 May 2022 707