prefect

Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines

APACHE-2.0 License

Downloads
8.4M
Stars
14.5K
Committers
148
View Code on GitHub Visit Website
Ecosystems: Python

Bot releases are visible (Hide)

prefect - Weekly Release Candidate 3.0.0rc11

Published by marvin-robot 3 months ago

What's Changed

Exciting New Features 🎉

Enhancements

Fixes

Documentation

Integrations

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc10...3.0.0rc11

prefect - Release 2.19.8

Published by bunchesofdonald 3 months ago

Release 2.19.8

Enhancements

Fixes

Integrations

New Contributors

prefect - Release 3.0.0rc10

Published by serinamarie 4 months ago

What's Changed

Enhancements

Fixes

Documentation

UI

All changes: https://github.com/PrefectHQ/prefect/compare/3.0.0rc9...3.0.0rc10

prefect - Release 2.19.7

Published by desertaxle 4 months ago

Fixes

Documentation

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.6...2.19.7

prefect - Release 3.0.0rc9

Published by desertaxle 4 months ago

What's Changed

Fixes

Documentation

Uncategorized

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc8...3.0.0rc9

prefect - Release 2.19.6

Published by desertaxle 4 months ago

Enhancements

Fixes

Documentation

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.5...2.19.6

prefect - Nightly Release Candidate 3.0.0rc8

Published by marvin-robot 4 months ago

What's Changed

Enhancements

Fixes

Documentation

Uncategorized

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc7...3.0.0rc8

prefect - Nightly Release Candidate 3.0.0rc7

Published by marvin-robot 4 months ago

What's Changed

Fixes

Documentation

Uncategorized

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc6...3.0.0rc7

prefect - Nightly Release Candidate 3.0.0rc6

Published by marvin-robot 4 months ago

What's Changed

Fixes

Documentation

Uncategorized

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc5...3.0.0rc6

prefect - Release 3.0.0rc5

Published by desertaxle 4 months ago

What's Changed

Enhancements

Fixes

Documentation

Refactor

Breaking

Integrations

prefect-docker

prefect-kubernetes

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc4...3.0.0rc5

prefect - Release 3.0.0rc4

Published by desertaxle 4 months ago

What's Changed

Enhancements

Fixes

Documentation

Breaking

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc3...3.0.0rc4

prefect - Release 3.0.0rc3

Published by desertaxle 4 months ago

Exciting New Features 🎉

Enhancements

Fixes

Documentation

Breaking Changes

Integrations

prefect-dbt

prefect-databricks

Contributors

  • @LarryUllman
  • @Waiwait
  • @padbk
  • @stellardave

All changes: https://github.com/PrefectHQ/prefect/compare/3.0.0rc2...3.0.0rc3

prefect - Release 2.19.5

Published by jakekaplan 4 months ago

Release 2.19.5

Fixes

Integrations

Documentation

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.4...2.19.5

prefect - Release 3.0.0rc2

Published by zangell44 4 months ago

We're excited to announce the release candidate of Prefect 3.0. It's the most flexible, powerful, fastest version of Prefect yet. Prefect 3.0 includes several exciting new features. Install it by running pip install prefect==3.0.0rc2 and check out the docs here.

Run tasks independently of flows

You can now run and serve tasks outside of flows and inside of other tasks.

from prefect import task

@task
def my_background_task(name: str):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    my_background_task.delay("ford")

Transactional semantics

Use rollback and commit hooks to facilitate idempotent python code.

from prefect import flow, task
from prefect.transactions import transaction
@task
def first_task():
    print('first')

@first_task.on_rollback
def roll(txn):
    print('rolling back')

@task
def second_task():
    raise RuntimeError("oopsie")

@flow
def txn_flow():
    with transaction():
        first_task()
        second_task()
if __name__ == "__main__":
    txn_flow()

Open source Events and Automations

Trigger actions, such as sending notifications, pausing schedules, starting flow runs and more in response to Prefect events.

More flexible variables and new artifact types

Variables can now be any JSON compatible type including dicts, lists, and integers. Progress and Image artifacts make it easy to add visual annotations to your flow run graph.

Faster and richer CLI

Improved CLI speed and several added commands and conveniences.

Updated navigation, styling, and interaction design

The new Runs page displays both flow and task run information, and an improved sidebar and switcher makes navigating Prefect simpler than ever.

Changes since 3.0.0rc1

Enhancements

Fixes

Documentation

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc1...3.0.0rc2

prefect - 2.19.4

Published by desertaxle 5 months ago

Fixes

Documentation

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.3...2.19.4

prefect - Release 3.0.0rc1

Published by zangell44 5 months ago

Release 3.0.0rc1

We're excited to announce the release candidate of Prefect 3.0. It's the most flexible, powerful, fastest version of Prefect yet. Prefect 3.0 includes several exciting new features. Install it by running pip install prefect==3.0.0rc1 and check out the docs here.

Run tasks independently of flows

You can now run and serve tasks outside of flows and inside of other tasks.

from prefect import task

@task
def my_background_task(name: str):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    my_background_task.delay("ford")

Transactional semantics

Use rollback and commit hooks to facilitate idempotent python code.

from prefect import flow, task
from prefect.transactions import transaction
@task
def first_task():
    print('first')

@first_task.on_rollback
def roll(txn):
    print('rolling back')

@task
def second_task():
    raise RuntimeError("oopsie")

@flow
def txn_flow():
    with transaction():
        first_task()
        second_task()
if __name__ == "__main__":
    txn_flow()

Open source Events and Automations

Trigger actions, such as sending notifications, pausing schedules, starting flow runs and more in response to Prefect events.

More flexible variables and new artifact types

Variables can now be any JSON compatible type including dicts, lists, and integers. Progress and Image artifacts make it easy to add visual annotations to your flow run graph.

Faster and richer CLI

Improved CLI speed and several added commands and conveniences.

Updated navigation, styling, and interaction design

The new Runs page displays both flow and task run information, and an improved sidebar and switcher makes navigating Prefect simpler than ever.

Enhancements

Fixes

Experimental

Documentation

Integrations

Breaking Changes

Contributors

  • @bsignoret

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.0...3.0.0rc1

See release notes for details.

prefect - Release 2.19.3

Published by abrookins 5 months ago

New method for generating parameter schemas without dependencies

prefect deploy now works even when dependencies are missing from the current environment. This can speed up deployment via CI by removing the need to install dependencies before deploying your flows.

See the following pull requests for implementation details:

Enhancements

Fixes

Integrations

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.2...2.19.3

prefect - Release 2.19.2

Published by serinamarie 5 months ago

This release includes a few bug fixes, ensuring:

prefect - 2.19.1

Published by desertaxle 5 months ago

Release 2.19.1

Enhancements to prefect-dbt for running dbt-core commands

Introducing prefect-dbt summary artifacts! With summary artifacts, you get a view of all of the runs that succeeded, failed, or were skipped as well as where the failed models live in your dbt project and why they failed. This exposes information about each dbt node in a succinct format that teams can use for development, debugging, insights, and so much more.

These changes are available in prefect-dbt version 0.5.0!

See the following pull requests for implementation details:

Enhancements

Experimental

Events and Automations

Documentation

Integrations

Integration library releases

This release was accompanied by the following integration library releases:

  • prefect-aws - 0.4.17
  • prefect-azure - 0.3.10
  • prefect-bitbucket - 0.2.5
  • prefect-dask - 0.2.9
  • prefect-databricks - 0.2.7
  • prefect-dbt - 0.5.0
  • prefect-docker - 0.5.2
  • prefect-email - 0.3.5
  • prefect-gcp - 0.5.12
  • prefect-github - 0.2.5
  • prefect-gitlab - 0.2.5
  • prefect-kubernetes - 0.3.10
  • prefect-ray - 0.3.6
  • prefect-shell - 0.2.5
  • prefect-slack - 0.2.6
  • prefect-snowflake - 0.27.6
  • prefect-sqlalchemy - 0.4.3

All changes: https://github.com/PrefectHQ/prefect/compare/2.19.0...2.19.1

prefect - Release 2.19.0

Published by serinamarie 5 months ago

✨ This release includes a number of enhancements and fixes!

Support for major infrastructure and distributed task integrations

As prefect-dask and other integrations have been added to the prefect codebase, this release adds these integrations as extra requirements of the prefect package, making it easier to install support for everything in your Prefect stack:

pip install 'prefect[dask]'

We loved this community contribution so much, we did it for all our first-party integrations:

pip install 'prefect[aws,kubernetes,dask,dbt,sqlalchemy,slack]'

You can see the full list of Prefect's extra requirements in our setup.py.

Support for timeout seconds in global concurrency context manager

You may want to fail immediately if a global concurrency slot is unavailable. Rather than block and wait, you can now specify a timeout_seconds argument in the global concurrency context manager and catch a TimeoutError if a slot is not available within the specified time.

@flow
def fail_immediately_flow():
    try:
        with concurrency("there-can-be-only-one", occupy=1, timeout_seconds=0.1):
            do_something_resource_intensive()
    except TimeoutError:
        return Cancelled(message="Another flow run is already running")

Manage global concurrency limits via the CLI

Global concurrency limits let you control how many operations can run simultaneously-- now you can create, read, edit, and delete global concurrency limits via the Prefect CLI!

To create a new concurrency limit, use the prefect gcl create command. You must specify a --limit argument, and can optionally specify a --slot-decay-per-second and --disable argument.

prefect gcl create my-concurrency-limit --limit 5 --slot-decay-per-second 1.0

You can inspect the details of a concurrency limit using the prefect gcl inspect command:

prefect gcl inspect my-concurrency-limit

To update a concurrency limit, use the prefect gcl update command. You can update the --limit, --slot-decay-per-second, --enable, and --disable arguments:

prefect gcl update my-concurrency-limit --limit 10

We also have many more bug fixes and in-flight work! See the release notes for details!

Package Rankings
Top 0.45% on Pypi.org
Top 4.06% on Proxy.golang.org
Top 4.91% on Conda-forge.org
Related Projects
echodataflow

Orchestrated sonar data processing workflow

21 Jan 2022 4
devops-resources

DevOps resources - Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes...

07 Jan 2019 8,538
runhouse

The fastest way to iterate and deploy AI workloads on your own infra. Unobtrusive, debuggable, Py...

10 May 2022 707
fastapi_cloudrun_pubsub

Python API, queues and workers on GCP: FastAPI + CloudRun + PubSub

22 Aug 2023 3
prefect-tutorial

Python Prefect tutorial

01 Jan 2022 9