prefect

Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines

APACHE-2.0 License

Downloads
8.4M
Stars
14.5K
Committers
148
View Code on GitHub Visit Website
Ecosystems: Python

Bot releases are visible (Hide)

prefect - 2.20.7: Big Data London is Next Week, see you there? Latest Release

Published by cicdw about 1 month ago

Enhancements โž•โž•

Bug Fixes ๐Ÿž

Integrations & Dependencies ๐Ÿค

Development & Tidiness ๐Ÿงน

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.6...2.20.7

prefect - 3.0.1: The Director's Cut

Published by cicdw about 1 month ago

3.0.1

This release contains a large collection of relatively small improvements and bug fixes on the back of our major 3.0.0 release. One particular set of improvements that I'd like to focus on are enhancements to server side logging and event handling. We received a few reports of task run states not showing up in the UI/API. It's important to note that events - and now task run states! - require a websocket connection to the API (the good news is that this will not disrupt workflow execution). Websockets are sometimes blocked by default in firewalls and network configuration, so we are working to raise failures to establish such a connection more prominently and potentially allow for alternative configurations. If you notice any issues with this, please let us know!

Enhancements โž•โž•

Bug Fixes ๐Ÿž

Integrations & Dependencies ๐Ÿค

Development & Tidiness ๐Ÿงน

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0...3.0.1

prefect - 2.20.6: An extra release to fix a problem with extras

Published by cicdw about 1 month ago

This release resolves an important issue that can occur when installing 2.x versions of prefect: because all prefect-{integration} packages also have 3.0 compatible versions, it's important to restrict the upper bounds of those packages alongside the needed upper bound on prefect itself. Please reference this list for a full catalogue of upper bounds that keep your integration package 2.x compatible.

To help alleviate this potential issue, this release applies these upper bounds automatically when installing packages via extras, i.e., prefect[kubernetes]<3.0.0 or prefect[aws]<3.0.0.

In addition, it fixes an issue with -kubernetes flavored Docker images on the 2.20.5 release that was affected by this incompatibility.

Bug Fixes ๐Ÿž

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.5...2.20.6

prefect - 2.20.5: Several Types of Small Code Fixes Gathered Together in a Release and Grooving with a K8s Pod

Published by cicdw about 1 month ago

Several Types of Small Code Fixes Gathered Together in a Release and Grooving with a K8s Pod

Bug Fixes ๐Ÿž

Integrations & Dependencies ๐Ÿค

Development & Tidiness ๐Ÿงน

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.4...2.20.5

prefect - 3.0.0

Published by cicdw about 2 months ago

Prefect 3.0.0

For more information see our blog post on 3.0.

New Contributors

Thank you to all the new contributors who helped make 3.0 happen!

@jaraics, @mitchnielsen, @swadhinbiswas, @elisalimli, @LarryUllman, @djsauble, @Waiwait, @mthatt, @rooperuu, @tonykipkemboi, @ravi77o, @ZhipengXue97, @bjorhn, @50Byte, @taranl, @peterbygrave, @s, @jongbinjung, @GalLadislav, @glesperance, @nialloriordanroo, and @tmylt

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.0...3.0.0

prefect - 2.20.4: We Held a Funeral for the Bugs. No One Came

Published by cicdw about 2 months ago

2.20.4: We Held a Funeral for the Bugs. No One Came

Bug Fixes ๐Ÿž

Integrations & Dependencies ๐Ÿค

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.3...2.20.4

prefect - Weekly Release Candidate 3.0.0rc20

Published by marvin-robot about 2 months ago

What's Changed

New Features ๐ŸŽ‰

Bug Fixes ๐Ÿž

Development & Tidiness ๐Ÿงน

Uncategorized

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc19...3.0.0rc20

prefect - 2.20.3: Background to the future

Published by cicdw about 2 months ago

2.20.3: Background to the future

Two items of note in this release:

A note on dependencies: Last week, a major version upgrade of one of Prefect's dependencies (griffe) caused quite the headache for some users' environments, and we are sorry for the inconvenience this caused; in general, we strive to keep major version upper bounds on all of our dependencies, but an oversight caused this dependency to go without an upper bound. This release fixes that issue and added in a few other upper bounds that weren't present.

prefect server start --background: By popular request, this release features a new flag on prefect server start that runs the server in a background process. When starting a background server, a server.pid file is written to the $PREFECT_HOME directory with the process ID of the background server. This lets prefect server stop know which process to interrupt. Currently, only one background server can be started at a time.

New Features ๐ŸŽ‰

Bug Fixes ๐Ÿž

Integrations & Dependencies ๐Ÿค

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.2...2.20.3

prefect - Weekly Release Candidate 3.0.0rc19

Published by marvin-robot about 2 months ago

What's Changed

New Features ๐ŸŽ‰

Bug Fixes ๐Ÿž

Integrations & Dependencies ๐Ÿค

Development & Tidiness ๐Ÿงน

Uncategorized

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc18...3.0.0rc19

prefect - 2.20.2

Published by cicdw 2 months ago

2.20.2: Save the Environment (Variables)

This release contains many bug fixes and small enhancements; two that I want to highlight are:

  • a fix to KubernetesJob.publish_as_work_pool: this method helps users migrate from agents to workers by exposing a method to convert an infrastructure block to a structured work pool. Work Pools allow for better governance of infrastructure, and will be easier to observe and audit than agents.
  • a fix to how environment variables on work pools are handled: specifically, users who provided per-flow-run environment variable overrides found that any other environment variables explicitly set on their work pool or deployment were being removed. This fix ensures that all environment variable configuration is carefully merged, preventing the loss of any variables at runtime.

New Features ๐ŸŽ‰

Bug Fixes ๐Ÿž

Development & Tidiness ๐Ÿงน

Integrations & Dependencies ๐Ÿค

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.1...2.20.2

prefect - Weekly Release Candidate 3.0.0rc18

Published by marvin-robot 2 months ago

What's Changed

New Features ๐ŸŽ‰

Development & Tidiness ๐Ÿงน

Uncategorized

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc17...3.0.0rc18

prefect - Weekly Release Candidate 3.0.0rc17

Published by marvin-robot 2 months ago

What's Changed

New Features ๐ŸŽ‰

Bug Fixes ๐Ÿž

Development & Tidiness ๐Ÿงน

Uncategorized

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc16...3.0.0rc17

prefect - Weekly Release Candidate 3.0.0rc16

Published by marvin-robot 2 months ago

What's Changed

New Features ๐ŸŽ‰

Bug Fixes ๐Ÿž

Development & Tidiness ๐Ÿงน

Uncategorized

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc15...3.0.0rc16

prefect - 2.20.1: pip, pip hooray!

Published by desertaxle 2 months ago

This release is small and includes fixes for various dependency issues, as well as a fix for submodule handling in git-based deployments.

Python 3.11 introduced the concept of "exception groups" for situations in which multiple unrelated exceptions need to be raised - for example, in concurrent execution models or retrying an operation across distinct error modes.
Prefect has begun to leverage this capability, and we rely on a separate package (exceptiongroup) which backports
this functionality to lower supported versions of Python. This dependency was missing from client-side requirements in 2.20.0.

Bug Fixes ๐Ÿž

Integrations & Dependencies ๐Ÿค

Development & Tidiness ๐Ÿงน

prefect - Weekly Release Candidate 3.0.0rc15

Published by marvin-robot 2 months ago

What's Changed

New Features ๐ŸŽ‰

Bug Fixes ๐Ÿž

Development & Tidiness ๐Ÿงน

Uncategorized

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc14...3.0.0rc15

prefect - 2.20.0: CVE you later

Published by cicdw 3 months ago

Continuing the 2.0 lineage, Prefect's 2.20 release includes a major compatibility upgrade with anyio: previous versions of anyio were subject to a critical race condition that many CVE scanners are flagging as a high severity vulnerability. Because of the changes necessary in this upgrade, please be aware that new versions of many Prefect integration packages such as prefect-dask and prefect-ray will require 2.20.0 to function properly.

(For those who have already upgraded to 3.0, don't worry - Prefect 3.0 release candidates are already anyio>=4.4.0 compatible).

Bug Fixes ๐Ÿž

Integrations & Dependencies ๐Ÿค

Development & Tidiness ๐Ÿงน

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.9...2.20

prefect - Weekly Release Candidate 3.0.0rc14

Published by marvin-robot 3 months ago

What's Changed

New Features ๐ŸŽ‰

Bug Fixes ๐Ÿž

Development & Tidiness ๐Ÿงน

Uncategorized

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc13...3.0.0rc14

prefect - Weekly Release Candidate 3.0.0rc13

Published by marvin-robot 3 months ago

What's Changed

Development & Tidiness ๐Ÿงน

Uncategorized

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc12...3.0.0rc13

prefect - 2.19.9: Sync or Swim

Published by cicdw 3 months ago

This PR continues Prefect's 2.0 lineage as we prepare for a major 3.0 release. This release is particularly special with respect to 3.0 as it adds warnings to APIs or interfaces that will need updating in 3.0.

There are many internal APIs in Prefect that can be run both synchronously and asynchronously depending on the runtime context in which they are called. If you've ever encountered the error AttributeError: 'coroutine' object has no attribute when loading a Block then you know what I'm referring to. In 3.0 we have updated the internal logic for switching between synchronous/asynchronous modes in an attempt to provide clearer guidelines and improved performance.

If you ever encounter such an error, you can now use the special _sync keyword argument to explicitly enforce the mode of execution you'd prefer:

my_block = Block.load("example/block", _sync=True)

my_block = await Block.load("example/block", _sync=False)

When this kwarg is not provided, Prefect makes an educated guess as to whether a coroutine should be returned to await or whether to run the logic synchronously on your behalf.

New Features ๐ŸŽ‰

Bug Fixes ๐Ÿž

Development & Tidiness ๐Ÿงน

Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.8...2.19.9

prefect - Weekly Release Candidate 3.0.0rc12

Published by marvin-robot 3 months ago

What's Changed

Enhancements

Fixes

Documentation

Uncategorized

New Contributors

Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc11...3.0.0rc12

Package Rankings
Top 0.45% on Pypi.org
Top 4.06% on Proxy.golang.org
Top 4.91% on Conda-forge.org
Related Projects
runhouse

The fastest way to iterate and deploy AI workloads on your own infra. Unobtrusive, debuggable, Py...

10 May 2022 707
prefect-tutorial

Python Prefect tutorial

01 Jan 2022 9
devops-resources

DevOps resources - Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes...

07 Jan 2019 8,538
echodataflow

Orchestrated sonar data processing workflow

21 Jan 2022 4
fastapi_cloudrun_pubsub

Python API, queues and workers on GCP: FastAPI + CloudRun + PubSub

22 Aug 2023 3