Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
APACHE-2.0 License
Bot releases are hidden (Show)
Published by cicdw about 1 month ago
safe_load_namespace
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15361
DbtCoreOperation
docstring 2.x by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15318
prefect-docker
cache key by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15263
google
into import guard by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15333
prefect-databricks
in 2.x by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15315
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.6...2.20.7
Published by cicdw about 1 month ago
This release contains a large collection of relatively small improvements and bug fixes on the back of our major 3.0.0 release. One particular set of improvements that I'd like to focus on are enhancements to server side logging and event handling. We received a few reports of task run states not showing up in the UI/API. It's important to note that events - and now task run states! - require a websocket connection to the API (the good news is that this will not disrupt workflow execution). Websockets are sometimes blocked by default in firewalls and network configuration, so we are working to raise failures to establish such a connection more prominently and potentially allow for alternative configurations. If you notice any issues with this, please let us know!
prompt_select_from_table
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15202
ResultStore
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15214
CausalOrdering
from task-run-recorder
by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/15244
ResultStore
with a Transaction
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15247
LockManager
protocol and add locking abilities to ResultStore
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15225
pull_code
instruction by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15215
prefect-docker
cache key by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15262
KeepAliveClientRequest
class for k8s async client by @kevingrismore in https://github.com/PrefectHQ/prefect/pull/15220
enable_socket_keep_alive
by @jeanluciano in https://github.com/PrefectHQ/prefect/pull/15142
prefect-kubernetes
: add missing config loaders by @srsapient in https://github.com/PrefectHQ/prefect/pull/15209
prefect-azure
tests by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15211
uv
dev dependency by @cicdw in https://github.com/PrefectHQ/prefect/pull/15251
pydantic==2.9.0
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15249
is not
for type comparison instead of !=
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15253
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0...3.0.1
Published by cicdw about 1 month ago
This release resolves an important issue that can occur when installing 2.x versions of prefect
: because all prefect-{integration}
packages also have 3.0 compatible versions, it's important to restrict the upper bounds of those packages alongside the needed upper bound on prefect
itself. Please reference this list for a full catalogue of upper bounds that keep your integration package 2.x compatible.
To help alleviate this potential issue, this release applies these upper bounds automatically when installing packages via extras, i.e., prefect[kubernetes]<3.0.0
or prefect[aws]<3.0.0
.
In addition, it fixes an issue with -kubernetes
flavored Docker images on the 2.20.5 release that was affected by this incompatibility.
setup.py
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15256
kubernetes
extra when building Docker image by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15257
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.5...2.20.6
Published by cicdw about 1 month ago
jobs_runs_submit_and_wait_for_completion
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15183
prefect-kubernetes
: add missing config loaders by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15218
KeepAliveClientRequest
class for k8s async client 2.x by @kevingrismore in https://github.com/PrefectHQ/prefect/pull/15230
apprise
version by @cicdw in https://github.com/PrefectHQ/prefect/pull/15186
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.4...2.20.5
Published by cicdw about 2 months ago
For more information see our blog post on 3.0.
Thank you to all the new contributors who helped make 3.0 happen!
@jaraics, @mitchnielsen, @swadhinbiswas, @elisalimli, @LarryUllman, @djsauble, @Waiwait, @mthatt, @rooperuu, @tonykipkemboi, @ravi77o, @ZhipengXue97, @bjorhn, @50Byte, @taranl, @peterbygrave, @s, @jongbinjung, @GalLadislav, @glesperance, @nialloriordanroo, and @tmylt
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.0...3.0.0
Published by cicdw about 2 months ago
prefect server start
for Windows machines by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15104
on_failure
hook runs upon parameter validation error by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15128
prefect-kubernetes
fix by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15095
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.3...2.20.4
Published by cicdw about 2 months ago
Two items of note in this release:
A note on dependencies: Last week, a major version upgrade of one of Prefect's dependencies (griffe
) caused quite the headache for some users' environments, and we are sorry for the inconvenience this caused; in general, we strive to keep major version upper bounds on all of our dependencies, but an oversight caused this dependency to go without an upper bound. This release fixes that issue and added in a few other upper bounds that weren't present.
prefect server start --background
: By popular request, this release features a new flag on prefect server start
that runs the server in a background process. When starting a background server, a server.pid
file is written to the $PREFECT_HOME
directory with the process ID of the background server. This lets prefect server stop
know which process to interrupt. Currently, only one background server can be started at a time.
--overwrite
for create_work_pool
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14977
griffe
requirement by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14983
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.2...2.20.3
Published by cicdw 2 months ago
This release contains many bug fixes and small enhancements; two that I want to highlight are:
KubernetesJob.publish_as_work_pool
: this method helps users migrate from agents to workers by exposing a method to convert an infrastructure block to a structured work pool. Work Pools allow for better governance of infrastructure, and will be easier to observe and audit than agents.$
character by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14888
prefect-client
across all supported Python versions by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14872
stream_output
to dbt tasks by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14866
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.1...2.20.2
Published by desertaxle 2 months ago
This release is small and includes fixes for various dependency issues, as well as a fix for submodule handling in git-based deployments.
Python 3.11 introduced the concept of "exception groups" for situations in which multiple unrelated exceptions need to be raised - for example, in concurrent execution models or retrying an operation across distinct error modes.
Prefect has begun to leverage this capability, and we rely on a separate package (exceptiongroup
) which backports
this functionality to lower supported versions of Python. This dependency was missing from client-side requirements in 2.20.0
.
exceptiongroup
to requirements-client.txt
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14871
flow_run_id
in Log
table by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14830
Published by cicdw 3 months ago
Continuing the 2.0 lineage, Prefect's 2.20 release includes a major compatibility upgrade with anyio
: previous versions of anyio
were subject to a critical race condition that many CVE scanners are flagging as a high severity vulnerability. Because of the changes necessary in this upgrade, please be aware that new versions of many Prefect integration packages such as prefect-dask
and prefect-ray
will require 2.20.0 to function properly.
(For those who have already upgraded to 3.0, don't worry - Prefect 3.0 release candidates are already anyio>=4.4.0
compatible).
prefect deploy
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14782
2.x
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14774
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.9...2.20
Published by cicdw 3 months ago
This PR continues Prefect's 2.0 lineage as we prepare for a major 3.0 release. This release is particularly special with respect to 3.0 as it adds warnings to APIs or interfaces that will need updating in 3.0.
There are many internal APIs in Prefect that can be run both synchronously and asynchronously depending on the runtime context in which they are called. If you've ever encountered the error AttributeError: 'coroutine' object has no attribute
when loading a Block
then you know what I'm referring to. In 3.0 we have updated the internal logic for switching between synchronous/asynchronous modes in an attempt to provide clearer guidelines and improved performance.
If you ever encounter such an error, you can now use the special _sync
keyword argument to explicitly enforce the mode of execution you'd prefer:
my_block = Block.load("example/block", _sync=True)
my_block = await Block.load("example/block", _sync=False)
When this kwarg is not provided, Prefect makes an educated guess as to whether a coroutine should be returned to await or whether to run the logic synchronously on your behalf.
container_create_kwargs
to Docker worker by @kevingrismore in https://github.com/PrefectHQ/prefect/pull/14693
load_flow_from_entrypoint
in 2.x by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14669
PrefectDeprecationWarning
in tests by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14708
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.8...2.19.9
Published by bunchesofdonald 3 months ago
ParameterTypeError.from_validation_error
โ https://github.com/PrefectHQ/prefect/pull/14420
--ui/--no-ui
flag on prefect server start
โ https://github.com/PrefectHQ/prefect/pull/14520
AwsClientParameters
is a dict
โ https://github.com/PrefectHQ/prefect/pull/14438
print_func
for run_namespaced_job
โ https://github.com/PrefectHQ/prefect/pull/14463
GcsBucket.put/get_directory
โ https://github.com/PrefectHQ/prefect/pull/14537
Published by serinamarie 4 months ago
PREFECT_API_URL
when starting server if not set โ https://github.com/PrefectHQ/prefect/pull/14294
prefect deploy
โ https://github.com/PrefectHQ/prefect/pull/14307
prefect deploy
breaks using a trigger with within
โ https://github.com/PrefectHQ/prefect/pull/14338
cache_result_in_memory=False
โ https://github.com/PrefectHQ/prefect/pull/14359
prefect deploy
โ https://github.com/PrefectHQ/prefect/pull/14405
ParameterTypeError.from_validation_error
โ https://github.com/PrefectHQ/prefect/pull/14419
flow-run
CLI log printing and escape special characters โ https://github.com/PrefectHQ/prefect/pull/14469
main
for migration script in upgrade doc โ https://github.com/PrefectHQ/prefect/pull/14365
Deployment
overview and add note on run_deployment
sdk use โ https://github.com/PrefectHQ/prefect/pull/14425
RunInput
examples โ https://github.com/PrefectHQ/prefect/pull/14473
All changes: https://github.com/PrefectHQ/prefect/compare/3.0.0rc9...3.0.0rc10
Published by desertaxle 4 months ago
prefect deploy
- https://github.com/PrefectHQ/prefect/pull/14403
uvicorn>0.29.0
- https://github.com/PrefectHQ/prefect/pull/14370
2.x
branch in GitHub - https://github.com/PrefectHQ/prefect/pull/14378
All changes: https://github.com/PrefectHQ/prefect/compare/2.19.6...2.19.7
Published by desertaxle 4 months ago
parameters
from flow definition if none are given otherwise by @bunchesofdonald in https://github.com/PrefectHQ/prefect/pull/14220
kubernetes-cluster
configto
collection_blocks_data.json` by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14266
requests
for prefect-docker
by @chrisguidry in https://github.com/PrefectHQ/prefect/pull/14276
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc8...3.0.0rc9
Published by desertaxle 4 months ago
Flow.deploy
- https://github.com/PrefectHQ/prefect/pull/13981
concurrency
timeout scoping - https://github.com/PrefectHQ/prefect/pull/14183
job_variables
in a prefect.yaml
before saving server-side - https://github.com/PrefectHQ/prefect/pull/14156
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.5...2.19.6
Published by desertaxle 4 months ago
PrefectFuture
are garbage collection before resolution by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14148
PrefectFuture
s to flows by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14152
ConcurrentTaskRunner
from docs and ensure examples use .wait()
by @zangell44 in https://github.com/PrefectHQ/prefect/pull/14144
KubernetesClusterConfig
import by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14154
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc4...3.0.0rc5
Published by desertaxle 4 months ago
TaskWorker
by @chrisguidry in https://github.com/PrefectHQ/prefect/pull/14089
PENDING
task rescheduling by default by @chrisguidry in https://github.com/PrefectHQ/prefect/pull/14128
KubernetesClusterConfig
block from the core library by @chrisguidry in https://github.com/PrefectHQ/prefect/pull/14107
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc3...3.0.0rc4
Published by desertaxle 4 months ago
@task
and @flow
โ https://github.com/PrefectHQ/prefect/pull/13944
task_worker
runs against ephemeral server โ https://github.com/PrefectHQ/prefect/pull/13848
ThreadPoolTaskRunner
โ https://github.com/PrefectHQ/prefect/pull/13866
not_any_
filters to flow run state name/type โ https://github.com/PrefectHQ/prefect/pull/13920
error_detail
for validation errors โ https://github.com/PrefectHQ/prefect/pull/13940
url_for
utility for generating URLs for Prefect objects โ https://github.com/PrefectHQ/prefect/pull/13885
shield
when re-enqueuing task runs โ https://github.com/PrefectHQ/prefect/pull/13883
ThreadPoolTaskRunner
concurrency by copying max_workers
on duplication โ https://github.com/PrefectHQ/prefect/pull/13943
prefect deploy
with dynamic @flow decorator args โ https://github.com/PrefectHQ/prefect/pull/13967
if __name__ == "__main__":
blocks when running prefect deploy
โ https://github.com/PrefectHQ/prefect/pull/13922
GlobalConcurrencyLimitUpdate
โ https://github.com/PrefectHQ/prefect/pull/14005
TaskRunWaiter
cache size and expiration time and pin task worker thread pool workers to passed limit
โ https://github.com/PrefectHQ/prefect/pull/14030
PREFECT_API_URL
to start a worker โ https://github.com/PrefectHQ/prefect/pull/13942
LiteralResult
and handle singleton values the same โ https://github.com/PrefectHQ/prefect/pull/13905
pydantic
syntax in docs โ https://github.com/PrefectHQ/prefect/pull/13888
EngineContext
โ https://github.com/PrefectHQ/prefect/pull/13917
jobs_runs_submit_by_id_and_wait_for_completion
โ https://github.com/PrefectHQ/prefect/pull/13958
All changes: https://github.com/PrefectHQ/prefect/compare/3.0.0rc2...3.0.0rc3
Published by jakekaplan 4 months ago
prefect deploy
when using dynamic @flow
parameters - https://github.com/PrefectHQ/prefect/pull/13991
prefect deploy
to prevent code execution inside main guard - https://github.com/PrefectHQ/prefect/pull/14004
GlobalConcurrencyLimitUpdate
- https://github.com/PrefectHQ/prefect/pull/14006
jobs_runs_submit_by_id_and_wait_for_completion
to prefect-databricks
- https://github.com/PrefectHQ/prefect/pull/13957
All changes: https://github.com/PrefectHQ/prefect/compare/2.19.4...2.19.5