Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
APACHE-2.0 License
Bot releases are visible (Hide)
Published by cicdw about 1 month ago
safe_load_namespace
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15361
DbtCoreOperation
docstring 2.x by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15318
prefect-docker
cache key by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15263
google
into import guard by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15333
prefect-databricks
in 2.x by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15315
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.6...2.20.7
Published by cicdw about 1 month ago
This release contains a large collection of relatively small improvements and bug fixes on the back of our major 3.0.0 release. One particular set of improvements that I'd like to focus on are enhancements to server side logging and event handling. We received a few reports of task run states not showing up in the UI/API. It's important to note that events - and now task run states! - require a websocket connection to the API (the good news is that this will not disrupt workflow execution). Websockets are sometimes blocked by default in firewalls and network configuration, so we are working to raise failures to establish such a connection more prominently and potentially allow for alternative configurations. If you notice any issues with this, please let us know!
prompt_select_from_table
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15202
ResultStore
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15214
CausalOrdering
from task-run-recorder
by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/15244
ResultStore
with a Transaction
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15247
LockManager
protocol and add locking abilities to ResultStore
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15225
pull_code
instruction by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15215
prefect-docker
cache key by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15262
KeepAliveClientRequest
class for k8s async client by @kevingrismore in https://github.com/PrefectHQ/prefect/pull/15220
enable_socket_keep_alive
by @jeanluciano in https://github.com/PrefectHQ/prefect/pull/15142
prefect-kubernetes
: add missing config loaders by @srsapient in https://github.com/PrefectHQ/prefect/pull/15209
prefect-azure
tests by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15211
uv
dev dependency by @cicdw in https://github.com/PrefectHQ/prefect/pull/15251
pydantic==2.9.0
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15249
is not
for type comparison instead of !=
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15253
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0...3.0.1
Published by cicdw about 1 month ago
This release resolves an important issue that can occur when installing 2.x versions of prefect
: because all prefect-{integration}
packages also have 3.0 compatible versions, it's important to restrict the upper bounds of those packages alongside the needed upper bound on prefect
itself. Please reference this list for a full catalogue of upper bounds that keep your integration package 2.x compatible.
To help alleviate this potential issue, this release applies these upper bounds automatically when installing packages via extras, i.e., prefect[kubernetes]<3.0.0
or prefect[aws]<3.0.0
.
In addition, it fixes an issue with -kubernetes
flavored Docker images on the 2.20.5 release that was affected by this incompatibility.
setup.py
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15256
kubernetes
extra when building Docker image by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15257
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.5...2.20.6
Published by cicdw about 1 month ago
jobs_runs_submit_and_wait_for_completion
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15183
prefect-kubernetes
: add missing config loaders by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15218
KeepAliveClientRequest
class for k8s async client 2.x by @kevingrismore in https://github.com/PrefectHQ/prefect/pull/15230
apprise
version by @cicdw in https://github.com/PrefectHQ/prefect/pull/15186
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.4...2.20.5
Published by cicdw about 2 months ago
For more information see our blog post on 3.0.
Thank you to all the new contributors who helped make 3.0 happen!
@jaraics, @mitchnielsen, @swadhinbiswas, @elisalimli, @LarryUllman, @djsauble, @Waiwait, @mthatt, @rooperuu, @tonykipkemboi, @ravi77o, @ZhipengXue97, @bjorhn, @50Byte, @taranl, @peterbygrave, @s, @jongbinjung, @GalLadislav, @glesperance, @nialloriordanroo, and @tmylt
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.0...3.0.0
Published by cicdw about 2 months ago
prefect server start
for Windows machines by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15104
on_failure
hook runs upon parameter validation error by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15128
prefect-kubernetes
fix by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15095
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.3...2.20.4
Published by marvin-robot about 2 months ago
Secret
block by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14980
Deployment.concurrency_limits
by @jeanluciano in https://github.com/PrefectHQ/prefect/pull/15022
PREFECT_API_URL
on server start by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15122
PrefectBaseModel
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15072
prefect server start
for Windows machines by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15103
CloudUnauthorized
case in prefect dashboard open
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15106
on_failure
hook runs upon parameter validation error by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15109
model_rebuild
in main.py
to explicitly pass namespace types by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15117
PersistedResultBlob
with ResultRecord
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15064
concurrency
ctx managers by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15108
Manifest
class by @desertaxle in https://github.com/PrefectHQ/prefect/pull/15111
experimental_defer_build_mode
with defer_build=True
by @abrookins in https://github.com/PrefectHQ/prefect/pull/15052
prefect dashboard open
to open dashboard from CLI by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14985
on_crash
hook example by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15079
Manage concurrency
docs by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/15088
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc19...3.0.0rc20
Published by cicdw about 2 months ago
Two items of note in this release:
A note on dependencies: Last week, a major version upgrade of one of Prefect's dependencies (griffe
) caused quite the headache for some users' environments, and we are sorry for the inconvenience this caused; in general, we strive to keep major version upper bounds on all of our dependencies, but an oversight caused this dependency to go without an upper bound. This release fixes that issue and added in a few other upper bounds that weren't present.
prefect server start --background
: By popular request, this release features a new flag on prefect server start
that runs the server in a background process. When starting a background server, a server.pid
file is written to the $PREFECT_HOME
directory with the process ID of the background server. This lets prefect server stop
know which process to interrupt. Currently, only one background server can be started at a time.
--overwrite
for create_work_pool
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14977
griffe
requirement by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14983
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.2...2.20.3
Published by marvin-robot about 2 months ago
create_work_pool
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14967
RedisRecordStore
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14974
FileSystemRecordStore
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14966
concurrency_limit
to flow.deploy
by @jeanluciano in https://github.com/PrefectHQ/prefect/pull/15001
concurrency_limit
to prefect.yaml
by @jeanluciano in https://github.com/PrefectHQ/prefect/pull/15007
prefect deploy
by @jeanluciano in https://github.com/PrefectHQ/prefect/pull/15005
concurrency
helpers that use v1 task run concurrency limits by @abrookins in https://github.com/PrefectHQ/prefect/pull/15037
prefect-redis
to the core repository by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14968
griffe
requirement by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14982
client.create_deployment
by @jeanluciano in https://github.com/PrefectHQ/prefect/pull/14997
write tasks
guide. by @aaazzam in https://github.com/PrefectHQ/prefect/pull/15017
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc18...3.0.0rc19
Published by cicdw 2 months ago
This release contains many bug fixes and small enhancements; two that I want to highlight are:
KubernetesJob.publish_as_work_pool
: this method helps users migrate from agents to workers by exposing a method to convert an infrastructure block to a structured work pool. Work Pools allow for better governance of infrastructure, and will be easier to observe and audit than agents.$
character by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14888
prefect-client
across all supported Python versions by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14872
stream_output
to dbt tasks by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14866
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.20.1...2.20.2
Published by marvin-robot 2 months ago
MemoryRecordStore
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14919
prefect block register
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14949
WorkQueue
action fields by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14953
obfuscate
instead of obfuscate_string
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14959
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc17...3.0.0rc18
Published by marvin-robot 2 months ago
__getitem__
calls on a PrefectFuture
or State
in checking Flow
parameter serialization by @Andrew-S-Rosen in https://github.com/PrefectHQ/prefect/pull/14900
is_interactive
function during server preflight check by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14909
TaskSource
key computation if no source code by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14926
test_futures.py
by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14902
test_task_results.py
by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14904
test_tasks.py
by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14905
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc16...3.0.0rc17
Published by marvin-robot 2 months ago
runtime
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14897
$
character by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14876
Deployment.is_schedule_active
and Deployment.schedule
by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14794
schedules: null
in prefect.yaml
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14873
say_hello.submit(name="Marvin)
by @Andrew-S-Rosen in https://github.com/PrefectHQ/prefect/pull/14880
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc15...3.0.0rc16
Published by desertaxle 2 months ago
This release is small and includes fixes for various dependency issues, as well as a fix for submodule handling in git-based deployments.
Python 3.11 introduced the concept of "exception groups" for situations in which multiple unrelated exceptions need to be raised - for example, in concurrent execution models or retrying an operation across distinct error modes.
Prefect has begun to leverage this capability, and we rely on a separate package (exceptiongroup
) which backports
this functionality to lower supported versions of Python. This dependency was missing from client-side requirements in 2.20.0
.
exceptiongroup
to requirements-client.txt
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14871
flow_run_id
in Log
table by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14830
Published by marvin-robot 2 months ago
checkpoint_every
by @bunchesofdonald in https://github.com/PrefectHQ/prefect/pull/14834
task-run.Scheduled
event emission by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14849
wrapt
and fail on incorrect await
usage by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14837
prefect_test_harness
to avoid error logs by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14854
pytest
command by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14856
flow_run_id
nullable in Log
table by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14828
test
profile to default profiles by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14836
Flow.serve
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14848
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc14...3.0.0rc15
Published by cicdw 3 months ago
Continuing the 2.0 lineage, Prefect's 2.20 release includes a major compatibility upgrade with anyio
: previous versions of anyio
were subject to a critical race condition that many CVE scanners are flagging as a high severity vulnerability. Because of the changes necessary in this upgrade, please be aware that new versions of many Prefect integration packages such as prefect-dask
and prefect-ray
will require 2.20.0 to function properly.
(For those who have already upgraded to 3.0, don't worry - Prefect 3.0 release candidates are already anyio>=4.4.0
compatible).
prefect deploy
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14782
2.x
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14774
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.9...2.20
Published by marvin-robot 3 months ago
as_completed
utility for PrefectFuture
by @jeanluciano in https://github.com/PrefectHQ/prefect/pull/14641
Completed(name="RolledBack")
by @collincchoy in https://github.com/PrefectHQ/prefect/pull/14721
prefect deploy
by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14694
PrefectHttpxSyncEphemeralClient
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14823
/download-logs/csv
endpoint to /logs/download
by @pleek91 in https://github.com/PrefectHQ/prefect/pull/14817
prefect profiles populate-defaults
CLI command by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14749
prefect profile populate-defaults
by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14795
PRAGMA temp_store = memory
in sqlite setup by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14812
prefect_test_harness
to run a test server in a subprocess by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14818
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc13...3.0.0rc14
Published by marvin-robot 3 months ago
EventsPipeline
and testing utilities by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14765
TaskWorker
set state api call by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14769
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc12...3.0.0rc13
Published by cicdw 3 months ago
This PR continues Prefect's 2.0 lineage as we prepare for a major 3.0 release. This release is particularly special with respect to 3.0 as it adds warnings to APIs or interfaces that will need updating in 3.0.
There are many internal APIs in Prefect that can be run both synchronously and asynchronously depending on the runtime context in which they are called. If you've ever encountered the error AttributeError: 'coroutine' object has no attribute
when loading a Block
then you know what I'm referring to. In 3.0 we have updated the internal logic for switching between synchronous/asynchronous modes in an attempt to provide clearer guidelines and improved performance.
If you ever encounter such an error, you can now use the special _sync
keyword argument to explicitly enforce the mode of execution you'd prefer:
my_block = Block.load("example/block", _sync=True)
my_block = await Block.load("example/block", _sync=False)
When this kwarg is not provided, Prefect makes an educated guess as to whether a coroutine should be returned to await or whether to run the logic synchronously on your behalf.
container_create_kwargs
to Docker worker by @kevingrismore in https://github.com/PrefectHQ/prefect/pull/14693
load_flow_from_entrypoint
in 2.x by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14669
PrefectDeprecationWarning
in tests by @zzstoatzz in https://github.com/PrefectHQ/prefect/pull/14708
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.19.8...2.19.9
Published by marvin-robot 3 months ago
end_time
during client-side task orchestration by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14681
IncrementRunTime
by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14683
SetExpectedStartTime
by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14684
container_create_kwargs
to Docker worker by @kevingrismore in https://github.com/PrefectHQ/prefect/pull/14686
sync_compatible
when running flows in remote environments by @desertaxle in https://github.com/PrefectHQ/prefect/pull/14660
State.state_details
client-side by @chrisguidry in https://github.com/PrefectHQ/prefect/pull/14679
TaskRunRecorder
process events into task runs/task run states by @jakekaplan in https://github.com/PrefectHQ/prefect/pull/14729
Full Changelog: https://github.com/PrefectHQ/prefect/compare/3.0.0rc11...3.0.0rc12