Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
APACHE-2.0 License
Bot releases are hidden (Show)
Published by zzstoatzz 8 months ago
2.16.1
In 2.16.1
, a regression related to logging non-str
objects (when a PREFECT_API_KEY
was set) was introduced. The fix is included in 2.16.2
.
See https://github.com/PrefectHQ/prefect/pull/12151 for implementation details.
.serve
or .deploy
from prefect_dbt.cloud.jobs import trigger_dbt_cloud_job_run_and_wait_for_completion
from prefect.deployments import EntrypointType
if __name__ == "__main__":
trigger_dbt_cloud_job_run_and_wait_for_completion.deploy(
name="demo-deploy-from-module",
work_pool_name="olympic",
entrypoint_type=EntrypointType.MODULE_PATH,
job_variables={"env": { "EXTRA_PIP_PACKAGES": "prefect-dbt" } }
)
See https://github.com/PrefectHQ/prefect/pull/12134 for implementation details.
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.16.1...2.16.2
See the release notes for more!
Published by zzstoatzz 8 months ago
prefect.yaml
now supports specifying multiple schedules via the schedules
key.
This allows you to define multiple schedules for a single deployment, and each schedule can have its own cron
, interval
, or rrule
configuration:
...
schedules:
- cron: "0 0 * * *"
active: false
- interval: 3600
active: true
- rrule: "FREQ=YEARLY"
active: true
In addition you can specify multiple schedules via arguments to prefect deploy
:
prefect deploy ... --cron '4 * * * *' --cron '1 * * * *' --rrule 'FREQ=DAILY'
We've also added support for multiple schedules to flow.serve
, flow.deploy
and prefect.runner.serve
. You can provide multiple schedules by passing a list to the cron
, interval
, or rrule
arguments:
import datetime
import random
from prefect import flow
@flow
def trees():
tree = random.choice(["๐ณ", "๐ด", "๐ฒ", "๐ต"])
print(f"Here's a happy little tree: {tree}")
if __name__ == "__main__":
trees.serve(
name="trees",
interval=[3600, 7200, 14400],
)
This will create a deployment with three schedules, one that runs every hour, one that runs every two hours, and one that runs every four hours. For more advanced cases, use the schedules
argument.
trees.serve(
name="trees",
schedules=[
IntervalSchedule(interval=datetime.timedelta(minutes=30)),
{"schedule": RRuleSchedule(rrule="FREQ=YEARLY"), "active": True},
MinimalDeploymentSchedule(schedule=CronSchedule(cron="0 0 * * *"), active=False),
]
)
Dive into these new scheduling capabilities today and streamline your workflows like never before.
For implementation details, see the following pull request:
- https://github.com/PrefectHQ/prefect/pull/12107
See the release notes for more details!
Published by abrookins 8 months ago
With todayโs release, weโre excited to roll out initial support for using multiple schedules with Deployments! You can now use multiple schedules in the following ways:
Deployment
classprefect deployment schedule <create, delete, pause, resume, ls, clear>
Coming soon, weโll round out support for multiple schedules in other areas, such as:
flow.serve()
and flow.deploy()
prefect deploy
The easiest way to get started with multiple schedules is to try out the new CLI commands:
$ prefect deployment schedule ls happy-flow/my-deployment
Deployment Schedules
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโณโโโโโโโโโ
โ ID โ Schedule โ Active โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ c7d3ddc4-9a5a-4dec-bd59-eed282ae55d5 โ cron: 0 0 1 * 1 โ True โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโโ
$ prefect deployment schedule create happy-flow/my-deployment --interval 60
Created deployment schedule!
$ prefect deployment schedule ls happy-flow/my-deployment
Deployment Schedules
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโ
โ ID โ Schedule โ Active โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 3638ed58-cab2-4462-a680-2f92fcf6c797 โ interval: 0:01:00s โ True โ
โ c7d3ddc4-9a5a-4dec-bd59-eed282ae55d5 โ cron: 0 0 1 * 1 โ True โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโ
flow.deploy
and deploy
โ https://github.com/PrefectHQ/prefect/pull/12017
deployment run --help
so it renders for more args โ https://github.com/PrefectHQ/prefect/pull/11960
Flow.with_options
logic for retries, retry_delay_seconds, flow_run_name โ https://github.com/PrefectHQ/prefect/pull/12020
wait_for_flow_run
โ https://github.com/PrefectHQ/prefect/pull/11999
Task.submit()
for autonomous task submission โ https://github.com/PrefectHQ/prefect/pull/12025
Full release notes: https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md
Published by desertaxle 8 months ago
We are excited to announce that task runs are now executed on the main thread!
When feasible, task runs are now executed on the main thread instead of a worker thread. Previously, all task runs were run in a new worker thread. This allows objects to be passed to and from tasks without worrying about thread safety unless you have opted into concurrency. For example, an HTTP client or database connection can be shared between a flow and its tasks now (unless synchronous concurrency is used). Some asynchronous and sequential use cases may see performance improvements.
Consider the following example:
import sqlite3
from prefect import flow, task
db = sqlite3.connect("threads.db")
try:
db.execute("CREATE TABLE fellowship(name)")
except sqlite3.OperationalError:
pass
else:
db.commit()
db.execute("DELETE FROM fellowship")
db.commit()
cur = db.cursor()
@task
def my_task(name: str):
global db, cur
cur.execute('INSERT INTO fellowship VALUES (?)', (name,))
db.commit()
@flow
def my_flow():
global db, cur
for name in ["Frodo", "Gandalf", "Gimli", "Aragorn", "Legolas", "Boromir", "Samwise", "Pippin", "Merry"]:
my_task(name)
print(cur.execute("SELECT * FROM fellowship").fetchall())
db.close()
if __name__ == "__main__":
my_flow()
In previous versions of Prefect, running this example would result in an error like this:
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 7977619456 and this is thread id 6243151872.
But now, with task runs executing on the main thread, this example will run without error! We're excited this change makes Prefect even more intuitive and flexible!
See the following pull request for implementation details:
- https://github.com/PrefectHQ/prefect/pull/11930
You can monitor the status of a flow run created from a deployment via the CLI. This is useful for observing a flow run's progress without navigating to the UI.
To monitor a flow run started from a deployment, use the --watch
option with prefect deployment run
:
prefect deployment run --watch <slugified-flow-name>/<slugified-deployment-name>
See the following pull request for implementation details:
- https://github.com/PrefectHQ/prefect/pull/11702
starlette
version to resolve vulnerability in python-mulipart
โ https://github.com/PrefectHQ/prefect/pull/11956
RUNNING
-> RUNNING
state transitions for autonomous task runs โ https://github.com/PrefectHQ/prefect/pull/11975
PENDING
state for autonomous task execution โ https://github.com/PrefectHQ/prefect/pull/11985
run_deployment
tags parameter documentation โ https://github.com/PrefectHQ/prefect/pull/11955
All changes: https://github.com/PrefectHQ/prefect/compare/2.14.21...2.15.0
Published by zangell44 8 months ago
We're excited to unveil the new status indicators for work queues in Prefect's UI, enhancing your ability to oversee and control flow run execution within our hybrid work pools.
Work queues will now display one of three distinct statuses:
Ready
- one or more online workers are actively polling the work queueNot Ready
- no online workers are polling the work queue, signaling a need for interventionPaused
- the work queue is intentionally paused, preventing executionWith the introduction of work queue status, you'll notice the absence of deprecated work queue health indicators in the UI.
See the documentation on work queue status for more information.
For now, this is an experimental feature, and can be enabled by running:
prefect config set PREFECT_EXPERIMENTAL_ENABLE_WORK_QUEUE_STATUS=True
See the following pull request for implementation details:
- https://github.com/PrefectHQ/prefect/pull/11829
WARNING
level log indicating a task run completed successfully โ https://github.com/PrefectHQ/prefect/pull/11810
deployments
section of a prefect.yaml
file were not resolved correctly โ https://github.com/PrefectHQ/prefect/pull/11740
pool_pre_ping
to improve stability for long-lived PostgreSQL connections โ https://github.com/PrefectHQ/prefect/pull/11911
All changes: https://github.com/PrefectHQ/prefect/compare/2.14.20...2.14.21
Published by collincchoy 9 months ago
All changes: https://github.com/PrefectHQ/prefect/compare/2.14.19...2.14.20
Published by bunchesofdonald 9 months ago
You can now include dynamic, markdown-formatted descriptions when pausing or suspending a flow run for human input. This description will be shown in the Prefect UI alongside the form when a user is resuming the flow run, enabling developers to give context and instructions to users when they need to provide input.
from datetime import datetime
from prefect import flow, pause_flow_run, get_run_logger
from prefect.input import RunInput
class UserInput(RunInput):
name: str
age: int
@flow
async def greet_user():
logger = get_run_logger()
current_date = datetime.now().strftime("%B %d, %Y")
description_md = f"""
**Welcome to the User Greeting Flow!**
Today's Date: {current_date}
Please enter your details below:
- **Name**: What should we call you?
- **Age**: Just a number, nothing more.
"""
user_input = await pause_flow_run(
wait_for_input=UserInput.with_initial_data(
description=description_md, name="anonymous"
)
)
if user_input.name == "anonymous":
logger.info("Hello, stranger!")
else:
logger.info(f"Hello, {user_input.name}!")
See the following PR for implementation details:
RunInput
saving to include descriptions, improving clarity and documentation for flow inputs โ https://github.com/PrefectHQ/prefect/pull/11776
azure_storage_container
for more flexible storage options โ https://github.com/PrefectHQ/prefect/pull/11784
TaskRunFilterFlowRunId
for both client and server, enhancing task run filtering capabilities โ https://github.com/PrefectHQ/prefect/pull/11748
Published by zzstoatzz 9 months ago
This release includes:
send_input
and receive_input
โ https://github.com/PrefectHQ/prefect/pull/11724
See the release notes for details!
Published by desertaxle 9 months ago
Runner
web serverYou can now submit runs of served flows without blocking the main thread, from inside or outside a flow run. If submitting flows from inside a parent flow, these submitted runs will be tracked as subflows of the parent flow run.
In order to use this feature, you must:
Runner
webserver endpoints via
prefect config set PREFECT_EXPERIMENTAL_ENABLE_EXTRA_RUNNER_ENDPOINTS=True
Runner
web server is enabled, either by:
webserver=True
to your serve
callprefect config set PREFECT_RUNNER_SERVER_ENABLE=True
You can then submit any flow available in the import space of the served flow, and you can submit multiple runs at once. If submitting flows from a parent flow, you may optionally block the parent flow run from completing until all submitted runs are complete with wait_for_submitted_runs()
.
import time
from pydantic import BaseModel
from prefect import flow, serve, task
from prefect.runner import submit_to_runner, wait_for_submitted_runs
class Foo(BaseModel):
bar: str
baz: int
class ParentFoo(BaseModel):
foo: Foo
x: int = 42
@task
def noop():
pass
@flow(log_prints=True)
async def child(foo: Foo = Foo(bar="hello", baz=42)):
print(f"received {foo.bar} and {foo.baz}")
print("going to sleep")
noop()
time.sleep(20)
@task
def foo():
time.sleep(2)
@flow(log_prints=True)
def parent(parent_foo: ParentFoo = ParentFoo(foo=Foo(bar="hello", baz=42))):
print(f"I'm a parent and I received {parent_foo=}")
submit_to_runner(
child, [{"foo": Foo(bar="hello", baz=i)} for i in range(9)]
)
foo.submit()
wait_for_submitted_runs() # optionally block until all submitted runs are complete
if __name__ == "__main__":
# either enable the webserver via `webserver=True` or via
# `prefect config set PREFECT_RUNNER_SERVER_ENABLE=True`
serve(parent.to_deployment(__file__), limit=10, webserver=True)
This feature is experimental and subject to change. Please try it out and let us know what you think!
See the PR for implementation details.
url
to prefect.runtime.flow_run
โ https://github.com/PrefectHQ/prefect/pull/11686
/ui-settings
endpoint โ https://github.com/PrefectHQ/prefect/pull/11701
pydantic
v2 types in schema generation for flow parameters โ https://github.com/PrefectHQ/prefect/pull/11656
PENDING
to PENDING
state transitions โ https://github.com/PrefectHQ/prefect/pull/11695
cache_result_in_memory
argument for flow
decorator โ https://github.com/PrefectHQ/prefect/pull/11669
flow.from_source()
โ https://github.com/PrefectHQ/prefect/pull/11690
All changes: https://github.com/PrefectHQ/prefect/compare/2.14.16...2.14.17
Published by urimandujano 9 months ago
prefect.yaml
templatingYou can now access fields on blocks used in your prefect.yaml
files. This enables you to use values stored in blocks to provide dynamic configuration for attributes like your work_pool_name
and job_variables
.
Here's what it looks like in action:
deployments:
- name: test
version: 0.1
tags: []
description: "Example flow"
schedule: {}
entrypoint: "flow.py:example_flow"
parameters: {}
work_pool:
name: "{{ prefect.blocks.json.default-config.value.work_pool }}"
work_queue: "{{ prefect.blocks.json.default-config.value.work_queue }}"
In the above example, we use fields from a JSON
block to configure which work pool and queue we deploy our flow to. We can update where our flow is deployed to by updating the referenced block without needing to change our prefect.yaml
at all!
Many thanks to @bjarneschroeder for contributing this functionality! Check out this PR for implementation details: https://github.com/PrefectHQ/prefect/pull/10938
wait_for_flow_run
method to PrefectClient
to allow waiting for a flow run to complete โ https://github.com/PrefectHQ/prefect/pull/11305
Modal
push work pools โ https://github.com/PrefectHQ/prefect/pull/11665
limit
kwarg in serve
to increase its visibility โ https://github.com/PrefectHQ/prefect/pull/11645
RunInput
subclasses โ https://github.com/PrefectHQ/prefect/pull/11636
prefect.yaml
and support accessing block fields using keypaths โ https://github.com/PrefectHQ/prefect/pull/10938
COMPLETED
state โ https://github.com/PrefectHQ/prefect/pull/11385
from_source
โ https://github.com/PrefectHQ/prefect/pull/11634
wait_for_flow_run
โ https://github.com/PrefectHQ/prefect/pull/11668
prefect deploy
docs โ https://github.com/PrefectHQ/prefect/pull/11644
Modal
infrastructure documentation โ https://github.com/PrefectHQ/prefect/pull/11676
Published by znicholasbrown 9 months ago
UI_SERVE_BASE_URL
lead to incorrect asset urls by @znicholasbrown in https://github.com/PrefectHQ/prefect/pull/11628
Full Changelog: https://github.com/PrefectHQ/prefect/compare/2.14.14...2.14.15
Published by collincchoy 9 months ago
You can now specify a prefect.yaml
deployment configuration file while running prefect deploy
by using the
--prefect-file
command line argument. This means that your configuration files can be in any directory
and can follow your own naming conventions. Using this feature provides more flexibility in defining
and managing your deployments.
prefect.yaml
You can now toggle your deployment schedules between active
and inactive
in your prefect.yaml
configuration file. This enables you to create deployments with initially inactive schedules, allowing for thorough testing or staged rollouts!
You can now install prefect
using Python 3.12! This support is experimental and will be hardened in future releases.
A big thanks to @brett-koonce, @jitvimol, @oz-elhassid, @Zyntogz, and @Andrew-S-Rosen for making their first contributions in this release.
See the release notes for more details.
Published by serinamarie 10 months ago
This release includes a few very exciting enhancements! ๐ช
Those who run Prefect server in an environment where arbitrary outbound internet traffic is not allowed were previously unable to retrieve up-to-date default work pool configurations (via the UI or otherwise). You can now access the worker metadata needed to access the corresponding work pool configurations in your server even in such an air-gapped environment. Upon each release of prefect
, the most recent version of this worker metadata will be embedded in the prefect
package so that it can be used as a fallback if the outbound call to retrieve the real-time metadata fails.
In this release, we're excited to introduce the ability to conditionally retry tasks by passing in an argument to retry_condition_fn
in your task decorator, enabling more nuanced and flexible retry mechanisms. This adds a significant level of control and efficiency, particularly in handling complex or unpredictable task outcomes. For more information on usage, check out our docs!
We've also added the ability to open your current Prefect Cloud workspace in the browser from the CLI, display work queue status details via CLI, and much more! See the release notes for details.
A big thanks to @dominictarro and @ConstantinoSchillebeeckx for contributing enhancements to this release, as well as to @yifanmai, who made their first contribution.
Published by serinamarie 10 months ago
This release brings some exciting enhancements and fixes!
ContainerInstanceProvisioner
See the release notes for details!
Published by desertaxle 10 months ago
In the past few releases, we've added the ability to provision infrastructure for push work pools via the CLI. This release adds the ability to customize the name of the resources created in your cloud environment when provisioning infrastructure for push work pools so you can follow your organization's naming conventions.
To customize your resource names when provisioning infrastructure for a push work pool, follow the interactive prompts:
? Proceed with infrastructure provisioning with default resource names? [Use arrows to move; enter to select]
โโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ Options: โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ โ Yes, proceed with infrastructure provisioning with default resource names โ
โ > โ Customize resource names โ
โ โ Do not proceed with infrastructure provisioning โ
โโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
? Please enter a name for the resource group (prefect-aci-push-pool-rg): new-rg
? Please enter a name for the app registration (prefect-aci-push-pool-app): new-app
? Please enter a prefix for the Azure Container Registry (prefect): newregistry
? Please enter a name for the identity (used for ACR access) (prefect-acr-identity): new-identity
? Please enter a name for the ACI credentials block (new-work-pool-push-pool-credentials): new-aci-block
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Provisioning infrastructure for your work pool new-work-pool will require: โ
โ โ
โ Updates in subscription: Azure subscription 1 โ
โ โ
โ - Create a resource group in location: eastus โ
โ - Create an app registration in Azure AD: new-app โ
โ - Create/use a service principal for app registration โ
โ - Generate a secret for app registration โ
โ - Create an Azure Container Registry with prefix newregistry โ
โ - Create an identity new-identity to allow access to the created registry โ
โ - Assign Contributor role to service account โ
โ - Create an ACR registry for image hosting โ
โ - Create an identity for Azure Container Instance to allow access to the registry โ
โ โ
โ Updates in Prefect workspace โ
โ โ
โ - Create Azure Container Instance credentials block: new-aci-block โ
โ โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Proceed with infrastructure provisioning? [y/n]: y
Creating resource group
Resource group 'new-rg' created successfully
Creating app registration
App registration 'new-app' created successfully
Generating secret for app registration
Secret generated for app registration with client ID '03923189-3151-4acd-8d59-76483752cd39'
Creating ACI credentials block
ACI credentials block 'new-aci-block' created in Prefect Cloud
Assigning Contributor role to service account
Service principal created for app ID '25329389-3151-4acd-8d59-71835252cd39'
Contributor role assigned to service principal with object ID '483h4c85-4a8f-4fdb-0394-bd0f0b1202d0'
Creating Azure Container Registry
Registry created
Logged into registry newregistry1702538242q2z2.azurecr.io
Creating identity
Identity 'new-identity' created
Provisioning infrastructure. โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 100% 0:00:00
Your default Docker build namespace has been set to 'newregistry1702538242q2z2.azurecr.io'.
Use any image name to build and push to this registry by default:
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโ example_deploy_script.py โโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ from prefect import flow โ
โ from prefect.deployments import DeploymentImage โ
โ โ
โ โ
โ @flow(log_prints=True) โ
โ def my_flow(name: str = "world"): โ
โ print(f"Hello {name}! I'm a flow running on an Azure Container Instance!") โ
โ โ
โ โ
โ if __name__ == "__main__": โ
โ my_flow.deploy( โ
โ name="my-deployment", โ
โ work_pool_name="my-work-pool", โ
โ image=DeploymentImage( โ
โ name="my-image:latest", โ
โ platform="linux/amd64", โ
โ ) โ
โ ) โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Infrastructure successfully provisioned for 'new-work-pool' work pool!
Created work pool 'new-work-pool'!
Using a push work pool with automatic infrastructure provisioning is a great way to get started with a production-level Prefect set up in minutes! Check out our push work pool guide for step-by-step instructions on how to get started!
See the following pull requests for implementation details:
We've added a new date and time filter to the workspace dashboard that gives greater control over the dashboard. You can now filter by days, hours, and even minutes. You can also specify a specific date and time range to filter by. You can also go backwards and forwards in time using that time window, for example, you can scroll through by hour.
See the following pull requests for implementation details:
KubernetesJob
blocks as work pools โ https://github.com/PrefectHQ/prefect/pull/11347
is_schedule_active
to flow.deploy
and flow.serve
โ https://github.com/PrefectHQ/prefect/pull/11375
prefect.utilities.processutils.stream_text
โ https://github.com/PrefectHQ/prefect/pull/11298
value
in create_flow_run_input
โ https://github.com/PrefectHQ/prefect/pull/11421
Runner
webserver โ https://github.com/PrefectHQ/prefect/pull/11333
flow.deploy
and examples โ https://github.com/PrefectHQ/prefect/pull/11373
Guides
to How-to Guides
โ https://github.com/PrefectHQ/prefect/pull/11388
Published by urimandujano 11 months ago
We're introducing an enhancement to the Azure Container Instance push pool experience. You can now conveniently provision necessary Azure infrastructure with the --provision-infra
flag during work pool creation, automating the provisioning of various Azure resources essential for ACI push pools, including resource groups, app registrations, service accounts, and more.
To provision Azure resources when creating an ACI push pool:
โฏ prefect work-pool create my-work-pool --provision-infra --type azure-container-instance:push
? Please select which Azure subscription to use: [Use arrows to move; enter to select]
โโโโโโณโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ Name โ Subscription ID โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ โ Engineering โ 123 โ
โ > โ Azure subscription 1 โ 234 โ
โโโโโโดโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Provisioning infrastructure for your work pool my-work-pool will require: โ
โ โ
โ Updates in subscription Azure subscription 1 โ
โ โ
โ - Create a resource group in location eastus โ
โ - Create an app registration in Azure AD โ
โ - Create a service principal for app registration โ
โ - Generate a secret for app registration โ
โ - Assign Contributor role to service account โ
โ - Create Azure Container Instance โ
โ โ
โ Updates in Prefect workspace โ
โ โ
โ - Create Azure Container Instance credentials block aci-push-pool-credentials โ
โ โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Proceed with infrastructure provisioning? [y/n]: y
Creating resource group
Provisioning infrastructure... โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 0% -:--:--Resource group 'prefect-aci-push-pool-rg' created in location 'eastus'
Creating app registration
Provisioning infrastructure... โโโโโโโโโบโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 20% -:--:--App registration 'prefect-aci-push-pool-app' created successfully
Generating secret for app registration
Provisioning infrastructure... โโโโโโโโโโโโโโโโโบโโโโโโโโโโโโโโโโโโโโโโโ 40% 0:00:06Secret generated for app registration with client ID 'abc'
ACI credentials block 'aci-push-pool-credentials' created
Assigning Contributor role to service account...
Provisioning infrastructure... โโโโโโโโโโโโโโโโโโโโโโโโโบโโโโโโโโโโโโโโโ 60% 0:00:06Contributor role assigned to service principal with object ID 'xyz'
Creating Azure Container Instance
Provisioning infrastructure... โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโบโโโโโโโ 80% 0:00:04Container instance 'prefect-acipool-container' created successfully
Creating Azure Container Instance credentials block
Provisioning infrastructure... โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 100% 0:00:00
Infrastructure successfully provisioned for 'my-work-pool' work pool!
Created work pool 'my-work-pool'!
This marks a step forward in Prefect's Azure capabilities, offering you an efficient and streamlined process for leveraging Azure Container Instances to execute their workflows.
See the following pull request for implementation details:
โ https://github.com/PrefectHQ/prefect/pull/11275
provision-infra
sub-command for enhanced push work pool managementThis enhancement allows you to directly provision infrastructure for existing push work pools. Rather than recreating a work pool, you can provision necessary infrastructure and
update the existing work pool base job template with the following command:
โฏ prefect work-pool provision-infra my-work-pool
? Please select which Azure subscription to use: [Use arrows to move; enter to select]
โโโโโโณโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ Name โ Subscription ID โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ โ Engineering โ 13d โ
โ > โ Azure subscription 1 โ 6h4 โ
โโโโโโดโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Provisioning infrastructure for your work pool my-work-pool will require: โ
โ โ
โ Updates in subscription Azure subscription 1 โ
โ โ
โ - Create a resource group in location eastus โ
โ - Create an app registration in Azure AD prefect-aci-push-pool-app โ
โ - Create/use a service principal for app registration โ
โ - Generate a secret for app registration โ
โ - Assign Contributor role to service account โ
โ - Create Azure Container Instance 'aci-push-pool-container' in resource group prefect-aci-push-pool-rg โ
โ โ
โ Updates in Prefect workspace โ
โ โ
โ - Create Azure Container Instance credentials block aci-push-pool-credentials โ
โ โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Proceed with infrastructure provisioning? [y/n]: y
...
This PR bolsters support for efficient work pool management across diverse cloud environments, delivering a tool for seamless infrastructure setup.
See the following pull request for implementation details:
suspend_flow_run
method to suspend a flow run โ https://github.com/PrefectHQ/prefect/pull/11291
--provision-infra
is used to only show supported work pool types - https://github.com/PrefectHQ/prefect/pull/11350
Infrastructure
blocks as work pools โ https://github.com/PrefectHQ/prefect/pull/11180
Process
blocks as work pools โ https://github.com/PrefectHQ/prefect/pull/11346
read_logs
return type to be List[Log]
โ https://github.com/PrefectHQ/prefect/pull/11303
Paused
state โ https://github.com/PrefectHQ/prefect/pull/11284
suspend_flow_run
to flows documentation โ https://github.com/PrefectHQ/prefect/pull/11300
work queues
tag to work pools concept page โ https://github.com/PrefectHQ/prefect/pull/11320
Published by bunchesofdonald 11 months ago
๐ช Automatic infrastructure provisioning for ECS push work pools
Following the introduction of automatic project configuration for Cloud Run push pools last week, we've added the ability to automatically provision infrastructure in your AWS account and set up your Prefect workspace to support a new ECS push pool!
You can create a new ECS push work pool and provision infrastructure in your AWS account with the following command:
prefect work-pool create --type ecs:push --provision-infra my-pool
Using the --provision-infra
flag will automatically set up your default AWS account to be ready to execute flows via ECS tasks:
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Provisioning infrastructure for your work pool my-work-pool will require: โ
โ โ
โ - Creating an IAM user for managing ECS tasks: prefect-ecs-user โ
โ - Creating and attaching an IAM policy for managing ECS tasks: prefect-ecs-policy โ
โ - Storing generated AWS credentials in a block โ
โ - Creating an ECS cluster for running Prefect flows: prefect-ecs-cluster โ
โ - Creating a VPC with CIDR 172.31.0.0/16 for running ECS tasks: prefect-ecs-vpc โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Proceed with infrastructure provisioning? [y/n]: y
Provisioning IAM user
Creating IAM policy
Generating AWS credentials
Creating AWS credentials block
Provisioning ECS cluster
Provisioning VPC
Creating internet gateway
Setting up subnets
Setting up security group
Provisioning Infrastructure โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 100% 0:00:00
Infrastructure successfully provisioned!
Created work pool 'my-pool'!
If you have yet to try using an ECS push pool, now is a great time!
If you use Azure, don't fret; we will add support for Azure Container Instances push work pools in a future release!
See the following pull request for implementation details:
โ https://github.com/PrefectHQ/prefect/pull/11267
๐ Enhancements
๐ ๏ธ Fixes
.serve
crashes due to process limiter โ https://github.com/PrefectHQ/prefect/pull/11264
GitRepository
when using provider-specific git credentials blocks โ https://github.com/PrefectHQ/prefect/pull/11282
Published by zhen0 11 months ago
This release is a follow-up to 2.14.7 which never made it to PyPI because of an issue with our Github workflow.
All changes: https://github.com/PrefectHQ/prefect/compare/2.14.7...2.14.8
Published by zhen0 11 months ago
This release fixes a bug introduced in 2.14.6 where deployments with default Docker image builds looked for images tagged v2.14.6
instead of 2.14.6
. Users of 2.14.6
should upgrade if planning to create deployments with an image other than a custom image.
parent_flow_run_id
as a new API filter for flow runs โ https://github.com/PrefectHQ/prefect/pull/11089
For more details see the release notes.
Published by zzstoatzz 11 months ago
This release includes:
๐ View the next run for a deployment at a glance
๐ง Automatic project configuration for Cloud Run push work pools
๐ Updated Guides
๐งโ๐คโ๐ง Contributors
See the the release notes for more details!