Releases: PrefectHQ/prefect
Release 3.0.0rc2
We're excited to announce the release candidate of Prefect 3.0. It's the most flexible, powerful, fastest version of Prefect yet. Prefect 3.0 includes several exciting new features. Install it by running pip install prefect==3.0.0rc2
and check out the docs here.
Run tasks independently of flows
You can now run and serve tasks outside of flows and inside of other tasks.
from prefect import task
@task
def my_background_task(name: str):
print(f"Hello, {name}!")
if __name__ == "__main__":
my_background_task.delay("ford")
Transactional semantics
Use rollback and commit hooks to facilitate idempotent python code.
from prefect import flow, task
from prefect.transactions import transaction
@task
def first_task():
print('first')
@first_task.on_rollback
def roll(txn):
print('rolling back')
@task
def second_task():
raise RuntimeError("oopsie")
@flow
def txn_flow():
with transaction():
first_task()
second_task()
if __name__ == "__main__":
txn_flow()
Open source Events and Automations
Trigger actions, such as sending notifications, pausing schedules, starting flow runs and more in response to Prefect events.
More flexible variables and new artifact types
Variables can now be any JSON compatible type including dicts, lists, and integers. Progress and Image artifacts make it easy to add visual annotations to your flow run graph.
Faster and richer CLI
Improved CLI speed and several added commands and conveniences.
Updated navigation, styling, and interaction design
The new Runs page displays both flow and task run information, and an improved sidebar and switcher makes navigating Prefect simpler than ever.
Changes since 3.0.0rc1
Enhancements
- Adds
deferred
kwarg toTask.map
to executed mapped tasks on a task server by @desertaxle in #13756 - Add pydantic version to cli
version
command by @pleek91 in #13715 - conditionally log retryable errors by @jakekaplan in #13773
- Update
Variable.set
return value + docs by @jakekaplan in #13557 - Add new flow runs pagination endpoint by @pleek91 in #13808
- Updates
PrefectDistributedFuture
to use an event websocket to determine task completion by @desertaxle in #13763 - Enhancement: Artifacts documentation by @dylanbhughes in #13724
- Migrate
prefect-kubernetes
to pydantic 2 by @zzstoatzz in #13687 - Add endpoint for bulk task run counts for flow runs by @pleek91 in #13759
- Copy pared down
prefect-redis
to integrations by @bunchesofdonald in #13758 - Cache key policies by @cicdw in #13791
- Refactor task run engine by @jlowin in #13793
- Cache Policy Upgrades by @cicdw in #13821
- Put
AwsCredentials
first to match default factory by @bunchesofdonald in #13819
Fixes
- Fix
load_flow_argument_from_entrypoint
to work with async flows by @elisalimli in #13716 - Add handling for positional only and keyword only arguments when parsing a function signature from source code by @desertaxle in #13774
- Load assignments when safeloading a namespace by @desertaxle in #13775
- Coerce work pool env values to
str
by @zzstoatzz in #13782 - Support
retry_delay_seconds
in task engine by @zzstoatzz in #13815 - Rename
RedisFilesystem
toRedisStorageContainer
to match naming semantics of other filesystem classes by @bunchesofdonald in #13771
Documentation
- Fixes over 500 broken links in docs by @discdiver in #13734
- Remove
agents
concept page and refs for 3.0 by @serinamarie in #13631 - WIP 3.0rc docs new IA (on Mintlify) by @LarryUllman in #13783
- Fixes a doc build from an admonition change and note managed execution not in beta. by @discdiver in #13789
- Update version instruction install docs page by @discdiver in #13795
- Move integration package docs out of versioning by @discdiver in #13760
- Update button text at top of page by @discdiver in #13797
- Update get-started/install page by @LarryUllman in #13822
- Porting over task guide, mdx bugfixes by @WillRaphaelson in #13754
New Contributors
- @elisalimli made their first contribution in #13716
- @LarryUllman made their first contribution in #13783
Full Changelog: 3.0.0rc1...3.0.0rc2
2.19.4
Fixes
- Fix
Could not find flow '[...]' in '[...].py'
errors when usingprefect deploy
to deploy async flows - #13769 - Fix parameter schema generation for flows with positional only and keyword only arguments when using
prefect deploy
- #13778 - Fix use of dynamic models in flow typing when using
prefect deploy
- #13781 - Allow clients to provide task run ID during creation to enable compatibility between 2.x servers and 3.0.0rc clients - #13683
Documentation
- Add Docker deployment steps page to centralized integration docs - #13720
All changes: 2.19.3...2.19.4
Release 3.0.0rc1
Release 3.0.0rc1
We're excited to announce the release candidate of Prefect 3.0. It's the most flexible, powerful, fastest version of Prefect yet. Prefect 3.0 includes several exciting new features. Install it by running pip install prefect==3.0.0rc1
and check out the docs here.
Run tasks independently of flows
You can now run and serve tasks outside of flows and inside of other tasks.
from prefect import task
@task
def my_background_task(name: str):
print(f"Hello, {name}!")
if __name__ == "__main__":
my_background_task.delay("ford")
Transactional semantics
Use rollback and commit hooks to facilitate idempotent python code.
from prefect import flow, task
from prefect.transactions import transaction
@task
def first_task():
print('first')
@first_task.on_rollback
def roll(txn):
print('rolling back')
@task
def second_task():
raise RuntimeError("oopsie")
@flow
def txn_flow():
with transaction():
first_task()
second_task()
if __name__ == "__main__":
txn_flow()
Open source Events and Automations
Trigger actions, such as sending notifications, pausing schedules, starting flow runs and more in response to Prefect events.
More flexible variables and new artifact types
Variables can now be any JSON compatible type including dicts, lists, and integers. Progress and Image artifacts make it easy to add visual annotations to your flow run graph.
Faster and richer CLI
Improved CLI speed and several added commands and conveniences.
Updated navigation, styling, and interaction design
The new Runs page displays both flow and task run information, and an improved sidebar and switcher makes navigating Prefect simpler than ever.
Enhancements
- Create artifact for unsuccessful dbt task runs — #13348
- Add filter on
task_run.expected_start_time
— #13491 - Add utilities to serialize context to a dictionary and hydrate context from a dictionary — #13529
- Add API endpoints for deployment count and next flow run — #13544
- Allow flow parameter schema generation when dependencies are missing — #13315
- Change the default value for
enforce_parameter_schema
fromFalse
toTrue
— #13594 - Migrate schemas to pydantic v2 — #13574
- Removes block auto-instrumentation — #13407
- Migrate all uses of the banned characters validation to a self-validator — #13370
- Ignore and warn on unrecognized settings - #13624
Fixes
- Remove unnecessary flow run infrastructure override access checks — #13401
- Enforce False case when flow run id is null — #13464
- Fix workspace variable hydration to allow for JSON data — #13548
- Remove unused settings/experimental work pool flags:
PREFECT_EXPERIMENTAL_ENABLE_WORK_POOLS
andPREFECT_EXPERIMENTAL_WARN_WORK_POOLS
— #13144 - Pin
pydantic>=2.7
forSecret
— #13613 - Skip on cancellation hooks if runner can't load flow — #13660
- Refactor lazy imports to avoid accidental eager imports — #13296
- Allow block registration to use client schemas for server model creation — #13602
- Replace our customized
Duration
types with plaintimedelta
s — #13603
Experimental
- Add
prefect.yaml
and cli support for new schedule fields — #13318
Documentation
- Transition documentation hosting from Netlify to Mintlify — #13634
- Add Python 3.12 to list of Docker images — #13321
- Update
index.md
— #13353 - Improve tutorial section — #13297
- Fix jinja template in automations doc — #13422
- Update development section docs — #13247
- Update Ray integration docs — #13467
- Update Variables docs to include JSON types — #13493
- Update quickstart guide for usability — #13562
- Remove
deployments-block-based
concept page and refs for 3.0 — #13626 - Remove
infrastructure
concept page and refs for 3.0 — #13629 - Update docs image paths and remove outdated images — #13666
- Remove references to
prefect.software
from docs — #13382 - Update
host.md
— #13351 - Simplify rate limits page — #13689
- Removing references to deprecated block types and add disclaimer — #13651
- Update guides — #13253
- Remove
storage
concept page and refs - #13630
Integrations
- Migrate
prefect-dbt
to pydantic v2 - #13718 - Migrate
prefect-email
to pydantic v2 — #13654 - Migrate
prefect-slack
to pydantic v2 — #13673 - Migrate
prefect-shell
to pydantic v2 — #13675 - Migrate
prefect-gcp
to pydantic v2 — #13650 - Migrate
prefect-github
to pydantic v2 — #13655 - Migrate
prefect-gitlab
to pydantic v2 — #13656 - Migrate
prefect-docker
to pydantic v2 - #13697 - Migrate
prefect-sqlalchemy
to pydantic v2 - #13700 - Add
PrefectDistributedClient
toprefect-dask
— #13537 - Update
RayTaskRunner
for compatibility with new engine — #13575 - Update
DaskTaskRunner
for compatibility with the updated engine — #13555 - prefect-dbt artifact consolidation and markdown fixes — #13379
- prefect-dbt - Cause unsuccessful dbt tasks to fail — #13405
- DBT Tasks extra_command_args Fix — #13308
- Update dbt-core dependency — #13394
Breaking Changes
- Remove
prefect deployment build
CLI frommain
— #13366 - Remove
prefect agent
CLI frommain
— #13365 - Remove
prefect deployment apply
CLI frommain
— #13367 - Remove
PrefectAgent
class — #13374 - Remove
prefect.software
— #13375 - Remove
deployments
module — #13373 - Remove
EcsTask
frommain
— #13417 - Remove
AzureContainerInstanceJob
frommain
— #13418 - Remove
VertexAICustomTrainingJob
frommain
— #13419 - Remove
CloudRunJob
frommain
— #13420 - Remove infrastructure blocks from
main
— #13424 - Remove
Infrastructure
,BlockWorker
frommain
— #13430 - Remove deprecated storage blocks from
main
— #13410 - Remove
prefect-agent
as a possible work pool type — #13444 - Remove old engine — #13542
- Remove Python 3.8 support — #13331
- Remove
deprecated
module and its references — #13345 - Remove old task runners and futures modules — #13593
- Remove
is_state
— #13569 - Remove deprecated options from
prefect work-queue
and refs to agents - #13638
Contributors
All changes: 2.19.0...3.0.0rc1
See release notes for details.
Release 2.19.3
New method for generating parameter schemas without dependencies
prefect deploy
now works even when dependencies are missing from the current environment. This can speed up deployment via CI by removing the need to install dependencies before deploying your flows.
See the following pull requests for implementation details:
Enhancements
- Provide URL in CLI output upon work pool creation — #13597
Fixes
- Ensure graceful cancellation of flow runs corresponding to deleted deployments — #13669
Integrations
- Add loading state to concurrency limits table in the Prefect UI — PrefectHQ/prefect-ui-library#2483
- Remove old schema properties from deployments in the Prefect UI — PrefectHQ/prefect-ui-library#2482
- Add handling for multi-word dbt CLI commands — #13616
All changes: 2.19.2...2.19.3
Release 2.19.2
This release includes a few bug fixes, ensuring:
- 🛠️ runs created from the
deployments/{id}/create_flow_run
endpoint hydrates workspace variables - 🔢 proper integer value display on the Variables page of the Prefect UI — PrefectHQ/prefect-ui-library#2454
- ⚙️ "Run a deployment" automation action parameter display input configuration persists during editing — PrefectHQ/prefect-ui-library#2458
- 📌
requests
dependency pinned to<2.32.0
inrequirements-dev.txt
- #13538 - 📄 Jinja template example is renderable in automations documentation — #13421
2.19.1
Release 2.19.1
Enhancements to prefect-dbt
for running dbt-core
commands
Introducing prefect-dbt
summary artifacts! With summary artifacts, you get a view of all of the runs that succeeded, failed, or were skipped as well as where the failed models live in your dbt project and why they failed. This exposes information about each dbt node in a succinct format that teams can use for development, debugging, insights, and so much more.
These changes are available in prefect-dbt
version 0.5.0!
See the following pull requests for implementation details:
Enhancements
- Enable flow run infra overrides section in the Prefect UI - PrefectHQ/prefect-ui-library#2417
Experimental
Events and Automations
- Add ability to create and manage automations in the Prefect UI - #13342
Documentation
- Update documentation formatting for
prefect-ray
- #13385
Integrations
- Update all integrations libraries to pin
prefect<3.0.0
- #13408
Integration library releases
This release was accompanied by the following integration library releases:
prefect-aws
- 0.4.17prefect-azure
- 0.3.10prefect-bitbucket
- 0.2.5prefect-dask
- 0.2.9prefect-databricks
- 0.2.7prefect-dbt
- 0.5.0prefect-docker
- 0.5.2prefect-email
- 0.3.5prefect-gcp
- 0.5.12prefect-github
- 0.2.5prefect-gitlab
- 0.2.5prefect-kubernetes
- 0.3.10prefect-ray
- 0.3.6prefect-shell
- 0.2.5prefect-slack
- 0.2.6prefect-snowflake
- 0.27.6prefect-sqlalchemy
- 0.4.3
All changes: 2.19.0...2.19.1
Release 2.19.0
✨ This release includes a number of enhancements and fixes!
Support for major infrastructure and distributed task integrations
As prefect-dask
and other integrations have been added to the prefect codebase, this release adds these integrations as extra requirements of the prefect package, making it easier to install support for everything in your Prefect stack:
pip install 'prefect[dask]'
We loved this community contribution so much, we did it for all our first-party integrations:
pip install 'prefect[aws,kubernetes,dask,dbt,sqlalchemy,slack]'
You can see the full list of Prefect's extra requirements in our setup.py.
Support for timeout seconds in global concurrency context manager
You may want to fail immediately if a global concurrency slot is unavailable. Rather than block and wait, you can now specify a timeout_seconds
argument in the global concurrency context manager and catch a TimeoutError
if a slot is not available within the specified time.
@flow
def fail_immediately_flow():
try:
with concurrency("there-can-be-only-one", occupy=1, timeout_seconds=0.1):
do_something_resource_intensive()
except TimeoutError:
return Cancelled(message="Another flow run is already running")
Manage global concurrency limits via the CLI
Global concurrency limits let you control how many operations can run simultaneously-- now you can create, read, edit, and delete global concurrency limits via the Prefect CLI!
To create a new concurrency limit, use the prefect gcl create
command. You must specify a --limit
argument, and can optionally specify a --slot-decay-per-second
and --disable
argument.
prefect gcl create my-concurrency-limit --limit 5 --slot-decay-per-second 1.0
You can inspect the details of a concurrency limit using the prefect gcl inspect
command:
prefect gcl inspect my-concurrency-limit
To update a concurrency limit, use the prefect gcl update
command. You can update the --limit
, --slot-decay-per-second
, --enable
, and --disable
arguments:
prefect gcl update my-concurrency-limit --limit 10
We also have many more bug fixes and in-flight work! See the release notes for details!
Release 2.18.3
Experimental
Engine
- Wire up new engine to deployment runs — #12914
Fixes
- Fix parameters becoming unresponsive and disappearing in Prefect UI — PrefectHQ/prefect-ui-library#2355
All changes: 2.18.2...2.18.3
Release 2.18.2
💡 Providing a deployment name to flow.serve
is now optional
When running flow.serve
, you can now omit the deployment name. If you do not provide a deployment name, the deployment name will default to the name of the flow. This change makes it easier to run flows without needing to specify a deployment name each time:
@flow
def etl_flow():
pass
if __name__ == "__main__":
etl_flow.serve()
results in:
Your flow 'etl-flow' is being served and polling for scheduled runs!
To trigger a run for this flow, use the following command:
$ prefect deployment run 'etl-flow/etl-flow'
🛠✨ We've also released a few important fixes to our deployment parameter form when creating a run in the Prefect UI! 🧑🎨
🚀 This release also includes a number of other fixes and in-flight feature work. See the release notes for more details!
Release 2.18.1
Release 2.18.1
Fixes
- Fix improper context access for nested async task outside of flow — #12810
- Fix using default interval schedule in
prefect deploy
— #12833 - Handle case in
validationUpdate
schema where definitions are falsy — #12880 - Allow
prefect cloud login
to override current workspace — #12867 - Remove extra quotes in
prefect deployment run --watch
— #12894
Experimental
Events and Automations
- Support filtering by automation name:
- Add support for using the "normal" Trigger classes for
flow.serve
and.deploy
— #12789 - Add an account-level event subscriber — #12808
- Emit flow run state change events — #12825
- Emit deployment status persistence and events — #12853
- Enable event streaming from
PrefectCloudEventSubscriber
via CLI — #12796 - Update the
prefect automation delete
CLI — #12876
Engine
- Add new experimental engine for tasks and flows with improved readability and extensibility — #12856
Documentation
- Improve installation instructions — #12783
- Improve quickstart — #12798
- Migrate
prefect-azure
docs to Integrations section of the Prefect docs — #12794 - Update storage guide credentials blocks — #12819
- Remove
server
import recommendations — #12823 - Remove link to removed API page — #12824
- Add Azure Container Instances worker guide — #12846
- Improve wording on integrations index page — #12852
Prefect UI Library
- Add
FormattedDate
component to display accessible, long-form timestamps consistently - Update modal buttons and add auto-close to the parameters and job variable modals — PrefectHQ/prefect-ui-library#2320
- Add flow run list information density — PrefectHQ/prefect-ui-library#2321
- Fix "Run a deployment" action not populating the default parameters from the deployment — PrefectHQ/prefect-ui-library#2322
- Fix schema form properties with no default value from defaulting to
null
(None
) — PrefectHQ/prefect-ui-library#2323 - Update date-fns and date-fns-tz — PrefectHQ/prefect-ui-library#2319
- Use correct icon colors for non-destructive actions in the UI — PrefectHQ/prefect-ui-library#2328
Integrations
Prefect CGP
- Remove API ref to nonexistent Google Cloud Run V2 page — PrefectHQ/prefect-gcp#260
- Fix VPC access for Cloud v2 worker — PrefectHQ/prefect-gcp#266
- Handle case where
vpc
isn't in job template — PrefectHQ/prefect-gcp#267
New Contributors
- @keizobabybear made their first contribution in #12852