Upgrade from Agents to Workers¶
Upgrading from agents to workers significantly enhances the experience of deploying flows. It simplifies the specification of each flow's infrastructure and runtime environment.
A worker is the fusion of an agent with an infrastructure block. Like agents, workers poll a work pool for flow runs that are scheduled to start. Like infrastructure blocks, workers are typed - they work with only one kind of infrastructure and they specify the default configuration for jobs submitted to that infrastructure.
Accordingly, workers are not a drop-in replacement for agents. Using workers requires deploying flows differently. In particular, deploying a flow with a worker does not involve specifying an infrastructure block. Instead, infrastructure configuration is specified on the work pool and passed to each worker that polls work from that pool.
This guide provides an overview of the differences between agents and workers. It also describes how to upgrade from agents to workers in just a few quick steps.
Agents to workers¶
- Improved visibility into the status of each worker, including when a worker was started and when it last polled.
- Better handling of race conditions for high availability use cases.
Agent infrastructure blocks to work pools¶
- Work pools expose a base job template that enables an unprecedented level of customization and governance over workers in that pool.
- Push work pools allow for flow execution without the need to host a worker at all.
prefect.yaml file for managing multiple deployments¶
- More easily define multiple deployments at once through a
- Prefect provides deployment actions that allow you to automatically build images for your flows.
- Templating enables dryer deployment definitions.
- You can use a deployment creation wizard now! 🧙
Command to build deployments:
prefect deployment build <entrypoint>-->
Prefect will now automatically detect flows in your repo and provide a wizard 🧙 to guide you through setting required attributes for your deployments.
Configuring remote flow code storage:
storage blocks --> pull action
And you can still use an existing storage block as your pull action! The more general pull steps specification allows for explicit customization of the job that defines each run (for example, by adding an additional step that sets a working directory).
Configuring flow run infrastructure:
run-infrastructure blocks --> typed work pool
Default infra config is now set on the typed work pool, and can be overwritten by both individual deployments or individual runs.
Managing multiple deployments:
Create and/or update many deployments at once through a
- Storage blocks can be set as the pull action in a
- Infrastructure blocks have similar configuration fields as typed work pools.
Deployment-level infra-overrides operate in much the same way.
The process for starting an agent and starting a worker in your environment are virtually identical.
prefect agent start --pool <work pool name>-->
prefect worker start --pool <work pool name>
If you use infrastructure-as-code
Notice the command to start a worker is very similar to the command to start an agent. If you previously used terraform, a helm chart, or other infrastructure-as-code methods to start an agent, you should be able to continue using it for a worker, provided all uses of
agentare changed to
How to upgrade quickly¶
If you have existing deployments that use infrastructure blocks, you can quickly upgrade them to be compatible with workers by following these steps:
Create a work pool of the same type as the infrastructure block you are currently using.
- Any work pool infrastructure type other than
Prefect Agentwill work.
- Referencing the configuration you've set on the infrastructure block, set similar flow run infrastructure configuration on the work pool.
- Any work pool infrastructure type other than
Start a worker to poll this work pool. You should see the command to start the worker as soon as you save your new work pool.
prefect worker start -p <work pool name>
Deploy your flow guided by the deployment creation wizard:
prefect deploycommands from the root level of your repo!
With agents you might have had multiple
deployment.yamlfiles, but under worker deployment patterns, each repo will have a single
prefect.yamlfile located at the root of the repo that contains deployment configuration for all flows in that repo.
For step 4, select
yon last prompt to save the configuration for the deployment.
Saving the configuration for your deployment will result in a
prefect.yamlfile populated with your first deployment. You can use this YAML file to edit and define multiple deployments for this repo.
prefect.yamlfile, configure a pull action referencing whatever configuration you used as your storage block.
- Continue to create more deployments that use workers by either adding them to the
deploymentslist in the
prefect.yamlfile and/or by continuing to use the deployment creation wizard.