ecs
typed work pool are executed as ECS tasks. Only Prefect ECS workers can poll an ecs
typed work pool.
When the ECS worker receives a scheduled flow run from the ECS work pool it is polling, it spins up the specified infrastructure on AWS ECS. The worker knows to build an ECS task definition for each flow run based on the configuration specified in the work pool.
Once the flow run completes, the ECS containers of the cluster are spun down to a single container that continues to run the Prefect worker. This worker continues polling for work from the Prefect work pool.
If you specify a task definition ARN (Amazon Resource Name) in the work pool, the worker will use that ARN when spinning up the ECS Task, rather than creating a task definition from the fields supplied in the work pool configuration.
You can use either EC2 or Fargate as the capacity provider. Fargate simplifies initiation, but lengthens infrastructure setup time for each flow run. Using EC2 for the ECS cluster can reduce setup time. In this example, we will show how to use Fargate.
taskExecutionRole
. To create an IAM role for the ECS task using the AWS CLI, follow these steps:
ecs-trust-policy.json
:
aws iam create-role
command to create the roles that you will be using. For this guide, the ecsTaskExecutionRole
will be used by the worker to start ECS tasks, and will also be the role assigned to the ECS tasks running your Prefect flows.
AmazonECSTaskExecutionRolePolicy
that grants the permissions necessary for starting ECS tasks. See here for other common execution role permissions. Attach this policy to your task execution role:
--role-name
and --policy-arn
with the actual role name and policy Amazon Resource Name (ARN) you want to use.
prefect worker start --pool my-ecs-pool
.
Create a JSON file with the following contents:
prefect config view
to view the PREFECT_API_URL
for your current Prefect profile. Use this to replace <prefect-api-url>
.
PREFECT_API_KEY
:
PREFECT_API_KEY
in a self-hosted Prefect server. You should either:
PREFECT_API_KEY
environment variable from your task definition (if your server has no authentication)PREFECT_API_AUTH_STRING
containing your basic auth credentials (if your server uses basic authentication)<ecs-task-role-arn>
with the ARN of the IAM role you created in Step 2. You can grab this by running:register-task-definition
command in the AWS CLI. Here is an example:
task-definition.json
with the name of your JSON file.
<ecs-cluster>
with the name of your ECS cluster.<task-definition-arn>
with the ARN of the task definition you just registered.<subnet-ids>
with a comma-separated list of your VPC subnet IDs. Ensure that these subnets are aligned with the vpc specified on the work pool in step 1. You can view subnet ids with the following command:
aws ec2 describe-subnets --filter Name=<vpc-id>
<security-group-ids>
with a comma-separated list of your VPC security group IDs.my_flow.py
prefect.yaml
fileprefect.yaml
file with the following specification:
CPU
?Launch Type
speed up your flow run execution?