- Create a push work pool that sends work to Amazon Elastic Container Service (AWS ECS), Azure Container Instances (ACI), Google Cloud Run, Modal, or Coiled
- Deploy a flow to that work pool
- Execute a flow without having to run a worker process to poll for flow runs
You can automatically provision infrastructure and create your push work pool using the
prefect work-pool create
CLI command with the --provision-infra
flag.
This approach greatly simplifies the setup process.Automatic infrastructure provisioning
With Perfect Cloud you can provision infrastructure for use with an AWS ECS, Google Cloud Run, ACI push work pool. Push work pools in Prefect Cloud simplify the setup and management of the infrastructure necessary to run your flows. However, setting up infrastructure on your cloud provider can still be a time-consuming process. Prefect dramatically simplifies this process by automatically provisioning the necessary infrastructure for you. We’ll use theprefect work-pool create
CLI command with the --provision-infra
flag to automatically provision your
serverless cloud resources and set up your Prefect workspace to use a new push pool.
Prerequisites
To use automatic infrastructure provisioning, you need:- the relevant cloud CLI library installed
- to be authenticated with your cloud provider
Install the AWS CLI,
authenticate with your AWS account,
and set a default region.If you already have the AWS CLI installed, be sure to update to the latest version.You will need the following permissions in your authenticated AWS account:IAM Permissions:
- iam:CreatePolicy
- iam:GetPolicy
- iam:ListPolicies
- iam:CreateUser
- iam:GetUser
- iam:AttachUserPolicy
- iam:CreateRole
- iam:GetRole
- iam:AttachRolePolicy
- iam:ListRoles
- iam:PassRole
- ecs:CreateCluster
- ecs:DescribeClusters
- ec2:CreateVpc
- ec2:DescribeVpcs
- ec2:CreateInternetGateway
- ec2:AttachInternetGateway
- ec2:CreateRouteTable
- ec2:CreateRoute
- ec2:CreateSecurityGroup
- ec2:DescribeSubnets
- ec2:CreateSubnet
- ec2:DescribeAvailabilityZones
- ec2:AuthorizeSecurityGroupIngress
- ec2:AuthorizeSecurityGroupEgress
- ecr:CreateRepository
- ecr:DescribeRepositories
- ecr:GetAuthorizationToken
- AmazonECS_FullAccess
- AmazonEC2FullAccess
- IAMFullAccess
- AmazonEC2ContainerRegistryFullAccess
Automatically create a new push work pool and provision infrastructure
To create a new push work pool and configure the necessary infrastructure, run this command for your particular cloud provider:--provision-infra
flag automatically sets up your default AWS account to execute
flows with ECS tasks.
In your AWS account, this command creates a new IAM user, IAM policy, and
ECS cluster that uses AWS Fargate, VPC, and ECR repository (if they don’t already exist).
In your Prefect workspace, this command creates an
AWSCredentials
block for storing the generated credentials.Here’s an abbreviated example output from running the command:Default Docker build namespaceAfter infrastructure provisioning completes, you will be logged into your new ECR repository and the default
Docker build namespace will be set to the URL of the registry.
example_deploy_script.py
<ecr-registry-url>/my-image:latest
and push it to the registry.Your image name needs to match the name of the repository created with your work pool. You can create
new repositories in the ECR console.Use existing resources with automatic infrastructure provisioning
If you already have the necessary infrastructure set up, Prefect detects that at work pool creation and the infrastructure provisioning for that resource is skipped. For example, here’s howprefect work-pool create my-work-pool --provision-infra
looks when existing Azure resources are detected:
Provision infrastructure for an existing push work pool
If you already have a push work pool set up, but haven’t configured the necessary infrastructure, you can use theprovision-infra
sub-command to provision the infrastructure for that work pool.
For example, you can run the following command if you have a work pool named “my-work-pool”.
my-work-pool
work pool and provides you with a summary
of the changes:
Manual infrastructure provisioning
If you prefer to set up your infrastructure manually, exclude the--provision-infra
flag in the CLI command.
In the examples below, you’ll create a push work pool through the Prefect Cloud UI.
To push work to ECS, AWS credentials are required.Create a user and attach the AmazonECS_FullAccess permissions.From that user’s page, create credentials and store them somewhere safe for use in the next section.
Work pool configuration
The push work pool stores information about what type of infrastructure the flow will run on, what default values to provide to compute jobs, and other important execution environment parameters. Because the push work pool needs to integrate securely with your serverless infrastructure, you need to store your credentials in Prefect Cloud by making a block.Create a Credentials block
Navigate to the blocks page, click create new block, and select AWS Credentials for the type.For use in a push work pool, set the region, access key, and access key secret.Provide any other optional information and create your block.
Create a push work pool
Now navigate to the work pools page. Click Create to configure your push work pool by selecting a push option in the infrastructure type step.Each step has several optional fields that are detailed in the
work pools documentation.
Select the block you created under the AWS Credentials field.
This allows Prefect Cloud to securely interact with your ECS cluster.
Deployment
You need to configure your deployment to send flow runs to your push work pool. For example, if you create a deployment through the interactive command line experience, choose the work pool you just created. If you are deploying an existingprefect.yaml
file, the deployment would contain:
my-push-pool
work pool ensures that runs that are ready for execution are
submitted immediately—without the need for a worker to poll for them.
Serverless infrastructure may require a certain image architectureServerless infrastructure may assume a certain Docker image architecture; for example,
Google Cloud Run will fail to run images built with
linux/arm64
architecture. If using Prefect to build your image,
you can change the image architecture through the platform
keyword (for example, platform="linux/amd64"
).