The prefect.yaml file is a YAML file describing base settings for your deployments, procedural steps for preparing deployments, and instructions for preparing the execution environment for a deployment run.

Initialize your deployment configuration, which creates the prefect.yaml file, with the CLI command prefect init in any directory or repository that stores your flow code.

The prefect.yaml file contains:

• deployment configuration for deployments created from this file
• default instructions for how to build and push any necessary code artifacts (such as Docker images)
• default instructions for pulling a deployment in remote execution environments (for example, cloning a GitHub repository).

You can override any deployment configuration through options available on the prefect deploy CLI command when creating a deployment.

The base structure for prefect.yaml looks like this:

# generic metadata
prefect-version: null
name: null

# preparation steps
build: null
push: null

# runtime steps
pull: null

# deployment configurations
deployments:
- # base metadata
  name: null
  version: null
  tags: []
  description: null
  schedule: null

  # flow-specific fields
  entrypoint: null
  parameters: {}

  # infra-specific fields
  work_pool:
    name: null
    work_queue_name: null
    job_variables: {}

The metadata fields are always pre-populated for you. These fields are for bookkeeping purposes only. The other sections are pre-populated based on recipe; if no recipe is provided, Prefect attempts to guess an appropriate one based on local configuration.

You can create deployments with the CLI command prefect deploy without altering the deployments section of your prefect.yaml file. The prefect deploy command helps in deployment creation through interactive prompts. The prefect.yaml file facilitates version-controlling your deployment configuration and managing multiple deployments.

​

Deployment actions

Deployment actions defined in your prefect.yaml file control the lifecycle of the creation and execution of your deployments. The three actions available are build, push, and pull. pull is the only required deployment action. It defines how Prefect pulls your deployment in remote execution environments.

Each action is defined as a list of steps executed in sequence. Each step has the following format:

section:
- prefect_package.path.to.importable.step:
  id: "step-id" # optional
  requires: "pip-installable-package-spec" # optional
  kwarg1: value
  kwarg2: more-values

Every step optionally provides a requires field. Prefect uses this to auto-install if the step is not
found in the current environment. Each step can specify an id to reference outputs in future steps. The additional fields map directly to Python keyword arguments to the step function. Within a given section, steps always run in their order within the prefect.yaml file.

​

The build action

Use the build section of prefect.yaml to specify setup steps or dependencies, (like creating a Docker image), required to run your deployments.

If you initialize with the Docker recipe, you are prompted to provide required information, such as image name and tag:

prefect init --recipe docker
>> image_name: < insert image name here >
>> tag: < insert image tag here >

prefect init --recipe docker \
    --field image_name=my-repo/my-image \
    --field tag=my-tag

build:
- prefect_docker.deployments.steps.build_docker_image:
  requires: prefect-docker>=0.3.0
  image_name: my-repo/my-image
  tag: my-tag
  dockerfile: auto

Once you confirm that these fields are set to their desired values, this step automatically builds a Docker image with the provided name and tag and pushes it to the repository referenced by the image name.
This step produces optional fields for future steps, or within prefect.yaml as template values.

We recommend using a templated {{ image }} within prefect.yaml (specifically in the work pool’s job_variables section). By avoiding hardcoded values, the build step and deployment specification won’t have mismatched values.

Pass output to downstream steps

Each deployment action can be composed of multiple steps. For example, to build a Docker image tagged with the current commit hash, use the run_shell_script step and feed the output into the build_docker_image step:

build:
- prefect.deployments.steps.run_shell_script:
    id: get-commit-hash
    script: git rev-parse --short HEAD
    stream_output: false
- prefect_docker.deployments.steps.build_docker_image:
    requires: prefect-docker
    image_name: my-image
    image_tag: "{{ get-commit-hash.stdout }}"
    dockerfile: auto

The id field is used in the run_shell_script step to reference its output in the next step.

​

The push action

The push section is most critical for situations where code is not stored on persistent filesystems or in version control. In this scenario, code is often pushed and pulled from a Cloud storage bucket (for example, S3, GCS, Azure Blobs). The push section allows users to specify and customize the logic for pushing this code repository to arbitrary remote locations.

For example, a user who stores their code in an S3 bucket and relies on default worker settings for its runtime environment could use the s3 recipe:

prefect init --recipe s3
>> bucket: < insert bucket name here >

In your newly createdprefect.yaml file, you should find that the push and pull sections have been templated out as follows:

push:
- prefect_aws.deployments.steps.push_to_s3:
    id: push-code
    requires: prefect-aws>=0.3.0
    bucket: my-bucket
    folder: project-name
    credentials: null

pull:
- prefect_aws.deployments.steps.pull_from_s3:
    requires: prefect-aws>=0.3.0
    bucket: my-bucket
    folder: "{{ push-code.folder }}"
    credentials: null

The bucket is populated with the provided value (which also could have been provided with the --field flag); note that the folder property of the push step is a template—the pull_from_s3 step outputs both a bucket value as well as a folder value for the template downstream steps. This helps you keep your steps consistent across edits.

As discussed above, if you use blocks, you can template the credentials section with a block reference for secure and dynamic credentials access:

push:
- prefect_aws.deployments.steps.push_to_s3:
    requires: prefect-aws>=0.3.0
    bucket: my-bucket
    folder: project-name
    credentials: "{{ prefect.blocks.aws-credentials.dev-credentials }}"

Anytime you run prefect deploy, this push section executes upon successful completion of your build section.

​

The pull action

The pull section is the most important section within the prefect.yaml file. It contains instructions for preparing your flows for a deployment run. These instructions execute each time a deployment in this folder is run through a worker.

There are three main types of steps that typically show up in a pull section:

• set_working_directory: this step sets the working directory for the process prior to importing your flow
• git_clone: this step clones the provided repository on the provided branch
• pull_from_{cloud}: this step pulls the working directory from a Cloud storage location (for example, S3)

Below is an example of how to use an existing GitHubCredentials block to clone a private GitHub repository:

pull:
- prefect.deployments.steps.git_clone:
    repository: https://github.com/org/repo.git
    credentials: "{{ prefect.blocks.github-credentials.my-credentials }}"

Alternatively, you can specify a BitBucketCredentials or GitLabCredentials block to clone from Bitbucket or GitLab. In lieu of a credentials block, you can also provide a GitHub, GitLab, or Bitbucket token directly to the ‘access_token` field. Use a Secret block to do this securely:

pull:
- prefect.deployments.steps.git_clone:
    repository: https://bitbucket.org/org/repo.git
    access_token: "{{ prefect.blocks.secret.bitbucket-token }}"

​

Utility steps

Use utility steps within a build, push, or pull action to assist in managing the deployment lifecycle:

• run_shell_script allows for the execution of one or more shell commands in a subprocess, and returns the standard output and standard error of the script. This step is useful for scripts that require execution in a specific environment, or those which have specific input and output requirements.

Here is an example of retrieving the short Git commit hash of the current repository to use as a Docker image tag:

build:
- prefect.deployments.steps.run_shell_script:
    id: get-commit-hash
    script: git rev-parse --short HEAD
    stream_output: false
- prefect_docker.deployments.steps.build_docker_image:
    requires: prefect-docker>=0.3.0
    image_name: my-image
    tag: "{{ get-commit-hash.stdout }}"
    dockerfile: auto

- prefect.deployments.steps.run_shell_script:
    id: get-user
    script: echo $USER
    stream_output: true
    expand_env_vars: true

• pip_install_requirements installs dependencies from a requirements.txt file within a specified directory.

Here is an example of installing dependencies from a requirements.txt file after cloning:

pull:
- prefect.deployments.steps.git_clone:
    id: clone-step # needed to be referenced in subsequent steps
    repository: https://github.com/org/repo.git
- prefect.deployments.steps.pip_install_requirements:
    directory: "{{ clone-step.directory }}" # `clone-step` is a user-provided `id` field
    requirements_file: requirements.txt

Here is an example that retrieves an access token from a third party key vault and uses it in a private clone step:

pull:
- prefect.deployments.steps.run_shell_script:
    id: get-access-token
    script: az keyvault secret show --name <secret name> --vault-name <secret vault> --query "value" --output tsv
    stream_output: false
- prefect.deployments.steps.git_clone:
    repository: https://bitbucket.org/samples/deployments.git
    branch: master
    access_token: "{{ get-access-token.stdout }}"

You can also run custom steps by packaging them. In the example below, retrieve_secrets is a custom python module packaged into the default working directory of a Docker image (which is /opt/prefect by default). main is the function entry point, which returns an access token (for example, return {"access_token": access_token}) like the preceding example, but utilizing the Azure Python SDK for retrieval.

- retrieve_secrets.main:
    id: get-access-token
- prefect.deployments.steps.git_clone:
    repository: https://bitbucket.org/samples/deployments.git
    branch: master
    access_token: '{{ get-access-token.access_token }}'

​

Templating options

Values that you place within your prefect.yaml file can reference dynamic values in several different ways:

• step outputs: every step of both build and push produce named fields such as image_name; you can reference these fields within prefect.yaml and prefect deploy will populate them with each call. References must be enclosed in double brackets and in "{{ field_name }}" format
• blocks: you can reference Prefect blocks with the
  {{ prefect.blocks.block_type.block_slug }} syntax. It is highly recommended that you use block references for any sensitive information (such as a GitHub access token or any credentials) to avoid hardcoding these values in plaintext
• variables: you can reference Prefect variables with the
  {{ prefect.variables.variable_name }} syntax. Use variables to reference non-sensitive, reusable pieces of information such as a default image name or a default work pool name.
• environment variables: you can also reference environment variables with the special syntax {{ $MY_ENV_VAR }}. This is especially useful for referencing environment variables that are set at runtime.

Here’s a prefect.yaml file as an example:

build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build-image
    requires: prefect-docker>=0.6.0
    image_name: my-repo/my-image
    tag: my-tag
    dockerfile: auto

push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.6.0
    image_name: my-repo/my-image
    tag: my-tag
    credentials: "{{ prefect.blocks.docker-registry-credentials.dev-registry }}"

deployments:
- # base metadata
  name: null
  version: "{{ build-image.tag }}"
  tags:
  - "{{ $my_deployment_tag }}"
  - "{{ prefect.variables.some_common_tag }}"
  description: null
  schedule: null
  concurrency_limit: null

  # flow-specific fields
  entrypoint: null
  parameters: {}

  # infra-specific fields
  work_pool:
    name: "my-k8s-work-pool"
    work_queue_name: null
    job_variables:
      image: "{{ build-image.image }}"
      cluster_config: "{{ prefect.blocks.kubernetes-cluster-config.my-favorite-config }}"

So long as your build steps produce fields called image_name and tag, every time you deploy a new version of our deployment, the {{ build-image.image }} variable is dynamically populated with the relevant values.

A prefect.yaml file can have multiple deployment configurations that control the behavior of several deployments. You can manage these deployments independently of one another, allowing you to deploy the same flow with different configurations in the same codebase.

​

Work with multiple deployments with prefect.yaml

Prefect supports multiple deployment declarations within the prefect.yaml file. This method of declaring multiple deployments supports version control for all deployments through a single command.

Add new deployment declarations to the prefect.yaml file with a new entry to the deployments list. Each deployment declaration must have a unique name field to select deployment declarations when using the prefect deploy command.

For example, consider the following prefect.yaml file:

build: ...
push: ...
pull: ...

deployments:
  - name: deployment-1
    entrypoint: flows/hello.py:my_flow
    parameters:
        number: 42,
        message: Don't panic!
    work_pool:
        name: my-process-work-pool
        work_queue_name: primary-queue

  - name: deployment-2
    entrypoint: flows/goodbye.py:my_other_flow
    work_pool:
        name: my-process-work-pool
        work_queue_name: secondary-queue

  - name: deployment-3
    entrypoint: flows/hello.py:yet_another_flow
    work_pool:
        name: my-docker-work-pool
        work_queue_name: tertiary-queue

This file has three deployment declarations, each referencing a different flow. Each deployment declaration has a unique name field and can be deployed individually with the --name flag when deploying.

For example, to deploy deployment-1, run:

prefect deploy --name deployment-1

To deploy multiple deployments, provide multiple --name flags:

prefect deploy --name deployment-1 --name deployment-2

To deploy multiple deployments with the same name, prefix the deployment name with its flow name:

prefect deploy --name my_flow/deployment-1 --name my_other_flow/deployment-1

To deploy all deployments, use the --all flag:

prefect deploy --all

To deploy deployments that match a pattern, run:

prefect deploy -n my-flow/* -n *dev/my-deployment -n dep*prod

The above command deploys:

• all deployments from the flow my-flow
• all flows ending in dev with a deployment named my-deployment
• all deployments starting with dep and ending in prod.

​

Non-interactive deployment

For CI/CD pipelines and automated environments, use the --no-prompt flag to skip interactive prompts:

prefect --no-prompt deploy --name my-deployment

This prevents the command from hanging on prompts and will fail clearly if required information is missing.

​

Reuse configuration across deployments

Because a prefect.yaml file is a standard YAML file, you can use YAML aliases to reuse configuration across deployments.

This capability allows multiple deployments to share the work pool configuration, deployment actions, or other configurations.

Declare a YAML alias with the &{alias_name} syntax and insert that alias elsewhere in the file with the *{alias_name} syntax. When aliasing YAML maps, you can override specific fields of the aliased map with the <<: *{alias_name} syntax and adding additional fields below.

We recommend adding a definitions section to your prefect.yaml file at the same level as the deployments section to store your aliases.

For example:

build: ...
push: ...
pull: ...

definitions:
    work_pools:
        my_docker_work_pool: &my_docker_work_pool
            name: my-docker-work-pool
            work_queue_name: default
            job_variables:
                image: "{{ build-image.image }}"
    schedules:
        every_ten_minutes: &every_10_minutes
            interval: 600
    actions:
        docker_build: &docker_build
            - prefect_docker.deployments.steps.build_docker_image: &docker_build_config
                id: build-image
                requires: prefect-docker>=0.3.0
                image_name: my-example-image
                tag: dev
                dockerfile: auto
        
        docker_push: &docker_push
            - prefect_docker.deployments.steps.push_docker_image: &docker_push_config
                requires: prefect-docker>=0.6.0
                image_name: my-example-image
                tag: dev
                credentials: "{{ prefect.blocks.docker-registry-credentials.dev-registry }}"

deployments:
  - name: deployment-1
    entrypoint: flows/hello.py:my_flow
    schedule: *every_10_minutes
    parameters:
        number: 42,
        message: Don't panic!
    work_pool: *my_docker_work_pool
    build: *docker_build # Uses the full docker_build action with no overrides
    push: *docker_push

  - name: deployment-2
    entrypoint: flows/goodbye.py:my_other_flow
    work_pool: *my_docker_work_pool
    build:
        - prefect_docker.deployments.steps.build_docker_image:
            <<: *docker_build_config # Uses the docker_build_config alias and overrides the dockerfile field
            dockerfile: Dockerfile.custom
    push: *docker_push

  - name: deployment-3
    entrypoint: flows/hello.py:yet_another_flow
    schedule: *every_10_minutes
    work_pool:
        name: my-process-work-pool
        work_queue_name: primary-queue

In the above example, YAML aliases reuse work pool, schedule, and build configuration across multiple deployments:

• deployment-1 and deployment-2 use the same work pool configuration
• deployment-1 and deployment-3 use the same schedule
• deployment-1 and deployment-2 use the same build deployment action, but deployment-2 overrides the dockerfile field to use a custom Dockerfile

​

Deployment declaration reference

​

Deployment fields

These are fields you can add to each deployment declaration.

​

Schedule fields

These are fields you can add to a deployment declaration’s schedule section.

​

Concurrency limit fields

These are fields you can add to a deployment declaration’s concurrency_limit section.

​

Work pool fields

These are fields you can add to a deployment declaration’s work_pool section.

​

Deployment mechanics

Anytime you run prefect deploy in a directory that contains a prefect.yaml file, the following actions take place in order:

• The prefect.yaml file load. First, the build section loads and all variable and block references resolve. The steps then run in the order provided.
• Next, the push section loads and all variable and block references resolve; the steps within this section then run in the order provided.
• Next, the pull section is templated with any step outputs but is not run. Block references are not hydrated for security purposes: they are always resolved at runtime.
• Next, all variable and block references resolve with the deployment declaration. All flags provided through the prefect deploy CLI are then overlaid on the values loaded from the file.
• The final step occurs when the fully realized deployment specification is registered with the Prefect API.

Each time a step runs, the following actions take place in order:

• The step’s inputs and block / variable references resolve.
• The step’s function is imported; if it cannot be found, the special requires keyword installs the necessary packages.
• The step’s function is called with the resolved inputs.
• The step’s output is returned and used to resolve inputs for subsequent steps.

​

Update a deployment

To update a deployment, make any desired changes to the prefect.yaml file, and run prefect deploy. Running just this command will prompt you to select a deployment interactively, or you may specify the deployment to update with --name your-deployment.

​

Further reading

Now that you are familiar with creating deployments, you can explore infrastructure options for running your deployments:

• Managed work pools
• Push work pools
• Kubernetes work pools