CI/CD With Prefect

Many organizations deploy Prefect workflows via their CI/CD process. Each organization has their own unique CI/CD setup, but a common pattern is to use CI/CD to manage Prefect deployments. Combining Prefect's deployment features with CI/CD tools enables efficient management of flow code updates, scheduling changes, and container builds. This guide uses GitHub Actions to implement a CI/CD process, but these concepts are generally applicable across many CI/CD tools.

Note that Prefect's primary ways for creating deployments, a .deploy flow method or a prefect.yaml configuration file, are both designed with building and pushing images to a Docker registry in mind.

Getting started with GitHub Actions and Prefect

In this example, you'll write a GitHub Actions workflow that will run each time you push to your repository's main branch. This workflow will build and push a Docker image containing your flow code to Docker Hub, then deploy the flow to Prefect Cloud.

Repository secrets

Your CI/CD process must be able to authenticate with Prefect in order to deploy flows.

Deploying flows securely and non-interactively in your CI/CD process can be accomplished by saving your PREFECT_API_URL and PREFECT_API_KEY as secrets in your repository's settings so they can be accessed in your CI/CD runner's environment without exposing them in any scripts or configuration files.

In this scenario, deploying flows involves building and pushing Docker images, so add DOCKER_USERNAME and DOCKER_PASSWORD as secrets to your repository as well.

You can create secrets for GitHub Actions in your repository under Settings -> Secrets and variables -> Actions -> New repository secret:

Writing a GitHub workflow

To deploy your flow via GitHub Actions, you'll need a workflow YAML file. GitHub will look for workflow YAML files in the .github/workflows/ directory in the root of your repository. In their simplest form, GitHub workflow files are made up of triggers and jobs.

The on: trigger is set to run the workflow each time a push occurs on the main branch of the repository.

The deploy job is comprised of four steps:

• Checkout clones your repository into the GitHub Actions runner so you can reference files or run scripts from your repository in later steps.
• Log in to Docker Hub authenticates to DockerHub so your image can be pushed to the Docker registry in your DockerHub account. docker/login-action is an existing GitHub action maintained by Docker. with: passes values into the Action, similar to passing parameters to a function.
• Setup Python installs your selected version of Python.
• Prefect Deploy installs the dependencies used in your flow, then deploys your flow. env: makes the PREFECT_API_KEY and PREFECT_API_URL secrets from your repository available as environment variables during this step's execution.

For reference, the examples below can be found on their respective branches of this repository.

.deployprefect.yaml

.
├── .github/
│   └── workflows/
│       └── deploy-prefect-flow.yaml
├── flow.py
└── requirements.txt

flow.py

from prefect import flow

@flow(log_prints=True)
def hello():
  print("Hello!")

if __name__ == "__main__":
    hello.deploy(
        name="my-deployment",
        work_pool_name="my-work-pool",
        image="my_registry/my_image:my_image_tag",
    )

.github/workflows/deploy-prefect-flow.yaml

name: Deploy Prefect flow

on:
  push:
    branches:
      - main

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Log in to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Prefect Deploy
        env:
          PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
          PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
        run: |
          pip install -r requirements.txt
          python flow.py

.
├── .github/
│   └── workflows/
│       └── deploy-prefect-flow.yaml
├── flow.py
├── prefect.yaml
└── requirements.txt

flow.py

from prefect import flow

@flow(log_prints=True)
def hello():
  print("Hello!")

prefect.yaml

name: cicd-example
prefect-version: 2.14.11

build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build_image
      requires: prefect-docker>=0.3.1
      image_name: my_registry/my_image
      tag: my_image_tag
      dockerfile: auto

push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.3.1
      image_name: "{{ build_image.image_name }}"
      tag: "{{ build_image.tag }}"

pull: null

deployments:
  - name: my-deployment
    entrypoint: flow.py:hello
    work_pool:
      name: my-work-pool
      work_queue_name: default
      job_variables:
        image: "{{ build-image.image }}"

.github/workflows/deploy-prefect-flow.yaml

name: Deploy Prefect flow

on:
  push:
    branches:
      - main

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Log in to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Prefect Deploy
        env:
          PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
          PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
        run: |
          pip install -r requirements.txt
          prefect deploy -n my-deployment

Running a GitHub workflow

After pushing commits to your repository, GitHub will automatically trigger a run of your workflow. The status of running and completed workflows can be monitored from the Actions tab of your repository.

You can view the logs from each workflow step as they run. The Prefect Deploy step will include output about your image build and push, and the creation/update of your deployment.

Successfully built image '***/cicd-example:latest'

Successfully pushed image '***/cicd-example:latest'

Successfully created/updated all deployments!

                Deployments
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ Name                ┃ Status  ┃ Details ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ hello/my-deployment │ applied │         │
└─────────────────────┴─────────┴─────────┘

Advanced example

In more complex scenarios, CI/CD processes often need to accommodate several additional considerations to enable a smooth development workflow:

• Making code available in different environments as it advances through stages of development
• Handling independent deployment of distinct groupings of work, as in a monorepo
• Efficiently using build time to avoid repeated work

This example repository demonstrates how each of these considerations can be addressed using a combination of Prefect's and GitHub's capabilities.

Deploying to multiple workspaces

Which deployment processes should run are automatically selected when changes are pushed depending on two conditions:

on:
  push:
    branches:
      - stg
      - main
    paths:
      - "project_1/**"

1. branches: - which branch has changed. This will ultimately select which Prefect workspace a deployment is created or updated in. In this example, changes on the stg branch will deploy flows to a staging workspace, and changes on the main branch will deploy flows to a production workspace.
2. paths: - which project folders' files have changed. Since each project folder contains its own flows, dependencies, and prefect.yaml, it represents a complete set of logic and configuration that can be deployed independently. Each project in this repository gets its own GitHub Actions workflow YAML file.

The prefect.yaml file in each project folder depends on environment variables that are dictated by the selected job in each CI/CD workflow, enabling external code storage for Prefect deployments that is clearly separated across projects and environments.

  .
  ├── cicd-example-workspaces-prod  # production bucket
  │   ├── project_1
  │   └── project_2
  └── cicd-example-workspaces-stg  # staging bucket
      ├── project_1
      └── project_2  

Since the deployments in this example use S3 for code storage, it's important that push steps place flow files in separate locations depending upon their respective environment and project so no deployment overwrites another deployment's files.

Caching build dependencies

Since building Docker images and installing Python dependencies are essential parts of the deployment process, it's useful to rely on caching to skip repeated build steps.

The setup-python action offers caching options so Python packages do not have to be downloaded on repeat workflow runs.

- name: Setup Python
  uses: actions/setup-python@v5
  with:
    python-version: "3.11"
    cache: "pip"

Using cached prefect-2.16.1-py3-none-any.whl (2.9 MB)
Using cached prefect_aws-0.4.10-py3-none-any.whl (61 kB)

The build-push-action for building Docker images also offers caching options for GitHub Actions. If you are not using GitHub, other remote cache backends are available as well.

- name: Build and push
  id: build-docker-image
  env:
      GITHUB_SHA: ${{ steps.get-commit-hash.outputs.COMMIT_HASH }}
  uses: docker/build-push-action@v5
  with:
    context: ${{ env.PROJECT_NAME }}/
    push: true
    tags: ${{ secrets.DOCKER_USERNAME }}/${{ env.PROJECT_NAME }}:${{ env.GITHUB_SHA }}-stg
    cache-from: type=gha
    cache-to: type=gha,mode=max

importing cache manifest from gha:***
DONE 0.1s

[internal] load build context
transferring context: 70B done
DONE 0.0s

[2/3] COPY requirements.txt requirements.txt
CACHED

[3/3] RUN pip install -r requirements.txt
CACHED

Prefect GitHub Actions

Prefect provides its own GitHub Actions for authentication and deployment creation. These actions can simplify deploying with CI/CD when using prefect.yaml, especially in cases where a repository contains flows that are used in multiple deployments across multiple Prefect Cloud workspaces.

Here's an example of integrating these actions into the workflow we created above:

name: Deploy Prefect flow

on:
  push:
    branches:
      - main

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Log in to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Prefect Auth
        uses: PrefectHQ/actions-prefect-auth@v1
        with:
          prefect-api-key: ${{ secrets.PREFECT_API_KEY }}
          prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }}

      - name: Run Prefect Deploy
        uses: PrefectHQ/actions-prefect-deploy@v3
        with:
          deployment-names: my-deployment
          requirements-file-paths: requirements.txt

Authenticating to other Docker image registries

The docker/login-action GitHub Action supports pushing images to a wide variety of image registries.

For example, if you are storing Docker images in AWS Elastic Container Registry, you can add your ECR registry URL to the registry key in the with: part of the action and use an AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as your username and password.

- name: Login to ECR
  uses: docker/login-action@v3
  with:
    registry: <aws-account-number>.dkr.ecr.<region>.amazonaws.com
    username: ${{ secrets.AWS_ACCESS_KEY_ID }}
    password: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Other resources

Check out the Prefect Cloud Terraform provider if you're using Terraform to manage your infrastructure.