When a deployment runs, the execution environment needs access to the flow code. Flow code is not stored in Prefect server or Prefect Cloud.

You have several flow code storage options for remote execution:

  • Git-based storage (GitHub, GitLab, Bitbucket)
  • Docker image-based storage
  • Cloud-provider storage (AWS S3, Azure Blob Storage, GCP GCS)

Each of these options is popular - your choice will depend upon your team’s needs and tools.

Local storage is also an option for deployments that run locally.

In the examples below, we show how to create a work pool-based deployment for each of these storage options.

​
Deployment creation options

You can create a deployment through Python code with the flow.deploy method or through a YAML specification defined in a prefect.yaml file.

When using the Python deploy method specify a flow storage location other than a Docker image requires the flow.from_source method. The source and entrypoint arguments are required.

The entrypoint is the path to the file the flow is located in and the function name, separated by a colon.

The source is either a URL to a git repository or a storage object.

To create a prefect.yaml file interactively, run prefect deploy from the CLI and select the appropriate storage option.

​
Git-based storage

Git-based version control platforms provide redundancy, version control, and collaboration capabilities. Prefect supports:

For a public repository, you can use the repository URL directly.

If you are using a private repository and are authenticated in your environment at deployment creation and deployment execution, you can use the repository URL directly.

Alternatively, for a private repository, you can create a Secret block or create a credentials block specific to your git-based version control platform to store your credentials. Then you can reference the block in the Python deploy method or the prefect.yaml file pull step.

If using the Python deploy method with a private repository that references a block, provide a GitRepository object instead of a URL, as shown below.

For accessing a private repository, we suggest creating a personal access token. We recommend using HTTPS with fine-grained Personal Access Tokens to limit access by repository.

See the GitHub docs for Personal Access Tokens (PATs).

Under Your Profile -> Developer Settings -> Personal access tokens -> Fine-grained token choose Generate New Token and fill in the required fields.

Under Repository access choose Only select repositories and grant the token permissions for Contents.

If using a Secret block, you can create it through code or the UI ahead of time and reference it at deployment creation as shown above.

If using a credentials block, you can create it ahead of time and reference it at deployment creation.

  1. Install prefect-github with pip install -U prefect-github
  2. Register the blocks in that library to make them available on the server with prefect block register -m prefect_github
  3. Create a GitHub Credentials block through code or the Prefect UI and reference it at deployment creation as shown above.

Note that you can specify a branch if creating a GitRepository object. The default is "main".

Push your code

When you make a change to your code, Prefect does not push your code to your git-based version control platform. You need to push your code manually or as part of your CI/CD pipeline. This is intentional to avoid confusion about the git history and push process.

​
Docker-based storage

Another popular flow code storage option is to include it in a Docker image. All work pool options except Process and Prefect Managed allow you to bake your code into a Docker image.

To create a deployment with Docker-based flow code storage use the Python deploy method or create a prefect.yaml file.

If you use the Python deploy method to store the flow code in a Docker image, you don’t need to use the from_source method.

The prefect.yaml file below was generated by running prefect deploy from the CLI (a few lines of metadata were excluded from the top of the file output for brevity).

Note that the build section is necessary if baking your flow code into a Docker image.

Any pip packages specified in a requirements.txt file will be included in the image.

In the examples above, we elected not to push the image to a remote registry.

To push the image to a remote registry, pass push=True in the Python deploy method or add a push_docker_image step to the push section of the prefect.yaml file.

​
Custom Docker image

By default, your deployment will use the base Prefect image when creating your image. Alternatively, you can create a custom Docker image outside of Prefect. If doing this with and you don’t need push or pull steps in the prefect.yaml file. Instead, the work pool can reference the image directly.

For more information, see this discussion of custom Docker images.

​
Cloud-provider storage

Another option for flow code storage is any fsspec-supported storage location, such as AWS S3, Azure Blob Storage, or GCP GCS.

If the storage location is publicly available, or if you are authenticated in the environment where you are creating and running your deployment, you can reference the storage location directly. You don’t need to pass credentials explicitly.

To pass credentials explicitly to authenticate to your storage location, you can use either of the following block types:

  • Prefect integration library storage blocks, such as the prefect-aws library’s S3Bucket block, which can use a AWSCredentials block when it is created.
  • Secret blocks

If you use a storage block such as the S3Bucket block, you need to have the prefect-aws library available in the environment where your flow code runs.

You can do any of the following to make the library available:

  1. Install the library into the execution environment directly
  2. Specify the library in the work pool’s Base Job Template in the Environment Variables section like this:{"EXTRA_PIP_PACKAGES":"prefect-aws"}
  3. Specify the library in the environment variables of the deploy method as shown in the examples below
  4. Specify the library in a requirements.txt file and reference the file in the pull step of the prefect.yaml file like this:
    - prefect.deployments.steps.pip_install_requirements:
        directory: "{{ pull_code.directory }}" 
        requirements_file: requirements.txt

The examples below show how to create a deployment with flow code in a cloud provider storage location. For each example, we show how to access code that is publicly available. The prefect.yaml example includes an additional line to reference a credentials block if authenticating to a private storage location through that option.

We also include Python code that shows how to use an existing storage block and an example of that creates, but doesn’t save, a storage block that references an existing nested credentials block.

To create an AwsCredentials block:

  1. Install the prefect-aws library with pip install -U prefect-aws
  2. Register the blocks in prefect-aws with prefect block register -m prefect_aws
  3. Create a user with a role with read and write permissions to access the bucket. If using the UI, create an access key pair with IAM -> Users -> Security credentials -> Access keys -> Create access key. Choose Use case -> Other and then copy the Access key and Secret access key values.
  4. Create an AWSCredentials block in code or the Prefect UI. In addition to the block name, most users will fill in the AWS Access Key ID and AWS Access Key Secret fields.
  5. Reference the block as shown above.

Another authentication option is to give the worker access to the storage location at runtime through SSH keys.

​
Store code locally

If using a Process work pool, you can use one of the remote code storage options shown above, or you can store your flow code in a local folder.

Here is an example of how to create a deployment with flow code stored locally:

​
Include or exclude files from storage

By default, Prefect includes all files in the current folder when you create a deployment.

When using a git repository, Docker image, or cloud-provider storage location, you may want to exclude certain files or directories. If you are familiar with Docker you are likely familiar with the .dockerignore file. For remote storage, the .prefectignore file serves the same purpose and follows a similar syntax. For example, an entry of *.pyc will exclude all .pyc files from upload.

​
Update flow code

After creating a deployment, you may need to change your flow code. If baking your flow code into a Docker image, you will need to rebuild your image. If storing your flow code in a git-based version control platform or a cloud-based storage location, often you can update your flow code without rebuilding your deployment.

The exception is when something the server needs to know about has changed, such as the flow entrypoint parameters.

Rerun the Python script with deploy or run prefect deploy from the CLI for YAML-based deployments to update your deployment with the new flow code.

​
Flow code storage for deployments created with serve

The Python serve method creates a deployment and a local long-running process to poll for flow runs at the same time.

The deployment creation mechanics for serve are similar to deploy. deployjust requires a work pool name and has a number of parameters dealing with flow code storage for Docker images.

Unlike serve, if you don’t specify an image to use for your flow, you must specify where to pull the flow code from at runtime with the from_source method; from_source is optional with serve.

Read more about when to consider using serve here.

Was this page helpful?

Learn about workersDeploy flows with Python