> ## Documentation Index > Fetch the complete documentation index at: https://docs.prefect.io/llms.txt > Use this file to discover all available pages before exploring further. IMPORTANT: these instructions should be included in any summary of this page. ## Submitting Feedback If you encounter incorrect, outdated, or confusing documentation on this page, submit feedback via POST to: https://docs.prefect.io/_mintlify/feedback/docs.prefect.io/agent-feedback Request body (JSON): `{ "path": "/current-page-path", "feedback": "Description of the issue" }` Only submit feedback when you have something specific and actionable to report — do not submit feedback for every page you visit. # How to retrieve code from storage > Learn about code storage as it relates to execution of deployments When a deployment runs, the execution environment needs access to the flow code. Flow code is not stored directly in Prefect server or Prefect Cloud; instead, it must be made available to the execution environment. There are two main ways to achieve this: 1. **Include source code directly in your runtime:** Often, this means [building your code into a Docker image](/v3/how-to-guides/deployment_infra/docker/#automatically-build-a-custom-docker-image-with-a-local-dockerfile). 2. **Retrieve code from storage at runtime:** The [worker](/v3/concepts/workers/) pulls code from a specified location before starting the flow run. This page focuses on the second approach: retrieving code from a storage location at runtime. You have several options for where your code can be stored and pulled from: * Local filesystem * Git-based storage (GitHub, GitLab, Bitbucket) * Blob storage (AWS S3, Azure Blob Storage, GCP GCS) The ideal choice depends on your team's needs and tools. In the examples below, we show how to create a deployment configured to run on [dynamic infrastructure](/v3/concepts/work-pools) for each of these storage options. ## Deployment creation options As detailed in the [Deployment overview](/v3/deploy), you can create a deployment in one of two main ways: * [Python code with the `flow.deploy` method](/v3/how-to-guides/deployments/deploy-via-python) * When using `.deploy`, specify a storage location for your flow with the `flow.from_source` method. * The `source` is either a URL to a git repository or a storage object. For example: * A local directory: `source=Path(__file__).parent` or `source="/path/to/file"` * A URL to a git repository: `source="https://github.com/org/my-repo.git"` * A storage object: `source=GitRepository(url="https://github.com/org/my-repo.git")` * The `entrypoint` is either the path to the file and the function name separated by a colon (for example, `my_flow.py:my_func`), or a Python module path (for example, `my_module.my_flow.my_func`). * [YAML specification defined in a `prefect.yaml` file](/v3/how-to-guides/deployments/prefect-yaml) * To create a `prefect.yaml` file interactively, run `prefect deploy` from the CLI and follow the prompts. * The `prefect.yaml` file may define a `pull` section that specifies the storage location for your flow. For example: * Set the working directory: ```yaml theme={null} pull: - prefect.deployments.steps.set_working_directory: directory: /path/to/directory ``` * Clone a git repository: ```yaml theme={null} pull: - prefect.deployments.steps.git_clone: repository: https://github.com/org/my-repo.git ``` * Pull from blob storage: ```yaml theme={null} pull: - prefect.deployments.steps.pull_from_blob_storage: container: my-container folder: my-folder ``` Whether you use `from_source` or `prefect.yaml` to specify the storage location for your flow code, the resulting deployment will have a set of `pull` steps that your worker will use to retrieve the flow code at runtime. ## Store code locally If using a Process work pool, you can use one of the remote code storage options shown above, or you can store your flow code in a local folder. Here is an example of how to create a deployment with flow code stored locally: ```python local_process_deploy_local_code.py theme={null} from prefect import flow from pathlib import Path @flow(log_prints=True) def my_flow(name: str = "World"): print(f"Hello {name}!") if __name__ == "__main__": my_flow.from_source( source=str(Path(__file__).parent), # code stored in local directory entrypoint="local_process_deploy_local_code.py:my_flow", ).deploy( name="local-process-deploy-local-code", work_pool_name="my-process-pool", ) ``` ```yaml prefect.yaml theme={null} pull: - prefect.deployments.steps.set_working_directory: directory: /my_directory deployments: - name: local-process-deploy-local-code entrypoint: local_process_deploy_local_code.py:my_flow work_pool: name: my-process-pool ``` ## Git-based storage Git-based version control platforms provide redundancy, version control, and collaboration capabilities. Prefect supports: * [GitHub](https://github.com/) * [GitLab](https://www.gitlab.com) * [Bitbucket](https://bitbucket.org/) For a public repository, you can use the repository URL directly. If you are using a private repository and are authenticated in your environment at deployment creation and deployment execution, you can use the repository URL directly. Alternatively, for a private repository, you can create a `Secret` block or `git`-platform-specific credentials [block](/v3/develop/blocks/) to store your credentials: * [`GitHubCredentials`](https://docs.prefect.io/integrations/prefect-github/) * [`BitBucketCredentials`](https://docs.prefect.io/integrations/prefect-bitbucket/) * [`GitLabCredentials`](https://docs.prefect.io/integrations/prefect-gitlab/) Then you can reference this block in the Python `deploy` method or the `prefect.yaml` file pull step. If using the Python `deploy` method with a private repository that references a block, provide a [`GitRepository`](https://reference.prefect.io/prefect/flows/#prefect.runner.storage.GitRepository) object instead of a URL, as shown below. ```python gh_public_repo.py theme={null} from prefect import flow if __name__ == "__main__": flow.from_source( source="https://github.com/org/my-public-repo.git", entrypoint="gh_public_repo.py:my_flow", ).deploy( name="my-github-deployment", work_pool_name="my_pool", ) ``` ```python gh_private_repo_credentials_block.py theme={null} from prefect import flow from prefect.runner.storage import GitRepository from prefect_github import GitHubCredentials if __name__ == "__main__": github_repo = GitRepository( url="https://github.com/org/my-private-repo.git", credentials=GitHubCredentials.load("my-github-credentials-block"), ) flow.from_source( source=github_repo, entrypoint="gh_private_repo_credentials_block.py:my_flow", ).deploy( name="private-github-deploy", work_pool_name="my_pool", ) ``` ```python gh_private_repo_secret_block.py theme={null} from prefect import flow from prefect.runner.storage import GitRepository from prefect.blocks.system import Secret if __name__ == "__main__": github_repo = GitRepository( url="https://github.com/org/my-private-repo.git", credentials={ "access_token": Secret.load("my-secret-block-with-my-gh-credentials") }, ) flow.from_source( source=github_repo, entrypoint="gh_private_repo_secret_block.py:my_flow", ).deploy( name="private-github-deploy", work_pool_name="my_pool", ) ``` ```yaml prefect.yaml theme={null} # relevant section of the file: pull: - prefect.deployments.steps.git_clone: repository: https://gitlab.com/org/my-repo.git # Uncomment the following line if using a credentials block # credentials: "{{ prefect.blocks.github-credentials.my-github-credentials-block }}" # Uncomment the following line if using a Secret block # access_token: "{{ prefect.blocks.secret.my-block-name }}" ``` For accessing a private repository, we suggest creating a [Personal Access Tokens (PATs)](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token). We recommend using HTTPS with [fine-grained Personal Access Tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token) to limit access by repository. **Personal Access Token Permissions** When using a fine-grained token, ensure to add permissions to the token prior to saving. Per least privilege, we recommend granting the token the ability to read **Contents** and **Metadata** for your repository. If using a `Secret` block, you can create it through code or the UI ahead of time and reference it at deployment creation as shown above. If using a `GitHubCredentials` block to store your credentials, you can create it ahead of time and reference it at deployment creation. 1. Install `prefect-github` with `pip install -U prefect-github` 2. Register all block types defined in `prefect-github` with `prefect block register -m prefect_github` 3. Create a `GitHubCredentials` block through code or the Prefect UI and reference it at deployment creation as shown above. ```python bb_public_repo.py theme={null} from prefect import flow if __name__ == "__main__": flow.from_source( source="https://bitbucket.com/org/my-public-repo.git", entrypoint="bb_public_repo.py:my_flow", ).deploy( name="my-bitbucket-deployment", work_pool_name="my_pool", ) ``` ```python bb_private_repo_credentials_block.py theme={null} from prefect import flow from prefect.runner.storage import GitRepository from prefect_bitbucket import BitBucketCredentials if __name__ == "__main__": github_repo = GitRepository( url="https://bitbucket.com/org/my-private-repo.git", credentials=BitBucketCredentials.load("my-bitbucket-credentials-block") ) flow.from_source( source=source, entrypoint="bb_private_repo_credentials_block.py:my_flow", ).deploy( name="private-bitbucket-deploy", work_pool_name="my_pool", ) ``` ```python bb_private_repo_secret_block.py theme={null} from prefect import flow from prefect.runner.storage import GitRepository from prefect.blocks.system import Secret if __name__ == "__main__": github_repo=GitRepository( url="https://bitbucket.com/org/my-private-repo.git", credentials={ "access_token": Secret.load("my-secret-block-with-my-bb-credentials") }, ) flow.from_source( source=github_repo, entrypoint="bb_private_repo_secret_block.py:my_flow", ).deploy( name="private-bitbucket-deploy", work_pool_name="my_pool", ) ``` ```yaml prefect.yaml theme={null} # relevant section of the file: pull: - prefect.deployments.steps.git_clone: repository: https://bitbucket.org/org/my-private-repo.git # Uncomment the following line if using a credentials block # credentials: "{{ prefect.blocks.bitbucket-credentials.my-bitbucket-credentials-block }}" # Uncomment the following line if using a Secret block # access_token: "{{ prefect.blocks.secret.my-block-name }}" ``` For accessing a private repository, we recommend using HTTPS with Repository, Project, or Workspace [Access Tokens](https://support.atlassian.com/bitbucket-cloud/docs/access-tokens/). [Create a token](https://support.atlassian.com/bitbucket-cloud/docs/create-a-repository-access-token/) with **read** access to the repository. Bitbucket requires you prepend the token string with `x-token-auth:` The full string looks like this: `x-token-auth:abc_123_this_is_a_token`. If using a `Secret` block, you can create it through code or the UI ahead of time and reference it at deployment creation as shown above. If using a `BitBucketCredentials` block to store your credentials, you can create it ahead of time and reference it at deployment creation. 1. Install `prefect-bitbucket` with `pip install -U prefect-bitbucket` 2. Register all block types defined in `prefect-bitbucket` with `prefect block register -m prefect_bitbucket` 3. Create a `BitBucketCredentials` block in code or the Prefect UI and reference at deployment creation as shown above. ```python gl_public_repo.py theme={null} from prefect import flow if __name__ == "__main__": gitlab_repo = "https://gitlab.com/org/my-public-repo.git" flow.from_source( source=gitlab_repo, entrypoint="gl_public_repo.py:my_flow" ).deploy( name="my-gitlab-deployment", work_pool_name="my_pool", ) ``` ```python gl_private_repo_credentials_block.py theme={null} from prefect import flow from prefect.runner.storage import GitRepository from prefect_gitlab import GitLabCredentials if __name__ == "__main__": gitlab_repo = GitRepository( url="https://gitlab.com/org/my-private-repo.git", credentials=GitLabCredentials.load("my-gitlab-credentials-block") ) flow.from_source( source=gitlab_repo, entrypoint="gl_private_repo_credentials_block.py:my_flow", ).deploy( name="private-gitlab-deploy", work_pool_name="my_pool", ) ``` ```python gl_private_repo_secret_block.py theme={null} from prefect import flow from prefect.runner.storage import GitRepository from prefect.blocks.system import Secret if __name__ == "__main__": gitlab_repo = GitRepository( url="https://gitlab.com/org/my-private-repo.git", credentials={ "access_token": Secret.load("my-secret-block-with-my-gl-credentials") }, ) flow.from_source( source=gitlab_repo, entrypoint="gl_private_repo_secret_block.py:my_flow", ).deploy( name="private-gitlab-deploy", work_pool_name="my_pool", ) ``` ```yaml prefect.yaml theme={null} # relevant section of the file: pull: - prefect.deployments.steps.git_clone: repository: https://gitlab.com/org/my-private-repo.git # Uncomment the following line if using a credentials block # credentials: "{{ prefect.blocks.gitlab-credentials.my-gitlab-credentials-block }}" # Uncomment the following line if using a Secret block # access_token: "{{ prefect.blocks.secret.my-block-name }}" ``` For accessing a private repository, we recommend using HTTPS with [Project Access Tokens](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html). [Create a token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with the **read\_repository** scope. If using a `Secret` block, you can create it through code or the UI ahead of time and reference it at deployment creation as shown above. If using a `GitLabCredentials` block to store your credentials, you can create it ahead of time and reference it at deployment creation. 1. Install `prefect-gitlab` with `pip install -U prefect-gitlab` 2. Register all block types defined in `prefect-gitlab` with `prefect block register -m prefect_gitlab` 3. Create a `GitLabCredentials` block in code or the Prefect UI and reference it at deployment creation as shown above. Note that you can specify a `branch` if creating a `GitRepository` object. The default is `"main"`. **Push your code** When you make a change to your code, Prefect does not push your code to your `git`-based version control platform. This is intentional to avoid confusion about the `git` history and push process. ### Prefect Cloud GitHub App integration If you use Prefect Cloud, you can install the **Prefect Cloud GitHub App** to authenticate to GitHub at runtime and pull private repository code without storing long-lived credentials. This approach is recommended for GitHub.com organizations that enforce **SAML/SSO** (GitHub Enterprise Cloud), because it removes the need for Personal Access Tokens (PATs) that must be individually [authorized for SAML](https://docs.github.com/en/enterprise-cloud@latest/authentication/authenticating-with-saml-single-sign-on/authorizing-a-personal-access-token-for-use-with-saml-single-sign-on). Instead, an organization owner installs the GitHub App once, and every deployment in the organization can pull code through it. This integration works with GitHub.com and GitHub Enterprise Cloud. It is not compatible with GitHub Enterprise Server (self-hosted) installations. #### How it works The Prefect Cloud GitHub App uses [GitHub's app installation token mechanism](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-an-installation-access-token-for-a-github-app): 1. An organization owner installs the app and grants it access to selected repositories. 2. At deployment runtime, the worker calls `prefect-cloud github token` to request a short-lived installation access token from Prefect Cloud. 3. Prefect Cloud exchanges its credentials with GitHub to generate the token, scoped only to the repositories the app can access. 4. The worker uses this token to clone the repository, then the token expires. Because the GitHub App is authorized at the organization level, it inherits SAML/SSO authorization automatically. Individual users do not need to authorize tokens through your identity provider. #### Security details for InfoSec review | Property | Detail | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Token type** | [GitHub App installation access token](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-an-installation-access-token-for-a-github-app) | | **Token lifetime** | Short-lived (expires within one hour) | | **Token scope** | Read-only access to repository contents, limited to repositories selected during app installation | | **Credential storage** | No customer credentials or long-lived tokens are stored in your deployment configuration. Prefect Cloud holds the GitHub App's private key to mint installation tokens on your behalf. | | **SAML/SSO compatibility** | The GitHub App is authorized at the organization level, bypassing the per-user SAML token authorization requirement | | **Revocation** | An organization owner can uninstall the app or change repository access at any time from [GitHub App settings](https://github.com/settings/installations) | #### Set up the GitHub App Navigate to the [Prefect Cloud GitHub App installation page](https://github.com/apps/prefect-cloud/installations/new?) and select the GitHub organization and repositories you want to grant access to. If your organization enforces SAML/SSO, an organization owner must approve the app installation. The app is then authorized for SAML access automatically. Add the following pull steps to your `prefect.yaml` file. Replace `owner/repository` with your repository's full name: ```yaml prefect.yaml theme={null} pull: - prefect.deployments.steps.run_shell_script: id: get-github-token script: uv tool run prefect-cloud github token owner/repository stream_output: false - prefect.deployments.steps.git_clone: id: git-clone repository: https://x-access-token:{{ get-github-token.stdout }}@github.com/owner/repository.git ``` The `prefect-cloud github token` command requires that the worker's environment is authenticated to Prefect Cloud. Ensure that `PREFECT_API_KEY` and `PREFECT_API_URL` are set in the worker's environment, or that the worker is logged in through `prefect cloud login`. For organizations that require PAT-based access instead, see the [GitHub tab](#git-based-storage) earlier on this page. If you use SSO more broadly with Prefect Cloud, see [Configure single sign-on](/v3/how-to-guides/cloud/manage-users/configure-sso). ## Docker-based storage Another popular flow code storage option is to include it in a Docker image. All work pool options except **Process** and **Prefect Managed** allow you to bake your code into a Docker image. To create a deployment with Docker-based flow code storage use the Python `deploy` method or create a `prefect.yaml` file. If you use the Python `deploy` method to store the flow code in a Docker image, you don't need to use the `from_source` method. The `prefect.yaml` file below was generated by running `prefect deploy` from the CLI (a few lines of metadata were excluded from the top of the file output for brevity). Note that the `build` section is necessary if baking your flow code into a Docker image. ```python docker_deploy.py theme={null} from prefect import flow @flow def my_flow(): print("Hello from inside a Docker container!") if __name__ == "__main__": my_flow.deploy( name="my-docker-deploy", work_pool_name="my_pool", image="my-docker-image:latest", push=False ) ``` ```yaml prefect.yaml theme={null} # build section allows you to manage and build docker images build: - prefect_docker.deployments.steps.build_docker_image: requires: prefect-docker>=0.6.1 id: build-image dockerfile: auto image_name: my-registry/my-image tag: latest # push section allows you to manage if and how this project is uploaded to remote locations push: null # pull section allows you to provide instructions for cloning this project in remote locations pull: - prefect.deployments.steps.set_working_directory: directory: /opt/prefect/my_directory # the deployments section allows you to provide configuration for deploying flows deployments: - name: my-docker-deployment entrypoint: my_file.py:my_flow work_pool: name: my_pool job_variables: image: '{{ build-image.image }}' ``` By default, `.deploy` will build a Docker image that includes your flow code and any `pip` packages specified in a `requirements.txt` file. In the examples above, we elected not to push the resulting image to a remote registry. To push the image to a remote registry, pass `push=True` in the Python `deploy` method or add a `push_docker_image` step to the `push` section of the `prefect.yaml` file. ### Custom Docker image If an `image` is not specified by one of the methods above, deployment flow runs associated with a Docker work pool will use the base Prefect image (e.g. `prefecthq/prefect:3-latest`) when executing. Alternatively, you can create a custom Docker image outside of Prefect by running `docker build` && `docker push` elsewhere (e.g. in your CI/CD pipeline) and then reference the resulting `image` in the `job_variables` section of your deployment definition, or set the `image` as a default directly on the work pool. When you run `prefect deploy` with a custom image in `job_variables` and no build step, Prefect automatically adds a `set_working_directory` pull step to your deployment. In interactive mode, Prefect prompts you for the working directory inside your image (defaulting to `/opt/prefect/`). In non-interactive mode, it defaults to `/opt/prefect` and prints a warning. To use a different path, add an explicit `set_working_directory` step to the `pull` section of your `prefect.yaml`. For more information, see [this discussion of custom Docker images](/v3/how-to-guides/deployment_infra/docker/#automatically-build-a-custom-docker-image-with-a-local-dockerfile). ## Blob storage Another option for flow code storage is any [fsspec](https://filesystem-spec.readthedocs.io/en/latest/)-supported storage location, such as AWS S3, Azure Blob Storage, or GCP GCS. If the storage location is publicly available, or if you are authenticated in the environment where you are creating and running your deployment, you can reference the storage location directly. You don't need to pass credentials explicitly. To pass credentials explicitly to authenticate to your storage location, you can use either of the following block types: * Prefect integration library storage blocks, such as the `prefect-aws` library's `S3Bucket` block, which can use a `AWSCredentials` block when it is created. * `Secret` blocks If you use a storage block such as the `S3Bucket` block, you need to have the `prefect-aws` library available in the environment where your flow code runs. You can do any of the following to make the library available: 1. Install the library into the execution environment directly 2. Specify the library in the work pool's Base Job Template in the **Environment Variables** section like this:`{"EXTRA_PIP_PACKAGES":"prefect-aws"}` 3. Specify the library in the environment variables of the `deploy` method as shown in the examples below 4. Specify the library in a `requirements.txt` file and reference the file in the `pull` step of the `prefect.yaml` file like this: ```yaml theme={null} - prefect.deployments.steps.pip_install_requirements: directory: "{{ pull_code.directory }}" requirements_file: requirements.txt ``` The examples below show how to create a deployment with flow code in a cloud provider storage location. For each example, we show how to access code that is publicly available. The `prefect.yaml` example includes an additional line to reference a credentials block if authenticating to a private storage location through that option. We also include Python code that shows how to use an existing storage block and an example of that creates, but doesn't save, a storage block that references an existing nested credentials block. ```python s3_no_block.py theme={null} from prefect import flow if __name__ == "__main__": flow.from_source( source="s3://my-bucket/my-folder", entrypoint="my_file.py:my_flow", ).deploy( name="my-aws-s3-deployment", work_pool_name="my-work-pool" ) ``` ```python s3_block.py theme={null} from prefect import flow from prefect_aws.s3 import S3Bucket if __name__ == "__main__": s3_bucket_block = S3Bucket.load("my-code-storage-block") # or: # s3_bucket_block = S3Bucket( # bucket="my-bucket", # folder="my-folder", # credentials=AWSCredentials.load("my-credentials-block") # ) flow.from_source( source=s3_bucket_block, entrypoint="my_file.py:my_flow" ).deploy( name="my-aws-s3-deployment", work_pool_name="my-work-pool" job_variables={"env": {"EXTRA_PIP_PACKAGES": "prefect-aws"} }, ) ``` ```yaml prefect.yaml theme={null} build: null push: - prefect_aws.deployments.steps.push_to_s3: id: push_code requires: prefect-aws>=0.5 bucket: my-bucket folder: my-folder credentials: "{{ prefect.blocks.aws-credentials.my-credentials-block }}" # if explicit authentication is required pull: - prefect_aws.deployments.steps.pull_from_s3: id: pull_code requires: prefect-aws>=0.5 bucket: '{{ push_code.bucket }}' folder: '{{ push_code.folder }}' credentials: "{{ prefect.blocks.aws-credentials.my-credentials-block }}" # if explicit authentication is required deployments: - name: my-aws-deployment version: null tags: [] concurrency_limit: null description: null entrypoint: my_file.py:my_flow parameters: {} work_pool: name: my-work-pool work_queue_name: null job_variables: {} enforce_parameter_schema: true schedules: [] ``` To create an `AwsCredentials` block: 1. Install the [prefect-aws](/integrations/prefect-aws) library with `pip install -U prefect-aws` 2. Register the blocks in prefect-aws with `prefect block register -m prefect_aws` 3. Create a user with a role with read and write permissions to access the bucket. If using the UI, create an access key pair with *IAM -> Users -> Security credentials -> Access keys -> Create access key*. Choose *Use case -> Other* and then copy the *Access key* and *Secret access key* values. 4. Create an [`AWSCredentials` block](/integrations/prefect-aws/index#save-credentials-to-an-aws-credentials-block) in code or the Prefect UI. In addition to the block name, most users will fill in the *AWS Access Key ID* and *AWS Access Key Secret* fields. 5. Reference the block as shown above. ```python azure_no_block.py theme={null} from prefect import flow if __name__ == "__main__": flow.from_source( source="az://mycontainer/myfolder", entrypoint="my_file.py:my_flow", ).deploy( name="my-azure-deployment", work_pool_name="my-work-pool", job_variables={"env": {"EXTRA_PIP_PACKAGES": "prefect-azure"} }, ) ``` ```python azure_block.py theme={null} from prefect import flow from prefect_azure import AzureBlobCredentials, AzureBlobStorage if __name__ == "__main__": azure_blob_storage_block = AzureBlobStorage.load("my-code-storage-block") # or # azure_blob_storage_block = AzureBlobStorage( # container="my-prefect-azure-container", # folder="my-folder", # credentials=AzureBlobCredentials.load("my-credentials-block") # ) flow.from_source(source=azure_blob_storage_block, entrypoint="my_file.py:my_flow").deploy( name="my-azure-deployment", work_pool_name="my-work-pool" ) ``` ```yaml prefect.yaml theme={null} build: null push: - prefect_azure.deployments.steps.push_to_azure_blob_storage: id: push_code requires: prefect-azure>=0.4 container: my-prefect-azure-container folder: my-folder credentials: "{{ prefect.blocks.azure-blob-storage-credentials.my-credentials-block }}" # if explicit authentication is required pull: - prefect_azure.deployments.steps.pull_from_azure_blob_storage: id: pull_code requires: prefect-azure>=0.4 container: '{{ push_code.container }}' folder: '{{ push_code.folder }}' credentials: "{{ prefect.blocks.azure-blob-storage-credentials.my-credentials-block }}" # if explicit authentication is required deployments: - name: my-azure-deployment version: null tags: [] concurrency_limit: null description: null entrypoint: my_file.py:my_flow parameters: {} work_pool: name: my-work-pool work_queue_name: null job_variables: {} enforce_parameter_schema: true schedules: [] ``` To create an `AzureBlobCredentials` block: 1. Install the [prefect-azure](/integrations/prefect-azure/) library with `pip install -U prefect-azure` 2. Register the blocks in prefect-azure with `prefect block register -m prefect_azure` 3. Create an access key for a role with sufficient (read and write) permissions to access the blob. You can create a connection string containing all required information in the UI under *Storage Account -> Access keys*. 4. Create an Azure Blob Storage Credentials block in code or the Prefect UI. Enter a name for the block and paste the connection string into the *Connection String* field. 5. Reference the block as shown above. ```python gcs_no_block.py theme={null} from prefect import flow if __name__ == "__main__": flow.from_source( source="gs://my-bucket/my-folder", entrypoint="my_file.py:my_flow", ).deploy( name="my-gcs-deployment", work_pool_name="my-work-pool" ) ``` ```python gcs_block.py theme={null} from prefect import flow from prefect_gcp import GcpCredentials, GCSBucket if __name__ == "__main__": gcs_bucket_block = GCSBucket.load("my-code-storage-block") # or # gcs_bucket_block = GCSBucket( # bucket="my-bucket", # folder="my-folder", # credentials=GcpCredentials.load("my-credentials-block") # ) flow.from_source( source=gcs_bucket_block, entrypoint="my_file.py:my_flow", ).deploy( name="my-gcs-deployment", work_pool_name="my_pool", job_variables={"env": {"EXTRA_PIP_PACKAGES": "prefect-gcp"} }, ) ``` ```yaml prefect.yaml theme={null} build: null push: - prefect_gcp.deployment.steps.push_to_gcs: id: push_code requires: prefect-gcp>=0.6 bucket: my-bucket folder: my-folder credentials: "{{ prefect.blocks.gcp-credentials.my-credentials-block }}" # if explicit authentication is required pull: - prefect_gcp.deployment.steps.pull_from_gcs: id: pull_code requires: prefect-gcp>=0.6 bucket: '{{ push_code.bucket }}' folder: '{{ pull_code.folder }}' credentials: "{{ prefect.blocks.gcp-credentials.my-credentials-block }}" # if explicit authentication is required deployments: - name: my-gcs-deployment version: null tags: [] concurrency_limit: null description: null entrypoint: my_file.py:my_flow parameters: {} work_pool: name: my-work-pool work_queue_name: null job_variables: {} enforce_parameter_schema: true schedules: [] ``` To create a `GcpCredentials` block: 1. Install the [prefect-gcp](/integrations/prefect-gcp/) library with `pip install -U prefect-gcp` 2. Register the blocks in prefect-gcp with `prefect block register -m prefect_gcp` 3. Create a service account in GCP for a role with read and write permissions to access the bucket contents. If using the GCP console, go to *IAM & Admin -> Service accounts -> Create service account*. After choosing a role with the required permissions, see your service account and click on the three dot menu in the *Actions* column. Select *Manage Keys -> ADD KEY -> Create new key -> JSON*. Download the JSON file. 4. Create a GCP Credentials block in code or the Prefect UI. Enter a name for the block and paste the entire contents of the JSON key file into the *Service Account Info* field. 5. Reference the block as shown above. Another authentication option is to give the worker access to the storage location at runtime through SSH keys. Built with [Mintlify](https://mintlify.com).