Learn about code storage as it relates to execution of deployments
flow.deploy
method
.deploy
, specify a storage location for your flow with the flow.from_source
method.
source
is either a URL to a git repository or a storage object. For example:
source=Path(__file__).parent
or source="/path/to/file"
source="https://github.com/org/my-repo.git"
source=GitRepository(url="https://github.com/org/my-repo.git")
entrypoint
is the path to the file the flow is located in and the function name, separated by a colon.
prefect.yaml
file
prefect.yaml
file interactively, run prefect deploy
from the CLI and follow the prompts.
prefect.yaml
file may define a pull
section that specifies the storage location for your flow. For example:
from_source
or prefect.yaml
to specify the storage location for your flow code, the resulting deployment will have a set of pull
steps that your worker will use to retrieve the flow code at runtime.
Secret
block or git
-platform-specific credentials block to store your credentials:
Then you can reference this block in the Python deploy
method or the prefect.yaml
file pull step.
If using the Python deploy
method with a private repository that references a block, provide a GitRepository
object instead of a URL, as shown below.
Secret
block, you can create it through code or the UI ahead of time and reference it at deployment creation as shown above.If using a GitHubCredentials
block to store your credentials, you can create it ahead of time and reference it at deployment creation.prefect-github
with pip install -U prefect-github
prefect-github
with prefect block register -m prefect_github
GitHubCredentials
block through code or the Prefect UI and reference it at deployment creation as shown above.branch
if creating a GitRepository
object.
The default is "main"
.
git
-based version control platform.
This is intentional to avoid confusion about the git
history and push process.prefect.yaml
file. Make sure to replace owner/repository
with the name of your repository:
deploy
method or create a prefect.yaml
file.
deploy
method to store the flow code in a Docker image, you don’t need to use the from_source
method.prefect.yaml
file below was generated by running prefect deploy
from the CLI (a few lines of metadata were excluded from the top of the file output for brevity).
Note that the build
section is necessary if baking your flow code into a Docker image.
.deploy
will build a Docker image that includes your flow code and any pip
packages specified in a requirements.txt
file.
In the examples above, we elected not to push the resulting image to a remote registry.
To push the image to a remote registry, pass push=True
in the Python deploy
method or add a push_docker_image
step to the push
section of the prefect.yaml
file.
image
is not specified by one of the methods above, deployment flow runs associated with a Docker work pool will use the base Prefect image (e.g. prefecthq/prefect:3-latest
) when executing.
Alternatively, you can create a custom Docker image outside of Prefect by running docker build
&& docker push
elsewhere (e.g. in your CI/CD pipeline) and then reference the resulting image
in the job_variables
section of your deployment definition, or set the image
as a default directly on the work pool.
For more information, see this discussion of custom Docker images.
prefect-aws
library’s S3Bucket
block, which can use a AWSCredentials
block when it is created.Secret
blocksS3Bucket
block, you need to have the prefect-aws
library available in the environment where your flow code runs.You can do any of the following to make the library available:{"EXTRA_PIP_PACKAGES":"prefect-aws"}
deploy
method as shown in the examples belowrequirements.txt
file and reference the file in the pull
step of the prefect.yaml
file like this:prefect.yaml
example includes an additional line to reference a credentials block if authenticating to a private storage location through that option.
We also include Python code that shows how to use an existing storage block and an example of that creates, but doesn’t save, a storage block that references an existing nested credentials block.
AwsCredentials
block:pip install -U prefect-aws
prefect block register -m prefect_aws
AWSCredentials
block in code or the Prefect UI. In addition to the block name, most users will fill in the AWS Access Key ID and AWS Access Key Secret fields.