gcloud
CLI installed on your local machine. You can follow Google Cloud’s installation guide. If you’re using Apple (or a Linux system) you can also use Homebrew for installation.gcloud
is installed. If you haven’t already authenticated with gcloud
, run the following command and follow the instructions to log in to your GCP account.
<PROJECT_ID>
with your GCP project’s ID.
prefect-project
the command will look like this:
cloud-run
via the CLI (you’ll want to replace the <WORK-POOL-NAME>
with the name of your work pool):
<ACCOUNT-ID>
and <WORKSPACE-ID>
with your Prefect account and workspace IDs (both will be available in the URL of the UI when previewing the workspace dashboard).
You’ll want to replace <YOUR-API-KEY>
with an active API key as well.
<YOUR-SERVICE-ACCOUNT-NAME>
with the name of the service account you created in the first step of this guide, and replace <WORK-POOL-NAME>
with the name of the work pool you created in the second step.
prefect-worker
service by navigating to the Cloud Run page of your Google Cloud console. Additionally, you should be able to see a record of this worker in the Prefect UI on the work pool’s page by navigating to the Worker
tab.
Let’s not leave our worker hanging, it’s time to give it a job.
<REPOSITORY-NAME>
with your own value. :
flows
.
Navigate to the flows
subdirectory and create a new file for your flow. Feel free to write your own flow, but here’s a ready-made one for your convenience:
weather_flow.py
, but you can name yours whatever you’d like.
prefect.yaml
fileprefect.yaml
file, which will be responsible for managing the deployments of this repository.
Navigate back to the root of your directory, and run the following command to create a prefect.yaml
file using Prefect’s docker deployment recipe.
us-docker.pkg.dev/<PROJECT-ID>/<REPOSITORY-NAME>/
. You’ll want to replace <PROJECT-ID>
with the ID of your project in GCP. This should match the ID of the project you used in first step of this guide. Here is an example of what this could look like:
prefect.yaml
file available at the root of your project. The contents will look similar to the example below, however, we’ve added in a combination of YAML templating options and Prefect deployment actions to build out a simple CI/CD process. Feel free to copy the contents and paste them in your prefect.yaml:
<WORKING-DIRECTORY>
with the name of the directory where your flow folder and prefect.yaml
live. You’ll also need to replace <PATH-TO-ARTIFACT-REGISTRY>
with the path to the Docker repository in your Google Artifact Registry.prefect.yaml
file above and what they do, feel free to read this next section. Otherwise, you can skip ahead to Flow Deployment.
In the build
section of the prefect.yaml
the following step is executed at deployment build time:
prefect_docker.deployments.steps.build_docker_image
: builds a Docker image automatically which uses the name and tag chosen previously.platform: linux/amd64
to your build_docker_image
step to ensure that your docker image uses an AMD architecture. For example:push
section sends the Docker image to the Docker repository in your Google Artifact Registry, so that it can be easily accessed by the worker for flow run execution.
The pull
section sets the working directory for the process prior to importing your flow.
In the deployments
section of the prefect.yaml
file above, you’ll see that there is a deployment declaration named gcp-weather-deploy
. Within the declaration, the entrypoint for the flow is specified along with some default parameters which will be passed to the flow at runtime. Last but not least, the name of the work pool that we created in step 2 of this guide is specified.
prefect.yaml
file, run the following command in the terminal to deploy your flow: