prefect-gcp
prefect-gcp
helps you leverage the capabilities of Google Cloud Platform (GCP) in your workflows.
For example, you can run flows on Vertex AI or Cloud Run, read and write data to BigQuery and Cloud Storage, and retrieve secrets with Secret Manager.
Getting started
Prerequisites
- A GCP account and the necessary permissions to access desired services.
Install prefect-gcp
Install prefect-gcp
as an extra of prefect
.
If you don’t already have prefect
installed, it will install the newest version of prefect
as well.
If using BigQuery, Cloud Storage, Secret Manager, or Vertex AI, see additional installation options.
Install extras
To install prefect-gcp
with all additional capabilities, run the install command above and then run the following command:
Or, install extras individually:
Register newly installed block types
Register the block types in the module to make them available for use.
Blocks setup
Credentials
Authenticate with a service account to use prefect-gcp
services.
- Refer to the GCP service account documentation to create and download a service account key file.
- Copy the JSON contents.
- Use the Python code below, replace the placeholders with your information.
This credential block can be used to create other prefect_gcp
blocks.
service_account_info
vs service_account_file
The advantage of using service_account_info
, instead of service_account_file
, is that it is accessible across containers.
If service_account_file
is used, the provided path must be available in the container executing the flow.
BigQuery
Read data from and write to Google BigQuery within your Prefect flows.
Be sure to install prefect-gcp
with the BigQuery extra.
Secret Manager
Manage secrets in Google Cloud Platform’s Secret Manager.
Cloud Storage
Create a block to interact with a GCS bucket.
Run flows on Google Cloud Run or Vertex AI
Run flows on Google Cloud Run or Vertex AI to dynamically scale your infrastructure.
Prefect Cloud offers Google Cloud Run push work pools. Push work pools submit runs directly to Google Cloud Run, instead of requiring a worker to actively poll for flow runs to execute.
See the Google Cloud Run Worker Guide for a walkthrough of using Google Cloud Run in a hybrid work pool.
Examples
Interact with BigQuery
This code creates a new dataset in BigQuery, defines a table, insert rows, and fetches data from the table:
Use Prefect with Google Cloud Storage
Interact with Google Cloud Storage.
The code below uses prefect_gcp
to upload a file to a Google Cloud Storage bucket and download the same file under a different filename.
Upload and download directories
GcsBucket
supports uploading and downloading entire directories.
Save secrets with Google Secret Manager
Read and write secrets with Google Secret Manager.
Be sure to install prefect-gcp
with the Secret Manager extra.
The code below writes a secret to the Secret Manager, reads the secret data, and deletes the secret.
Resources
For assistance using GCP, consult the Google Cloud documentation.
GCP can also authenticate without storing credentials in a block. See Access third-party secrets for an example that uses AWS Secrets Manager and Snowflake.
Refer to the prefect-gcp
SDK documentation to explore all of the capabilities of the prefect-gcp
library.
Was this page helpful?