Securely store typed configuration
Prefect blocks allow you to manage configuration schemas, infrastructure, and secrets for use with deployments or flow scripts.
Prefect blocks store typed configuration that can be used across workflows and deployments. The most common use case for blocks is storing credentials used to access external systems such as AWS or GCP. Prefect supports a large number of common blocks and it’s also easy to create your own!
Blocks and parameters
Blocks are useful for sharing configuration across flow runs and between flows.
For configuration that will change between flow runs, we recommend using parameters.
How blocks work
To get the most out of blocks it’s important to understand how they work.
There are three layers to a block: its type, a document, and a Python class.
Block type
A block type is essentially a schema registered with the Prefect API. This schema can be inspected and discovered in the UI on the Blocks page.
To see block types available for configuration, use prefect block type ls
from the CLI or navigate to the Blocks page in the UI and click +.
These types separate blocks from Prefect variables, which are unstructured JSON documents.
In addition, block schemas allow for fields of SecretStr
type which are stored with additional encryption and not displayed by default in the UI.
Block types are identified by a slug that is not configurable.
Users should rarely need to register types in this way - saving a block document will also automatically register its type.
Block document
A block document is an instantiation of the schema, or block type. A document contains specific values for each field defined in the schema.
All block types allow for the creation of as many documents as you wish. Building on our example above:
Block documents can also be created and updated in the UI or API for easy change management. This allows you to work with slowly changing configuration without having to redeploy all workflows that rely on it; for example, you may use this to rotate credentials on a regular basis without touching your deployments.
Block class
A block class is the primary user-facing object; it is a Python class whose attributes are loaded from a block document. Most Prefect blocks encapsulate additional functionality built on top of the block document.
For example, an S3Bucket
block contains methods for downloading data from, or upload data to, an S3 bucket; a SnowflakeConnector
block contains methods for querying Snowflake databases.
Returning to our Cube
example from above:
The class itself is not stored server-side when registering block types and block documents.
For this reason, we highly recommend loading block documents by first importing the block class and then calling its load
method with the relevant document name.
Using blocks
Using existing or built-in Prefect blocks is as simple as importing the relevant class and calling its save
and load
methods.
Note that many Prefect blocks are packaged into one of Prefect’s integration libraries.
To instantiate a block document that stores an S3 bucket value, use the S3Bucket
block:
To store this block document for future use, use the .save()
method:
To update a previously saved block document, overwrite the existing document by passing overwrite=True
:
You can now use the block document name to load the block:
Alternatively, load a block with the unique slug that is a combination of the block type slug and the block name.
To load the S3Bucket
block from above, run the following:
When using the generic Block.load
interface for block documents, the Python class that created the type must be importable.
For this reason, we highly recommend loading block documents by first importing the block class and then calling its load
method with the relevant document name.
You can also delete a block document with the .delete()
method:
Alternatively, use the CLI to delete specific blocks with a given slug or id:
Nested blocks
Blocks are composable: a block can be used within other blocks. You can create a block type that uses capabilities from another block type by giving it the appropriate type.
Nested blocks are loosely coupled, and configuration can be changed for each block independently. This allows sharing configuration across multiple block documents.
For example, we can expand our basic S3Bucket
example above using the AwsCredentials
block:
The aws_secret_access_key
field is typed as SecretStr
, meaning it will be stored and displayed securely.
Alternatively, we can also load previously saved credentials:
Both of these create a reference from the AwsCredentials
block document my_aws_credentials
to the S3Bucket
block document my_s3_bucket
,
so that changes to the values in my_aws_credentials
propagate to my_s3_bucket
.
Create a new block type
To create a custom block type, define a class that subclasses Block
. The Block
base class builds
on Pydantic’s BaseModel
, so you can declare custom fields just like a Pydantic model.
We’ve already seen an example of a Cube
block that represents a cube and holds information about the length of each edge in inches:
Register custom blocks
In addition to the register_type_and_schema
method shown above, you can register blocks from a Python module with a CLI command:
This command is useful for registering all blocks found within a module in a Prefect Integration library.
Alternatively, if a custom block was created in a .py
file, you can register the block with the CLI command:
Block documents can now be created with the registered block schema.
Secret fields
All block values are encrypted before being stored.
If you have values that you would not like visible in the UI or in logs,
use the SecretStr
field type provided by Pydantic to automatically obfuscate those values.
You can use this capability for fields that store credentials such as passwords and API tokens.
Here’s an example of an AwsCredentials
block that uses SecretStr
:
Since aws_secret_access_key
has the SecretStr
type hint assigned to it,
the value of that field is not exposed if the object is logged:
Prefect’s SecretDict
field type allows you to add a dictionary field to your block
that automatically obfuscates values at all levels in the UI or in logs.
This capability is useful for blocks where typing or structure of secret fields is not known until configuration time.
Here’s an example of a block that uses SecretDict
:
system_secrets
is obfuscated when system_configuration_block
is displayed, but system_variables
show up in plain-text:
Customize a block’s display
You can set metadata fields on a block type’s subclass to control how a block displays.
Available metadata fields include:
Property | Description |
---|---|
_block_type_name | Display name of the block in the UI. Defaults to the class name. |
_block_type_slug | Unique slug used to reference the block type in the API. Defaults to a lowercase, dash-delimited version of the block type name. |
_logo_url | URL pointing to an image that should be displayed for the block type in the UI. Default to None . |
_description | Short description of block type. Defaults to docstring, if provided. |
_code_example | Short code snippet shown in UI for how to load/use block type. Defaults to first example provided in the docstring of the class, if provided. |
Update custom Block
types
Here’s an example of how to add a bucket_folder
field to your custom S3Bucket
block; it represents the default path to
read and write objects from (this field exists on our implementation).
Add the new field to the class definition:
Then register the updated block type with either your Prefect Cloud account or your self-hosted Prefect server instance.
If you have any existing blocks of this type that were created before the update and you’d prefer to not re-create them, migrate them to the new version of your block type by adding the missing values:
Pre-registered blocks
Built-in blocks
Commonly used block types come built-in with Prefect. You can create and use these block types through the UI without installing any additional packages.
Block | Slug | Description |
---|---|---|
Custom Webhook | custom-webhook | Call custom webhooks. |
Discord Webhook | discord-webhook | Call Discord webhooks. |
Local File System | local-file-system | Store data as a file on a local file system. |
Mattermost Webhook | mattermost-webhook | Send notifications through a provided Mattermost webhook. |
Microsoft Teams Webhook | ms-teams-webhook | Send notifications through a provided Microsoft Teams webhook. |
Opsgenie Webhook | opsgenie-webhook | Send notifications through a provided Opsgenie webhook. |
Pager Duty Webhook | pager-duty-webhook | Send notifications through a provided PagerDuty webhook. |
Remote File System | remote-file-system | Access files on a remote file system. |
Secret | secret | Store a secret value. The value will be obfuscated when this block is logged or shown in the UI. |
Sendgrid Email | sendgrid-email | Send notifications through Sendgrid email. |
Slack Webhook | slack-webhook | Send notifications through a provided Slack webhook. |
SMB | smb | Store data as a file on a SMB share. |
Twilio SMS | twilio-sms | Send notifications through Twilio SMS. |
The S3
, Azure
, GCS
, and GitHub
blocks are deprecated in favor of the corresponding S3Bucket
,
AzureBlobStorageCredentials
, GCSBucket
, and GitHubRepository
blocks found in the
Prefect integration libraries.
The JSON, DateTime, and String blocks are deprecated in favor of Variables.
Blocks in Prefect integration libraries
Some block types that appear in the UI can be created immediately, with the corresponding integration library installed for use.
For example, an AWS Secret block can be created, but not used until the prefect-aws
library is installed.
Find available block types in many of the published Prefect integrations libraries. If a block type is not available in the UI, you can register it through the CLI.
Block | Slug | Integration |
---|---|---|
ECS Task | ecs-task | prefect-aws |
MinIO Credentials | minio-credentials | prefect-aws |
S3 Bucket | s3-bucket | prefect-aws |
Azure Blob Storage Credentials | azure-blob-storage-credentials | prefect-azure |
Azure Container Instance Credentials | azure-container-instance-credentials | prefect-azure |
Azure Container Instance Job] | azure-container-instance-job | prefect-azure |
Azure Cosmos DB Credentials | azure-cosmos-db-credentials | prefect-azure |
AzureML Credentials | azureml-credentials | prefect-azure |
BitBucket Credentials | bitbucket-credentials | prefect-bitbucket |
BitBucket Repository | bitbucket-repository | prefect-bitbucket |
Databricks Credentials | databricks-credentials | prefect-databricks |
dbt CLI BigQuery Target Configs | dbt-cli-bigquery-target-configs | prefect-dbt |
dbt CLI Profile | dbt-cli-profile | prefect-dbt |
dbt Cloud Credentials | dbt-cloud-credentials | prefect-dbt |
dbt CLI Global Configs | dbt-cli-global-configs | prefect-dbt |
dbt CLI Postgres Target Configs | dbt-cli-postgres-target-configs | prefect-dbt |
dbt CLI Snowflake Target Configs | dbt-cli-snowflake-target-configs | prefect-dbt |
dbt CLI Target Configs | dbt-cli-target-configs | prefect-dbt |
Docker Container | docker-container | prefect-docker |
Docker Host | docker-host | prefect-docker |
Docker Registry Credentials | docker-registry-credentials | prefect-docker |
Email Server Credentials | email-server-credentials | prefect-email |
BigQuery Warehouse | bigquery-warehouse | prefect-gcp |
GCP Cloud Run Job | cloud-run-job | prefect-gcp |
GCP Credentials | gcp-credentials | prefect-gcp |
GcpSecret | gcpsecret | prefect-gcp |
GCS Bucket | gcs-bucket | prefect-gcp |
Vertex AI Custom Training Job | vertex-ai-custom-training-job | prefect-gcp |
GitHub Credentials | github-credentials | prefect-github |
GitHub Repository | github-repository | prefect-github |
GitLab Credentials | gitlab-credentials | prefect-gitlab |
GitLab Repository | gitlab-repository | prefect-gitlab |
Kubernetes Cluster Config | kubernetes-cluster-config | prefect-kubernetes |
Kubernetes Credentials | kubernetes-credentials | prefect-kubernetes |
Kubernetes Job | kubernetes-job | prefect-kubernetes |
Shell Operation | shell-operation | prefect-shell |
Slack Credentials | slack-credentials | prefect-slack |
Slack Incoming Webhook | slack-incoming-webhook | prefect-slack |
Snowflake Connector | snowflake-connector | prefect-snowflake |
Snowflake Credentials | snowflake-credentials | prefect-snowflake |
Database Credentials | database-credentials | prefect-sqlalchemy |
SQLAlchemy Connector | sqlalchemy-connector | prefect-sqlalchemy |
Anyone can create a custom block type and, optionally, share it with the community.
Was this page helpful?