prefect-azure - Prefect

prefect-azure makes it easy to leverage the capabilities of Azure in your workflows. For example, you can retrieve secrets, read and write Blob Storage objects, and deploy your flows on Azure Container Instances (ACI).

Getting started

Prerequisites

An Azure account and the necessary permissions to access desired services.

Install `prefect-azure`

The following command will install a version of prefect-azure compatible with your installed version of prefect. If you don’t already have prefect installed, it will install the newest version of prefect as well.

pip install "prefect[azure]"

Upgrade to the latest versions of prefect and prefect-azure:

pip install -U "prefect[azure]"

If necessary, see additional installation options for Blob Storage, Cosmos DB, and ML Datastore. To install prefect-azure with all additional capabilities, run the install command above and then run the following command:

pip install "prefect-azure[all_extras]"

Register newly installed block types

prefect block register -m prefect_azure

Examples

Download a blob

from prefect import flow

from prefect_azure import AzureBlobStorageCredentials
from prefect_azure.blob_storage import blob_storage_download

@flow
def example_blob_storage_download_flow():
    connection_string = "connection_string"
    blob_storage_credentials = AzureBlobStorageCredentials(
        connection_string=connection_string,
    )
    data = blob_storage_download(
        blob="prefect.txt",
        container="prefect",
        blob_storage_credentials=blob_storage_credentials,
    )
    return data

example_blob_storage_download_flow()

Use with_options to customize options on any existing task or flow:

custom_blob_storage_download_flow = example_blob_storage_download_flow.with_options(
    name="My custom task name",
    retries=2,
    retry_delay_seconds=10,
)

Using Service Principal Authentication

Azure Blob Storage credentials support Service Principal Name (SPN) authentication for secure access to your storage account.

Example: SPN Authentication

Create an AzureBlobStorageCredentials block with your service principal credentials:

from prefect_azure import AzureBlobStorageCredentials

credentials = AzureBlobStorageCredentials(
    account_url="https://myaccount.blob.core.windows.net/",
    tenant_id="your-tenant-id",
    client_id="your-client-id",
    client_secret="your-client-secret"
)
credentials.save("my-spn-credentials")

Then reference this block in your prefect.yaml deployment steps:

push:
    - prefect_azure.deployments.steps.push_to_azure_blob_storage:
        requires: prefect-azure[blob_storage]
        container: my-container
        folder: my-folder
        credentials: "{{ prefect.blocks.azure-blob-storage-credentials.my-spn-credentials }}"

pull:
    - prefect_azure.deployments.steps.pull_from_azure_blob_storage:
        requires: prefect-azure[blob_storage]
        container: "{{ container }}"
        folder: "{{ folder }}"
        credentials: "{{ prefect.blocks.azure-blob-storage-credentials.my-spn-credentials }}"

When all three SPN fields (tenant_id, client_id, client_secret) are provided, the credentials will use ClientSecretCredential for authentication. If any SPN fields are missing, it will fall back to DefaultAzureCredential.

Run flows on Azure Container Instances

Run flows on Azure Container Instances (ACI) to dynamically scale your infrastructure. See the Azure Container Instances Worker Guide for a walkthrough of using ACI in a hybrid work pool. If you’re using Prefect Cloud, ACI push work pools provide all the benefits of ACI with a quick setup and no worker needed.

Managed Identity Authentication for PostgreSQL (Experimental)

prefect-azure includes a plugin that authenticates the Prefect server to Azure Database for PostgreSQL using a Microsoft Entra ID token from a managed identity, so no database password is stored anywhere. The token is acquired via DefaultAzureCredential and supplied to the database driver on every new connection, so short-lived tokens are refreshed automatically.

Prerequisites

Before using managed identity authentication, configure Azure:

Enable Microsoft Entra authentication on your Azure Database for PostgreSQL flexible server.

Create a database principal for the identity and grant it access, using the pgaadauth extension while connected as an Entra administrator:

SELECT * FROM pgaadauth_create_principal('<identity-name>', false, false);
GRANT CONNECT ON DATABASE prefect TO "<identity-name>";

Make the identity available to the Prefect server — a user-assigned or system-assigned managed identity when running in Azure, or your az login identity when developing locally (both are resolved by DefaultAzureCredential).

Enable the Plugin

Enable the plugin system:

# on Prefect < 3.7 use PREFECT_EXPERIMENTS_PLUGINS_ENABLED=true instead
export PREFECT_PLUGINS_ENABLED=true

Enable managed identity authentication:

export PREFECT_INTEGRATIONS_AZURE_POSTGRES_MANAGED_IDENTITY_ENABLED=true

(Optional) Select a specific user-assigned identity by client ID:

export PREFECT_INTEGRATIONS_AZURE_POSTGRES_MANAGED_IDENTITY_CLIENT_ID=<client-id>

Configure a password-less database connection URL (the plugin supplies the token):

export PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://<identity-name>@<host>:5432/prefect"

The plugin will automatically acquire and inject a Microsoft Entra token as the password when connecting to the database.

Resources

For assistance using Azure, consult the Azure documentation. Refer to the prefect-azure SDK Reference to explore all the capabilities of the prefect-azure library.

Additional installation options

First install the main library compatible with your prefect version:

pip install "prefect[azure]"

Then install the additional capabilities you need. To use Blob Storage:

pip install "prefect-azure[blob_storage]"

To use Cosmos DB:

pip install "prefect-azure[cosmos_db]"

To use ML Datastore:

pip install "prefect-azure[ml_datastore]"

​Getting started

​Prerequisites

​Install prefect-azure

​Register newly installed block types

​Examples

​Download a blob

​Using Service Principal Authentication

​Run flows on Azure Container Instances

​Managed Identity Authentication for PostgreSQL (Experimental)

​Prerequisites

​Enable the Plugin

​Resources

​Additional installation options

Getting started

Prerequisites

Install `prefect-azure`

Register newly installed block types

Examples

Download a blob

Using Service Principal Authentication

Run flows on Azure Container Instances

Managed Identity Authentication for PostgreSQL (Experimental)

Prerequisites

Enable the Plugin

Resources

Additional installation options