Automations

Automations in Prefect Cloud enable you to configure actions that Prefect executes automatically based on trigger conditions related to your flows and work pools.

Using triggers and actions you can automatically kick off flow runs, pause deployments, or send custom notifications in response to real-time monitoring events.

Automations are only available in Prefect Cloud

Notifications in an open-source Prefect server provide a subset of the notification message-sending features available in Automations.

Automations overview

The Automations page provides an overview of all configured automations for your workspace.

Selecting the toggle next to an automation pauses execution of the automation.

The button next to the toggle provides commands to copy the automation ID, edit the automation, or delete the automation.

Select the name of an automation to view Details about it and relevant Events.

Create an automation

On the Automations page, select the + icon to create a new automation. You'll be prompted to configure:

• A trigger condition that causes the automation to execute.
• One or more actions carried out by the automation.
• Details about the automation, such as a name and description.

Triggers

Triggers specify the conditions under which your action should be performed. Triggers can be of several types, including triggers based on:

• Flow run state change
• Note - Flow Run Tags currently are only evaluated with OR criteria
• Work pool status
• Custom event triggers

Automations API

The automations API enables further programmatic customization of trigger and action policies based on arbitrary events.

Importantly, triggers can be configured not only in reaction to events, but also proactively: to trigger in the absence of an event you expect to see.

For example, in the case of flow run state change triggers, you might expect production flows to finish in no longer than thirty minutes. But transient infrastructure or network issues could cause your flow to get “stuck” in a running state. A trigger could kick off an action if the flow stays in a running state for more than 30 minutes. This action could be on the flow itself, such as canceling or restarting it, or it could take the form of a notification so someone can take manual remediation steps.

Custom Triggers

Custom triggers allow advanced configuration of the conditions on which a trigger executes its actions.

For example, if you would like a trigger to execute an action if it receives two flow run failure events of a specific deployment within ten seconds, you could paste in the following trigger configuration:

{
  "match": {
    "prefect.resource.id": "prefect.flow-run.*"
  },
  "match_related": {
    "prefect.resource.id": "prefect.deployment.70cb25fe-e33d-4f96-b1bc-74aa4e50b761",
    "prefect.resource.role": "deployment"
  },
  "for_each": [
    "prefect.resource.id"
  ],
  "after": [],
  "expect": [
    "prefect.flow-run.Failed"
  ],
  "posture": "Reactive",
  "threshold": 2,
  "within": 10
}

Matching on multiple resources

Each key in match and match_related can accept a list of multiple values that are OR'd together. For example, to match on multiple deployments:

"match_related": {
  "prefect.resource.id": [
    "prefect.deployment.70cb25fe-e33d-4f96-b1bc-74aa4e50b761",
    "prefect.deployment.c33b8eaa-1ba7-43c4-ac43-7904a9550611"
  ],
  "prefect.resource.role": "deployment"
},

Actions

Actions specify what your automation does when its trigger criteria are met. Current action types include:

• Cancel a flow run
• Pause a flow run
• Run a deployment
• Pause or resume a deployment schedule
• Pause or resume a work queue
• Pause or resume an automation
• Send a notification
• Call a webhook

Selected and inferred action targets

Some actions require you to either select the target of the action, or specify that the target of the action should be inferred.

Selected targets are simple, and useful for when you know exactly what object your action should act on — for example, the case of a cleanup flow you want to run or a specific notification you’d like to send.

Inferred targets are deduced from the trigger itself.

For example, if a trigger fires on a flow run that is stuck in a running state, and the action is to cancel an inferred flow run, the flow run to cancel is inferred as the stuck run that caused the trigger to fire.

Similarly, if a trigger fires on a work queue event and the corresponding action is to pause an inferred work queue, the inferred work queue is the one that emitted the event.

Prefect tries to infer the relevant event whenever possible, but sometimes one does not exist.

Specify a name and, optionally, a description for the automation.

Create an automation via deployment triggers

To enable the simple configuration of event-driven deployments, Prefect provides deployment triggers - a shorthand for creating automations that are linked to specific deployments to run them based on the presence or absence of events.

# prefect.yaml
deployments:
  - name: my-deployment
    entrypoint: path/to/flow.py:decorated_fn
    work_pool:
      name: my-process-pool
    triggers:
      - enabled: true
        match:
          prefect.resource.id: my.external.resource
        expect:
          - external.resource.pinged
        parameters:
          param_1: "{{ event }}"

At deployment time, this will create a linked automation that is triggered by events matching your chosen grammar, which will pass the templatable event as a parameter to the deployment's flow run.

Pass triggers to prefect deploy

You can pass one or more --trigger arguments to prefect deploy, which can be either a JSON string or a path to a .yaml or .json file.

# Pass a trigger as a JSON string
prefect deploy -n test-deployment \
  --trigger '{
    "enabled": true, 
    "match": {
      "prefect.resource.id": "prefect.flow-run.*"
    }, 
    "expect": ["prefect.flow-run.Completed"]
  }'

# Pass a trigger using a JSON/YAML file
prefect deploy -n test-deployment --trigger triggers.yaml
prefect deploy -n test-deployment --trigger my_stuff/triggers.json

For example, a triggers.yaml file could have many triggers defined:

triggers:
  - enabled: true
    match:
      prefect.resource.id: my.external.resource
    expect:
      - external.resource.pinged
    parameters:
      param_1: "{{ event }}"
  - enabled: true
    match:
      prefect.resource.id: my.other.external.resource
    expect:
      - some.other.event
    parameters:
      param_1: "{{ event }}"

Both of the above triggers would be attached to test-deployment after running prefect deploy.

Triggers passed to prefect deploy will override any triggers defined in prefect.yaml

While you can define triggers in prefect.yaml for a given deployment, triggers passed to prefect deploy will take precedence over those defined in prefect.yaml.

Note that deployment triggers contribute to the total number of automations in your workspace.

Automation notifications

Notifications enable you to set up automation actions that send a message.

Automation notifications support sending notifications via any predefined block that is capable of and configured to send a message. That includes, for example:

• Slack message to a channel
• Microsoft Teams message to a channel
• Email to a configured email address

Notification blocks must be pre-configured

Notification blocks must be pre-configured prior to creating a notification action. Any existing blocks capable of sending messages will be shown in the block drop-down list.

The Add + button cancels the current automation creation process and enables configuration a notification block.

Templating notifications with Jinja

The notification body can include templated variables using Jinja syntax. Templated variable enable you to include details relevant to automation trigger, such as a flow or pool name.

Jinja templated variable syntax wraps the variable name in double curly brackets, like {{ variable }}.

You can access properties of the underlying flow run objects including:

• flow_run
• flow
• deployment
• work_queue
• work_pool

In addition to its native properties, each object includes an id along with created and updated timestamps.

The flow_run|ui_url token returns the URL for viewing the flow run in Prefect Cloud.

Here’s an example for something that would be relevant to a flow run state-based notification:

Flow run {{ flow_run.name }} entered state {{ flow_run.state.name }}. 

    Timestamp: {{ flow_run.state.timestamp }}
    Flow ID: {{ flow_run.flow_id }}
    Flow Run ID: {{ flow_run.id }}
    State message: {{ flow_run.state.message }}

The resulting Slack webhook notification would look something like this:

You could include flow and deployment properties.

Flow run {{ flow_run.name }} for flow {{ flow.name }}
entered state {{ flow_run.state.name }}
with message {{ flow_run.state.message }}

Flow tags: {{ flow_run.tags }}
Deployment name: {{ deployment.name }}
Deployment version: {{ deployment.version }}
Deployment parameters: {{ deployment.parameters }}

An automation that reports on work pool status might include notifications using work_pool properties.

Work pool status alert!

Name: {{ work_pool.name }}
Last polled: {{ work_pool.last_polled }}

In addition to those shortcuts for flows, deployments, and work pools, you have access to the automation and the event that triggered the automation. See the Automations API for additional details.

Automation: {{ automation.name }}
Description: {{ automation.description }}

Event: {{ event.id }}
Resource:
{% for label, value in event.resource %}
{{ label }}: {{ value }}
{% endfor %}
Related Resources:
{% for related in event.related %}
    Role: {{ related.role }}
    {% for label, value in event.resource %}
    {{ label }}: {{ value }}
    {% endfor %}
{% endfor %}

Note that this example also illustrates the ability to use Jinja features such as iterator and for loop control structures when templating notifications.