Override job variables on a deployment to set environment variables, specify a Docker image, allocate resources, and more.
There are two ways to deploy flows to work pools: with a prefect.yaml
file or using the Python deploy
method.
In both cases, you can add or override job variables to the work pool’s defaults for a given deployment.
You can override both a work pool and a deployment when a flow run is triggered.
This guide explores common patterns for overriding job variables in both deployment methods.
Job variables are infrastructure-related values that are configurable on a work pool. You can override job variables on a per-deployment or per-flow run basis. This allows you to dynamically change infrastructure from the work pool’s defaults.
For example, when you create or edit a work pool, you can specify a set of environment variables to set in the runtime environment of the flow run.
You could set the following value in the env
field of any work pool:
Rather than hardcoding these values into your work pool in the UI (and making them available to all deployments associated with that work pool), you can override these values for a specific deployment.
Here’s an example repo structure:
With a demo_flow.py
file like:
prefect.yaml
fileImagine you have the following deployment definition in a prefect.yaml
file at the
root of your repository:
While not the focus of this guide, this deployment definition uses a default “global” pull
step,
because one is not explicitly defined on the deployment. For reference, here’s what that would look like at
the top of the prefect.yaml
file:
To provide the EXECUTION_ENVIRONMENT
and MY_NOT_SO_SECRET_CONFIG
environment variables to this deployment,
you can add a job_variables
section to your deployment definition in the prefect.yaml
file:
Then run prefect deploy -n demo-deployment
to deploy the flow with these job variables.
You should see the job variables in the Configuration
tab of the deployment in the UI:
To use environment variables that are already set in your local environment, you can template
these in the prefect.yaml
file using the {{ $ENV_VAR_NAME }}
syntax:
This assumes that the machine where prefect deploy
is run would have these environment variables set.
Run prefect deploy -n demo-deployment
to deploy the flow with these job variables,
and you should see them in the UI under the Configuration
tab.
.deploy()
methodIf you’re using the .deploy()
method to deploy your flow, the process is similar. But instead of
prefect.yaml
defining the job variables, you can pass them as a dictionary to the job_variables
argument
of the .deploy()
method.
Add the following block to your demo_project/daily_flow.py
file from the setup section:
The above example works assuming a couple things:
demo_project/daily_flow.py
already exists in the repository at the specified pathRun the script to deploy the flow with the specified job variables.
The job variables should be visible in the UI under the Configuration
tab.
When running flows, you can pass in job variables that override any values set on the work pool or deployment. Any interface that runs deployments can accept job variables.
Custom runs allow you to pass in a dictionary of variables into your flow run infrastructure. Using the same
env
example from above, you could do the following:
Similarly, runs kicked off through the CLI accept job variables with the -jv
or --job-variable
flag.
You can manage job variables with the Terraform provider for Prefect.
Additionally, runs kicked off through automation actions can use job variables, including ones rendered from Jinja templates.
Override job variables on a deployment to set environment variables, specify a Docker image, allocate resources, and more.
There are two ways to deploy flows to work pools: with a prefect.yaml
file or using the Python deploy
method.
In both cases, you can add or override job variables to the work pool’s defaults for a given deployment.
You can override both a work pool and a deployment when a flow run is triggered.
This guide explores common patterns for overriding job variables in both deployment methods.
Job variables are infrastructure-related values that are configurable on a work pool. You can override job variables on a per-deployment or per-flow run basis. This allows you to dynamically change infrastructure from the work pool’s defaults.
For example, when you create or edit a work pool, you can specify a set of environment variables to set in the runtime environment of the flow run.
You could set the following value in the env
field of any work pool:
Rather than hardcoding these values into your work pool in the UI (and making them available to all deployments associated with that work pool), you can override these values for a specific deployment.
Here’s an example repo structure:
With a demo_flow.py
file like:
prefect.yaml
fileImagine you have the following deployment definition in a prefect.yaml
file at the
root of your repository:
While not the focus of this guide, this deployment definition uses a default “global” pull
step,
because one is not explicitly defined on the deployment. For reference, here’s what that would look like at
the top of the prefect.yaml
file:
To provide the EXECUTION_ENVIRONMENT
and MY_NOT_SO_SECRET_CONFIG
environment variables to this deployment,
you can add a job_variables
section to your deployment definition in the prefect.yaml
file:
Then run prefect deploy -n demo-deployment
to deploy the flow with these job variables.
You should see the job variables in the Configuration
tab of the deployment in the UI:
To use environment variables that are already set in your local environment, you can template
these in the prefect.yaml
file using the {{ $ENV_VAR_NAME }}
syntax:
This assumes that the machine where prefect deploy
is run would have these environment variables set.
Run prefect deploy -n demo-deployment
to deploy the flow with these job variables,
and you should see them in the UI under the Configuration
tab.
.deploy()
methodIf you’re using the .deploy()
method to deploy your flow, the process is similar. But instead of
prefect.yaml
defining the job variables, you can pass them as a dictionary to the job_variables
argument
of the .deploy()
method.
Add the following block to your demo_project/daily_flow.py
file from the setup section:
The above example works assuming a couple things:
demo_project/daily_flow.py
already exists in the repository at the specified pathRun the script to deploy the flow with the specified job variables.
The job variables should be visible in the UI under the Configuration
tab.
When running flows, you can pass in job variables that override any values set on the work pool or deployment. Any interface that runs deployments can accept job variables.
Custom runs allow you to pass in a dictionary of variables into your flow run infrastructure. Using the same
env
example from above, you could do the following:
Similarly, runs kicked off through the CLI accept job variables with the -jv
or --job-variable
flag.
You can manage job variables with the Terraform provider for Prefect.
Additionally, runs kicked off through automation actions can use job variables, including ones rendered from Jinja templates.