This module contains functions for validating job variables for deployments, work pools,
flow runs, and RunDeployment actions. These functions are used to validate that job
variables provided by users conform to the JSON schema defined in the work pool’s base job
template.Note some important details:
The order of applying job variables is: work pool’s base job template, deployment, flow
run. This means that flow run job variables override deployment job variables, which
override work pool job variables.
The validation of job variables for work pools and deployments ignores required keys in
because we don’t know if the full set of overrides will include values for any required
fields.
Work pools can include default values for job variables. These can be normal types or
references to blocks. We have not been validating these values or whether default blocks
satisfy job variable JSON schemas. To avoid failing validation for existing (otherwise
working) data, we ignore invalid defaults when validating deployment and flow run
variables, but not when validating the work pool’s base template, e.g. during work pool
creation or updates. If we find defaults that are invalid, we have to ignore required
fields when we run the full validation.
A flow run is the terminal point for job variables, so it is the only place where
we validate required variables and default values. Thus,
validate_job_variables_for_deployment_flow_run and
validate_job_variables_for_run_deployment_action check for required fields.
We have been using Pydantic v1 to generate work pool base job templates, and it produces
invalid JSON schemas for some fields, e.g. tuples and optional fields. We try to fix these
schemas on the fly while validating job variables, but there is a case we can’t resolve,
which is whether or not an optional field supports a None value. In this case, we allow
None values to be passed in, which means that if an optional field does not actually
allow None values, the Pydantic model will fail to validate at runtime.
Validate job variables for a flow run created for a deployment.Flow runs are the terminal point for job variable overlays, so we validate required
job variables because all variables should now be present.
Validate job variables for deployment creation and updates.This validation applies only to deployments that have a work pool. If the deployment
does not have a work pool, we cannot validate job variables because we don’t have a
base job template to validate against, so we skip this validation.Unlike validations for flow runs, validation here ignores required keys in the schema
because we don’t know if the full set of overrides will include values for any
required fields. If the full set of job variables when a flow is running, including
the deployment’s and flow run’s overrides, fails to specify a value for the required
key, that’s an error.
Validate the default job variables for a work pool.This validation checks that default values for job variables match the JSON schema
defined in the work pool’s base job template. It also resolves references to block
documents in the default values and hydrates them to perform the validation.Unlike validations for flow runs, validation here ignores required keys in the schema
because we’re only concerned with default values. The absence of a default for a
required field is not an error, but if the full set of job variables when a flow is
running, including the deployment’s and flow run’s overrides, fails to specify a value
for the required key, that’s an error.NOTE: This will raise an HTTP 404 error if a referenced block document does not exist.
Validate the job variables for a RunDeployment action.This action is equivalent to creating a flow run for a deployment, so we validate
required job variables because all variables should now be present.
