Learn how to use Modal as a push work pool to schedule, run, and debug your Prefect flows
Term | Definition |
---|---|
Flow | A Python function in a file, decorated with @flow , that is an entrypoint to perform a task in Prefect. |
Run | An instance of a flow, executed at a specific time with a specific payload. |
Deployment | A manifest around a flow, including where it should run, when it should run, and any variables it needs to run. A deployment can schedule your flows, creating runs when needed. |
Work Pool | An interface to the hardware used to run a given flow. |
prefect.yaml
deployment configuration file, standard Python tooling, and CI/CD.
Makefile
to keep things easy:
uv
.
uv
creates a python-version
that is pinned to a micro version (for example, 3.12.6), remove the micro version (for example, 3.12) to avoid downstream issues.pyproject.toml
file created by uv
:
git
to store code within a git repository.
prefect.yaml
file that you generated.
pull
, which defines the repository and branch to pull from, as well as deployments
, which define the flows
that will be run
at some schedule
on a work_pool
.
The work pool will be set to modal later in this tutorial.
Finally, create your flow.
prefect-modal
.
In prefect.yaml
, edit the pull:
section as follows:
access_token
key name with credentials
.uv sync
creates a new virtual environment.
The following steps show how to tell uv
to use the root Python version.main
.
This means that if you push new code to main, the next run of a flow in Modal will change to whatever is in main.
To keep this stable, you can set that to a release branch, for example, to only update that release branch when you’re stable and ready.
Now, configure a few more options on the deployment
to get it ready: entrypoint
, schedule
, and description
.
pref-modal-pool
in the last command is custom, you can name it whatever you want.
Take the work pool name and update your prefect.yaml
file with it.
UV_PROJECT_ENVIRONMENT
is how you tell uv to use the root python.
You could pip install
without uv
but it would be significantly slower (minutes instead of seconds) and you can’t leverage your lockfile, resulting in unexpected behavior!git add . && git commit -m "prefect modal" && git push
to get your code into GitHub, and then uv run prefect deploy --all
to get your deployment live.
But first, learn how to automate Prefect deployments.
PREFECT_API_KEY
and PREFECT_API_URL
.
PREFECT_API_URL
, the format is as follows: https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}
.
Go to the Prefect Cloud Dashboard, and then extract the account ID and workspace ID from the URL.
/account
and /workspace
but the API uses /accounts
and /workspaces
, respectively.PREFECT_API_KEY
, go to the API keys page and create a new API key.
make prefect-deploy
command is used to deploy the flows in your deployment.
make prefect-deploy
locally at any time to keep your deployment up to date.Makefile
.
make prefect-deploy
once to make sure everything works.
Prefect will suggest changes to prefect.yaml
, which you should accept.
make prefect-deploy
shows the command to run your deployed flow.
However, since the code isn’t in GitHub yet, this will fail.
To fix this, push your code: