Run tasks in the background

​

Introduction

Prefect tasks are atomic pieces of work that you might want to cache or retry. Tasks have .submit() and .map() methods to simplify concurrent execution of a task in a given workflow.

Sometimes your parent workflow doesn’t need to resolve a task’s result, it just needs the task to produce some side effect or save a result. In this case, the caller doesn’t need to waste time blocking on the task’s completion - instead, it can “background” that task run somewhere else.

Background tasks are built for this use case. They allow you to defer a task’s execution to a task worker running in a separate process.

​

Motivating example

Background tasks are useful for dispatching heavy and/or blocking work from your application or workflow to task workers on static infrastructure that you can scale or manage independently.

For example, imagine a web application that needs to trigger an agentic while loop for each request, which we encapsulate as a @task-decorated function named run_agent_loop. The task will likely run longer than an acceptable request-response cycle. You can delay() the expensive task from your endpoint to any task workers subscribed to run_agent_loop’s runs.

​

Define a background task

Define a task by adding the @task decorator to a Python function (like any other Prefect task)

from prefect import task

@task(log_prints=True)
def add(a: int, b: int):
    print(f"{a} + {b} = {a + b}")

All task configuration options (e.g. log_prints, retries, result_storage) are supported.

​

Background task methods

• Use the .delay() method to background a run of this task
• Use the .serve() method or serve() function to start a task worker and execute any waiting task runs

add.delay(1, 2) # background one task run
add.delay(42, 100) # background another task run

add.serve() # start a task worker and execute any waiting task runs

The .serve() method starts a task worker subscribed to that specific task’s runs.

Subscribe to many background tasks at once by providing the serve() utility more than one task:

from prefect import task
from prefect.task_worker import serve

@task(log_prints=True)
def add(a: int, b: int):
    print(f"{a} + {b} = {a + b}")

@task(log_prints=True)
def multiply(a: int, b: int):
    print(f"{a} * {b} = {a * b}")

A = [1, 2, 3]
B = [4, 5, 6]

add.map(A, B, deferred=True) # background 3 task runs - i.e. zip(A, B)
multiply.map(A, B, deferred=True) # background 3 task runs - i.e. zip(A, B)

serve(add, multiply) # start a task worker listening for both `add` and `multiply`

​

Task workers

Task workers are push-based consumers that subscribe to some set of tasks’ runs. They can subscribe to many tasks, and be safely scaled horizontally (e.g. replicas: 4).

They generally do not need to be interacted with by Prefect users. Instead, they are started and stopped implicitly when you call .serve() or serve().

​

Tour of the Prefect background task examples repository

The prefect-background-task-examples repository contains reference implementations of applications leveraging background tasks.

Examples are generally Docker Compose setups that can be run locally with docker compose up. However, as shown above, you can decouple the task submission and execution however you like.

​

Step 0: Clone the repository

git clone https://github.com/PrefectHQ/prefect-background-task-examples.git
cd prefect-background-task-examples

​

Step 1: Setup python environment

This example uses uv, which is generally recommended for python dependency management.

uv venv

source .venv/bin/activate

uv pip install -U prefect

​

Step 2: Connect to Prefect Cloud or a self-hosted Prefect server

Use either Prefect Cloud or a self-hosted Prefect server for these examples.

You must have PREFECT_API_URL set to a Prefect server or Prefect Cloud API URL.

prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api

If using Prefect Cloud, make sure your PREFECT_API_URL and PREFECT_API_KEY are set.

Otherwise, start a Prefect server by running one of the following commands:

# blocks the current terminal session
prefect server start

# run in a detached container
docker run -d -p 4200:4200 --name prefect-server prefecthq/prefect:3-latest prefect server start --host 0.0.0.0

​

Step 3: Run the minimal local example

In minimal-local-setup you’ll find a minimal fastapi application using background tasks.

cd minimal-local-setup

├── README.md
├── api.py
├── requirements.txt
├── tasks.py
└── test

There’s a test script that starts an ephemeral web server and task worker, then sends a demo request and cleans up.

cat test

If you’re comfortable, permit and run the script.

chmod +x test
./test

Otherwise feel free to run the commands at your own pace in separate terminals.

# start the web server
uv run --with-requirements requirements.txt uvicorn api:app --reload

# start the task worker
uv run --with-requirements requirements.txt tasks.py

docker kill prefect-server

​

Step 4: Run the minimal docker compose example

In minimal-docker-compose you’ll find a minimal fastapi application defined in main.py with a /POST /job endpoint that calls process_job.delay(**job_request).

cd minimal-docker-compose

├── README.md
├── _types.py
├── compose.yaml
├── main.py
├── pyproject.toml
└── tasks.py

Start the Docker Compose stack in detached mode:

docker compose up -d

Navigate to http://localhost:8000/docs and try out the /POST /job endpoint.

Watch the logs:

docker compose logs -f

​

Step 5: Explore the rest of the examples

Check out the rest of the examples in the repository, like fastapi-user-signups and chaos-duck.

​

Next steps

• Learn about Results to enable task caching and idempotency.
• Explore a complete example of a FastAPI application that backgrounds tasks