Task runners

Task runners are a simple and consistent interface to concurrency primitives - they are not required for task execution. Calling a task function directly executes the function in the main thread by default, blocking execution of its caller until the task completes.

To enable concurrent, parallel, or distributed execution of tasks, use the .submit() or .map() methods to submit tasks to a task runner.

The default task runner in Prefect is the ThreadPoolTaskRunner, which runs tasks concurrently in independent threads.

For truly parallel or distributed task execution, you must use one of the following task runners, which are available as extras of the prefect library:

• DaskTaskRunner can run tasks using dask.distributed (install prefect[dask])
• RayTaskRunner can run tasks using Ray (install prefect[ray])

​

Configure a task runner

To configure a specific task runner, provide a task_runner keyword argument to the parent flow:

from prefect import flow, task
from prefect.futures import wait
from prefect.task_runners import ThreadPoolTaskRunner
import time

@task
def stop_at_floor(floor):
    print(f"elevator moving to floor {floor}")
    time.sleep(floor)
    print(f"elevator stops on floor {floor}")

@flow(task_runner=ThreadPoolTaskRunner(max_workers=3))
def elevator():
    floors = []

    for floor in range(10, 0, -1):
        floors.append(stop_at_floor.submit(floor))

    wait(floors) # wait for the sequence of futures to complete

The max_workers parameter of the ThreadPoolTaskRunner controls the number of threads that the task runner will use to execute tasks concurrently.

​

Submit tasks to a task runner

When you use .submit() to submit a task to a task runner, the task runner creates a PrefectFuture for access to the state and result of the task.

A PrefectFuture is an object that provides:

• a reference to the result returned by the task
• a State indicating the current state of the task run

When you pass a future into a task, Prefect automatically waits for the “upstream” task (the one that the future references), to reach a final state before starting the downstream task.

This means that the downstream task won’t receive the PrefectFuture you passed as an argument. Instead, the downstream task receives the value that the upstream task returned.

For example:

from prefect import flow, task

@task
def say_hello(name):
    return f"Hello {name}!"

@task
def print_result(result):
    print(type(result))
    print(result)

@flow(name="hello-flow")
def hello_world():
    future = say_hello.submit("Marvin")
    print_result.submit(future).wait()

hello_world()

If we run this, we see that we only had to wait for the final print_result future as Prefect automatically resolved say_hello to a string.

​

Access results from submitted tasks

You can access the result of a future explicitly with the .result() method.

from prefect import flow, task

@task
def my_task():
    return 42

@flow
def my_flow():
    future = my_task.submit()
    result = future.result()
    print(result)

my_flow()

The .result() method waits for the task to complete before returning the result to the caller. If the task run fails, .result() will raise the task run’s exception. Disable this behavior with the raise_on_failure option:

from prefect import flow, task

@task
def my_task():
    return "I'm a task!"


@flow
def my_flow():
    future = my_task.submit()
    result = future.result(raise_on_failure=False)
    if future.state.is_failed():
        # `result` is an exception! handle accordingly
        ...
    else:
        # `result` is the expected return value of our task
        ...

​

Design considerations

When choosing how and when to achieve concurrency using task runners, consider the following:

1. Task granularity: The “proper” size for tasks depends on the nature and complexity of the work you’re doing, e.g. too many small tasks might create overhead - see Writing tasks for more.

2. Resource constraints: Be aware of environment limitations. Using .map can create many task instances very quickly, which might exceed your resource availability.

3. Data transfer: Large data passed between tasks can impact performance. Consider passing references to external storage when dealing with large datasets.

4. Parallelism: For true parallelism (rather than just concurrency), consider using a specialized task runner like the DaskTaskRunner or RayTaskRunner (or propose a new task runner type).

5. Beware of unsafe global state: Use of concurrency or parallelism features like .submit and .map must respect the underlying primitives to avoid unexpected behavior. For example, the default ThreadPoolTaskRunner runs each task in a separate thread, which means that any global state must be threadsafe. Similarly, DaskTaskRunner and RayTaskRunner are multi-process runners that require global state to be picklable.

​

Further reading

• Run tasks concurrently