How to scale self-hosted Prefect

Running multiple Prefect server instances enables high availability and distributes load across your infrastructure. This guide covers configuration and deployment patterns for scaling self-hosted Prefect.

Requirements

Multi-server deployments require:

PostgreSQL database version 14.9 or higher (SQLite does not support multi-server synchronization)
Redis for event messaging
Load balancer for API traffic distribution

Architecture

A scaled Prefect deployment typically includes:

Multiple API server instances - Handle UI and API requests
Background services - Runs the scheduler, automation triggers, and other loop services
PostgreSQL database - Stores all persistent data and synchronizes state across servers
Redis - Distributes events between services
Load balancer - Routes traffic to healthy API instances (e.g. NGINX or Traefik)

Configuration

Database setup

Configure PostgreSQL as your database backend:

export PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://user:password@host:5432/prefect"

PostgreSQL version 14.9 or higher is required for multi-server deployments. SQLite does not support the features needed for state synchronization across multiple servers.

Redis setup

Configure Redis as your message broker:

export PREFECT_MESSAGING_BROKER="prefect_redis.messaging"
export PREFECT_MESSAGING_CACHE="prefect_redis.messaging"
export PREFECT_REDIS_MESSAGING_HOST="redis-host"
export PREFECT_REDIS_MESSAGING_PORT="6379"
export PREFECT_REDIS_MESSAGING_DB="0"

If your Redis instance requires authentication, you may configure a username and password:

export PREFECT_REDIS_MESSAGING_USERNAME="marvin"
export PREFECT_REDIS_MESSAGING_PASSWORD="dontpanic!"

For Redis instances that require an encrypted connection, you can enable SSL/TLS:

export PREFECT_REDIS_MESSAGING_SSL="true"

For multi-server deployments with Redis, configure causal ordering to ensure proper task state synchronization:

export PREFECT_SERVER_EVENTS_CAUSAL_ORDERING="prefect_redis.ordering"

Each server instance automatically generates a unique consumer name to prevent message delivery conflicts.

Service separation

For optimal performance, run API servers and background services separately: API servers (multiple instances):

prefect server start --host 0.0.0.0 --port 4200 --no-services

Background services:

prefect server services start

Database migrations

Disable automatic migrations in multi-server deployments:

export PREFECT_API_DATABASE_MIGRATE_ON_START="false"

Run migrations separately before deployment:

prefect server database upgrade -y

Load balancer configuration

Configure health checks for your load balancer:

Health endpoint: /api/health
Expected response: HTTP 200 with JSON {"status": "healthy"}
Check interval: 5-10 seconds

Example NGINX configuration:

upstream prefect_api {
    least_conn;
    server prefect-api-1:4200 max_fails=3 fail_timeout=30s;
    server prefect-api-2:4200 max_fails=3 fail_timeout=30s;
    server prefect-api-3:4200 max_fails=3 fail_timeout=30s;
}

server {
    listen 4200;

    location /api/health {
        proxy_pass http://prefect_api;
        proxy_connect_timeout 1s;
        proxy_read_timeout 1s;
    }

    location / {
        proxy_pass http://prefect_api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Reverse proxy configuration

When hosting Prefect behind a reverse proxy, ensure proper header forwarding:

server {
    listen 80;
    server_name prefect.example.com;

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name prefect.example.com;

    ssl_certificate /path/to/ssl/certificate.pem;
    ssl_certificate_key /path/to/ssl/certificate_key.pem;

    location /api {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;

        # WebSocket support
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Authentication headers
        proxy_set_header Authorization $http_authorization;
        proxy_pass_header Authorization;

        proxy_pass http://prefect_api;
    }

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://prefect_api;
    }
}

UI proxy settings

When self-hosting the UI behind a proxy:

PREFECT_UI_API_URL: Connection URL from UI to API
PREFECT_UI_SERVE_BASE: Base URL path to serve the UI
PREFECT_UI_URL: URL for clients to access the UI

SSL certificates

For self-signed certificates:

Add certificate to system bundle and set:

export SSL_CERT_FILE=/path/to/certificate.pem

Or disable verification (testing only):

export PREFECT_API_TLS_INSECURE_SKIP_VERIFY=True

Environment proxy settings

Prefect respects standard proxy environment variables:

export HTTPS_PROXY=http://proxy.example.com:8080
export HTTP_PROXY=http://proxy.example.com:8080
export NO_PROXY=localhost,127.0.0.1,.internal

Deployment examples

Docker Compose

3 servers, 1 background service, postgres, redis

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: prefect
      POSTGRES_DB: prefect
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready -h localhost -U $$POSTGRES_USER
      interval: 2s
      timeout: 5s
      retries: 15

  redis:
    image: redis:7

  migrate:
    image: prefecthq/prefect:3-latest
    depends_on:
      postgres:
        condition: service_healthy
    command: prefect server database upgrade -y
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect

  prefect-api:
    image: prefecthq/prefect:3-latest
    depends_on:
      migrate:
        condition: service_completed_successfully
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    deploy:
      replicas: 3
    command: prefect server start --host 0.0.0.0 --no-services
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect
      PREFECT_API_DATABASE_MIGRATE_ON_START: "false"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
      PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: "6379"
    ports:
      - "4200-4202:4200"  # Maps to different ports for each replica

  prefect-background:
    image: prefecthq/prefect:3-latest
    depends_on:
      migrate:
        condition: service_completed_successfully
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    command: prefect server services start
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect
      PREFECT_API_DATABASE_MIGRATE_ON_START: "false"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
      PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: "6379"

volumes:
  postgres_data:

Deploying Prefect self-hosted somehow else? Consider opening a PR to add your deployment pattern to this guide.

Operations

Migration considerations

Handling large databases

When running migrations on large database instances (especially where tables like events, flow_runs, or task_runs can reach millions of rows), the default database timeout of 10 seconds may not be sufficient for creating indexes. If you encounter a TimeoutError during migrations, increase the database timeout:

# Set timeout to 10 minutes (adjust based on your database size)
export PREFECT_API_DATABASE_TIMEOUT=600

# Then run the migration
prefect server database upgrade -y

For Docker deployments:

docker run -e PREFECT_API_DATABASE_TIMEOUT=600 prefecthq/prefect:latest prefect server database upgrade -y

Index creation time scales with table size. A database with millions of events may require 30+ minutes for some migrations. If a migration fails due to timeout, you may need to manually clean up any partially created indexes before retrying.

Recovering from failed migrations

If a migration times out while creating indexes, you may need to manually complete it. For example, if migration 7a73514ca2d6 fails:

First, check which indexes were partially created:

SELECT indexname FROM pg_indexes WHERE tablename = 'events' AND indexname LIKE 'ix_events%';

Manually create the missing indexes using CONCURRENTLY to avoid blocking:

-- Drop any partial indexes from the failed migration
DROP INDEX IF EXISTS ix_events__event_related_occurred;
DROP INDEX IF EXISTS ix_events__related_resource_ids;

-- Create the new indexes
CREATE INDEX CONCURRENTLY ix_events__related_gin ON events USING gin(related);
CREATE INDEX CONCURRENTLY ix_events__event_occurred ON events (event, occurred);
CREATE INDEX CONCURRENTLY ix_events__related_resource_ids_gin ON events USING gin(related_resource_ids);

Mark the migration as complete:

UPDATE alembic_version SET version_num = '7a73514ca2d6';

Only use manual recovery if increasing the timeout and retrying the migration doesn’t work. Always verify the correct migration version and index definitions from the migration files.

Monitoring

Monitor your multi-server deployment:

Database connections: Watch for connection pool exhaustion
Redis memory: Ensure adequate memory for message queues
API response times: Track latency across different endpoints
Background service lag: Monitor time between event creation and processing

Best practices

Start with 2-3 API instances and scale based on load
Use connection pooling to manage database connections efficiently
Monitor extensively before scaling further (e.g. Prometheus + Grafana or Logfire)
Test failover scenarios regularly

Workflows

Automations

Workflow Infrastructure

Platform

Extensibility

How to scale self-hosted Prefect

Requirements

Architecture

Configuration

Database setup

Redis setup

Service separation

Database migrations

Load balancer configuration

Reverse proxy configuration

UI proxy settings

SSL certificates

Environment proxy settings

Deployment examples

Docker Compose

Operations

Migration considerations

Handling large databases

Recovering from failed migrations

Monitoring

Best practices

Further reading

Workflows

Automations

Workflow Infrastructure

Platform

Extensibility

​Requirements

​Architecture

​Configuration

​Database setup

​Redis setup

​Service separation

​Database migrations

​Load balancer configuration

​Reverse proxy configuration

​UI proxy settings

​SSL certificates

​Environment proxy settings

​Deployment examples

​Docker Compose

​Operations

​Migration considerations

​Handling large databases

​Recovering from failed migrations

​Monitoring

​Best practices

​Further reading

Requirements

Architecture

Configuration

Database setup

Redis setup

Service separation

Database migrations

Load balancer configuration

Reverse proxy configuration

UI proxy settings

SSL certificates

Environment proxy settings

Deployment examples

Docker Compose

Operations

Migration considerations

Handling large databases

Recovering from failed migrations

Monitoring

Best practices

Further reading