> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prefect.io/llms.txt
> Use this file to discover all available pages before exploring further.

# How to scale self-hosted Prefect

> Learn how to run multiple Prefect server instances for high availability and load distribution

Running multiple Prefect server instances enables high availability and distributes load across your infrastructure. This guide covers configuration and deployment patterns for scaling self-hosted Prefect.

## Requirements

Multi-server deployments require:

* PostgreSQL database version 14.9 or higher (SQLite does not support multi-server synchronization)
* Redis for event messaging
* Load balancer for API traffic distribution

## Architecture

A scaled Prefect deployment typically includes:

* **Multiple API server instances** - Handle UI and API requests
* **Background services** - Runs the scheduler, automation triggers, and other loop services. Can be scaled horizontally by running multiple instances coordinated through Redis
* **[PostgreSQL](https://www.postgresql.org/) database** - Stores all persistent data and synchronizes state across servers
* **[Redis](https://redis.io/)** - Distributes events between services and coordinates background service work
* **Load balancer** - Routes traffic to healthy API instances (e.g. [NGINX](https://www.f5.com/go/product/welcome-to-nginx) or [Traefik](https://doc.traefik.io/traefik/))

```mermaid theme={null}
%%{
  init: {
    'theme': 'neutral',
    'flowchart': {
      'curve' : 'linear',
      'rankSpacing': 120,
      'nodeSpacing': 80
    }
  }
}%%

flowchart TB
    %% Style definitions
    classDef userClass fill:#ede7f6db,stroke:#4527a0db,stroke-width:2px
    classDef lbClass fill:#e3f2fddb,stroke:#1565c0db,stroke-width:2px
    classDef apiClass fill:#1860f2db,stroke:#1860f2db,stroke-width:2px
    classDef bgClass fill:#7c3aeddb,stroke:#7c3aeddb,stroke-width:2px
    classDef dataClass fill:#16a34adb,stroke:#16a34adb,stroke-width:2px
    classDef workerClass fill:#f59e0bdb,stroke:#f59e0bdb,stroke-width:2px

    %% Nodes
    subgraph clients[Client Side]
        direction TB
        Users[Users / UI / API Clients]:::userClass
        Workers[Workers poll any available API server<br/>Process / K8s / Docker / Serverless]:::workerClass
    end

    LB[Load Balancer<br/>NGINX / HAProxy / ALB<br/>Port 4200]:::lbClass

    subgraph servers[Prefect Server Components]
        direction TB
        subgraph api[API Servers - Horizontal Scaling]
            direction LR
            API1[API Server 1<br/>--no-services]:::apiClass
            API2[API Server 2<br/>--no-services]:::apiClass
            API3[API Server N...<br/>--no-services]:::apiClass
        end

        subgraph bg[Background Services - Horizontal Scaling]
            direction LR
            BG1[Background Services 1<br/>prefect server services start]:::bgClass
            BG2[Background Services 2<br/>prefect server services start]:::bgClass
            BG3[Background Services N...<br/>prefect server services start]:::bgClass
        end
    end

    subgraph data[Data Layer]
        direction LR
        PG[(PostgreSQL<br/>• Flow/Task State<br/>• Configuration<br/>• History)]:::dataClass
        Redis[(Redis<br/>• Events<br/>• Automations<br/>• Real-time Updates)]:::dataClass
    end

    %% Connections
    Users --> |HTTPS| LB
    LB --> |Round Robin| api

    api --> |Read/Write| PG
    api --> |Publish| Redis

    bg --> |Read/Write| PG
    bg --> |Subscribe / Coordinate| Redis

    Workers -.-> |Poll Work| api
```

## Configuration

### Database setup

Configure PostgreSQL as your database backend:

```bash theme={null}
export PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://user:password@host:5432/prefect"
```

<Warning>
  PostgreSQL version 14.9 or higher is required for multi-server deployments. SQLite does not support the features needed for state synchronization across multiple servers.
</Warning>

### AWS RDS IAM Authentication

To use AWS IAM authentication for your PostgreSQL database (experimental):

1. **Install the AWS integration**:
   ```bash theme={null}
   pip install prefect-aws
   ```

2. **Create an IAM policy** with `rds-db:connect` permission and attach it to your IAM user/role.

3. **Enable plugins and IAM authentication**:
   ```bash theme={null}
   export PREFECT_PLUGINS_ENABLED=true
   export PREFECT_INTEGRATIONS_AWS_RDS_IAM_ENABLED=true
   # Optional: export PREFECT_INTEGRATIONS_AWS_RDS_IAM_REGION_NAME=us-east-1
   ```

4. **Configure your connection URL**:
   ```bash theme={null}
   export PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://iam_user@host:5432/prefect"
   ```

### Redis setup

Configure Redis as your server's message broker, cache, and lease storage:

```bash theme={null}
export PREFECT_MESSAGING_BROKER="prefect_redis.messaging"
export PREFECT_MESSAGING_CACHE="prefect_redis.messaging"
export PREFECT_SERVER_EVENTS_CAUSAL_ORDERING="prefect_redis.ordering"
export PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE="prefect_redis.lease_storage"
export PREFECT_REDIS_MESSAGING_HOST="redis-host"
export PREFECT_REDIS_MESSAGING_PORT="6379"
export PREFECT_REDIS_MESSAGING_DB="0"
```

If your Redis instance requires authentication, you may configure a username and password:

```bash theme={null}
export PREFECT_REDIS_MESSAGING_USERNAME="marvin"
export PREFECT_REDIS_MESSAGING_PASSWORD="dontpanic!"
```

For Redis instances that require an encrypted connection, you can enable SSL/TLS:

```bash theme={null}
export PREFECT_REDIS_MESSAGING_SSL="true"
```

Alternatively, configure the Redis connection with a single URL instead of individual fields. When `PREFECT_REDIS_MESSAGING_URL` is set, it takes precedence and the individual host, port, db, username, password, and SSL fields are ignored:

```bash theme={null}
export PREFECT_REDIS_MESSAGING_URL="redis://username:password@redis-host:6379/0"
```

Use `rediss://` for TLS connections:

```bash theme={null}
export PREFECT_REDIS_MESSAGING_URL="rediss://redis-host:6379/0"
```

#### Docket URL for background services

Prefect uses [Docket](https://github.com/chrisguidry/docket) to coordinate background services like the scheduler, late run detection, and automation triggers. By default, Docket uses in-memory storage (`memory://`), which only works for single-server deployments.

For high-availability deployments, configure Docket to use Redis:

```bash theme={null}
export PREFECT_SERVER_DOCKET_URL="redis://redis-host:6379/0"
```

If your Redis instance requires authentication:

```bash theme={null}
export PREFECT_SERVER_DOCKET_URL="redis://username:password@redis-host:6379/0"
```

For Redis instances that require SSL/TLS:

```bash theme={null}
export PREFECT_SERVER_DOCKET_URL="rediss://redis-host:6379/0"
```

<Note>
  The Docket URL can use the same Redis instance as the messaging configuration above, but you may use a different database number (e.g., `/1` instead of `/0`) to keep the data separate.
</Note>

### Service separation

For optimal performance, run API servers and background services separately:

**API servers** (multiple instances):

```bash theme={null}
prefect server start --host 0.0.0.0 --port 4200 --no-services
```

**Background services**:

```bash theme={null}
prefect server services start
```

<Tip>
  For high-volume deployments, consider reducing the event retention period from the default 7 days to prevent rapid database growth. See [database maintenance](/v3/advanced/database-maintenance#configure-event-retention) for configuration details.
</Tip>

### Running multiple background services

For high availability and throughput, you can run multiple `prefect server services start` processes in parallel. Prefect uses [Docket](https://github.com/chrisguidry/docket) (backed by Redis) to coordinate work across background service processes so that periodic work (like scheduling, late run detection, and automation trigger evaluation) runs exactly once per interval even when multiple processes are running.

To run multiple background service processes:

1. Configure `PREFECT_SERVER_DOCKET_URL` to point at a shared Redis instance (see [Docket URL for background services](#docket-url-for-background-services)). The default in-memory backend (`memory://`) is only safe for a single process.
2. Ensure every background service process connects to the same PostgreSQL database, Redis messaging instance, and Docket URL as the API servers.
3. Start one `prefect server services start` process per replica. Each replica runs the same set of enabled services; Docket ensures only one replica picks up each scheduled run.

<Warning>
  Do not run multiple background service processes without configuring `PREFECT_SERVER_DOCKET_URL` to use Redis. With the default `memory://` backend each process schedules its own work independently, which causes duplicate scheduled runs, duplicate automation actions, and other correctness problems.
</Warning>

#### Running specific services on dedicated processes

All services run together in a single `prefect server services start` process by default. To dedicate a process to a specific subset of services (for example, to scale a noisy neighbor independently), disable the services you don't want to run on that process with their `*_ENABLED` environment variable.

List all services and their enable/disable environment variable:

```bash theme={null}
prefect server services ls
```

For example, to run a process that only handles the scheduler and late run detection:

```bash theme={null}
export PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false
export PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED=false
export PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=false
export PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=false
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=false
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=false
# ...disable any other services you don't want on this process
prefect server services start
```

Then run another process with the complementary set of services enabled. As long as every enabled service is running on at least one process (and all processes share the same Docket Redis), every enabled service continues to operate.

<Note>
  Some services maintain their own at-least-once semantics at the database or Redis level rather than relying on Docket's run-once guarantee. Running multiple processes with the same service enabled is supported, but starting with one process per service (and scaling up only when you observe a specific bottleneck) keeps operations simple.
</Note>

### Database migrations

Disable automatic migrations in multi-server deployments:

```bash theme={null}
export PREFECT_API_DATABASE_MIGRATE_ON_START="false"
```

Run migrations separately before deployment:

```bash theme={null}
prefect server database upgrade -y
```

### Load balancer configuration

Configure health checks for your load balancer:

* **Health endpoint**: `/api/health`
* **Expected response**: HTTP 200 with JSON `{"status": "healthy"}`
* **Check interval**: 5-10 seconds

Example NGINX configuration:

```nginx theme={null}
upstream prefect_api {
    least_conn;
    server prefect-api-1:4200 max_fails=3 fail_timeout=30s;
    server prefect-api-2:4200 max_fails=3 fail_timeout=30s;
    server prefect-api-3:4200 max_fails=3 fail_timeout=30s;
}

server {
    listen 4200;

    location /api/health {
        proxy_pass http://prefect_api;
        proxy_connect_timeout 1s;
        proxy_read_timeout 1s;
    }

    location / {
        proxy_pass http://prefect_api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
```

### Reverse proxy configuration

When hosting Prefect behind a reverse proxy, ensure proper header forwarding:

```nginx theme={null}
server {
    listen 80;
    server_name prefect.example.com;

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name prefect.example.com;

    ssl_certificate /path/to/ssl/certificate.pem;
    ssl_certificate_key /path/to/ssl/certificate_key.pem;

    location /api {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;

        # WebSocket support
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Authentication headers
        proxy_set_header Authorization $http_authorization;
        proxy_pass_header Authorization;

        proxy_pass http://prefect_api;
    }

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://prefect_api;
    }
}
```

#### UI proxy settings

When self-hosting the UI behind a proxy:

* `PREFECT_UI_API_URL`: Connection URL from UI to API
* `PREFECT_UI_SERVE_BASE`: Base URL path to serve the UI
* `PREFECT_UI_URL`: URL for clients to access the UI

#### SSL certificates

For self-signed certificates:

1. Add certificate to system bundle and set:
   ```bash theme={null}
   export SSL_CERT_FILE=/path/to/certificate.pem
   ```

2. Or disable verification (testing only):
   ```bash theme={null}
   export PREFECT_API_TLS_INSECURE_SKIP_VERIFY=True
   ```

#### Environment proxy settings

Prefect respects standard proxy environment variables:

```bash theme={null}
export HTTPS_PROXY=http://proxy.example.com:8080
export HTTP_PROXY=http://proxy.example.com:8080
export NO_PROXY=localhost,127.0.0.1,.internal
```

## Deployment examples

### Docker Compose

<Accordion title="3 API servers, 2 background services, postgres, redis">
  ```yaml theme={null}
  services:
    postgres:
      image: postgres:15
      environment:
        POSTGRES_USER: prefect
        POSTGRES_PASSWORD: prefect
        POSTGRES_DB: prefect
      volumes:
        - postgres_data:/var/lib/postgresql/data
      healthcheck:
        test: pg_isready -h localhost -U $$POSTGRES_USER
        interval: 2s
        timeout: 5s
        retries: 15

    redis:
      image: redis:7

    migrate:
      image: prefecthq/prefect:3-latest
      depends_on:
        postgres:
          condition: service_healthy
      command: prefect server database upgrade -y
      environment:
        PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect

    prefect-api:
      image: prefecthq/prefect:3-latest
      depends_on:
        migrate:
          condition: service_completed_successfully
        postgres:
          condition: service_healthy
        redis:
          condition: service_started
      deploy:
        replicas: 3
      command: prefect server start --host 0.0.0.0 --no-services
      environment:
        PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect
        PREFECT_API_DATABASE_MIGRATE_ON_START: "false"
        PREFECT_MESSAGING_BROKER: prefect_redis.messaging
        PREFECT_MESSAGING_CACHE: prefect_redis.messaging
        PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
        PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
        PREFECT_REDIS_MESSAGING_HOST: redis
        PREFECT_REDIS_MESSAGING_PORT: "6379"
        PREFECT_SERVER_DOCKET_URL: redis://redis:6379/1
      ports:
        - "4200-4202:4200"  # Maps to different ports for each replica

    prefect-background:
      image: prefecthq/prefect:3-latest
      depends_on:
        migrate:
          condition: service_completed_successfully
        postgres:
          condition: service_healthy
        redis:
          condition: service_started
      deploy:
        replicas: 2
      command: prefect server services start
      environment:
        PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect
        PREFECT_API_DATABASE_MIGRATE_ON_START: "false"
        PREFECT_MESSAGING_BROKER: prefect_redis.messaging
        PREFECT_MESSAGING_CACHE: prefect_redis.messaging
        PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
        PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
        PREFECT_REDIS_MESSAGING_HOST: redis
        PREFECT_REDIS_MESSAGING_PORT: "6379"
        PREFECT_SERVER_DOCKET_URL: redis://redis:6379/1

  volumes:
    postgres_data:
  ```
</Accordion>

<Tip>
  Deploying Prefect self-hosted somehow else? Consider [opening a PR](/contribute/docs-contribute) to add your deployment pattern to this guide.
</Tip>

## Operations

### Migration considerations

#### Handling large databases

When running migrations on large database instances (especially where tables like `events`, `flow_runs`, or `task_runs` can reach millions of rows), the default database timeout of 10 seconds may not be sufficient for creating indexes.

If you encounter a `TimeoutError` during migrations, increase the database timeout:

```bash theme={null}
# Set timeout to 10 minutes (adjust based on your database size)
export PREFECT_API_DATABASE_TIMEOUT=600

# Then run the migration
prefect server database upgrade -y
```

For Docker deployments:

```bash theme={null}
docker run -e PREFECT_API_DATABASE_TIMEOUT=600 prefecthq/prefect:latest prefect server database upgrade -y
```

<Note>
  Index creation time scales with table size. A database with millions of events may require 30+ minutes for some migrations. If a migration fails due to timeout, you may need to manually clean up any partially created indexes before retrying.
</Note>

#### Recovering from failed migrations

If a migration times out while creating indexes, you may need to manually complete it. For example, if migration `7a73514ca2d6` fails:

1. First, check which indexes were partially created:
   ```sql theme={null}
   SELECT indexname FROM pg_indexes WHERE tablename = 'events' AND indexname LIKE 'ix_events%';
   ```

2. Manually create the missing indexes using `CONCURRENTLY` to avoid blocking:
   ```sql theme={null}
   -- Drop any partial indexes from the failed migration
   DROP INDEX IF EXISTS ix_events__event_related_occurred;
   DROP INDEX IF EXISTS ix_events__related_resource_ids;

   -- Create the new indexes
   CREATE INDEX CONCURRENTLY ix_events__related_gin ON events USING gin(related);
   CREATE INDEX CONCURRENTLY ix_events__event_occurred ON events (event, occurred);
   CREATE INDEX CONCURRENTLY ix_events__related_resource_ids_gin ON events USING gin(related_resource_ids);
   ```

3. Mark the migration as complete:
   ```sql theme={null}
   UPDATE alembic_version SET version_num = '7a73514ca2d6';
   ```

<Warning>
  Only use manual recovery if increasing the timeout and retrying the migration doesn't work. Always verify the correct migration version and index definitions from the migration files.
</Warning>

### Monitoring

Monitor your multi-server deployment:

* **Database connections**: Watch for connection pool exhaustion
* **Redis memory**: Ensure adequate memory for message queues
* **API response times**: Track latency across different endpoints
* **Background service lag**: Monitor time between event creation and processing

### Best practices

1. **Start with 2-3 API instances** and scale based on load
2. **Use connection pooling** to manage database connections efficiently
3. **Monitor extensively** before scaling further (e.g. [Prometheus](https://prometheus.io/) + [Grafana](https://grafana.com/) or [Logfire](https://logfire.pydantic.dev/docs/why/))
4. **Test failover scenarios** regularly

## Further reading

* [Database maintenance](/v3/advanced/database-maintenance) - Monitor table sizes, configure event retention, and manage data growth
* [Server concepts](/v3/concepts/server)
* Deploy [Helm charts](/v3/advanced/server-helm) for Kubernetes