prefect-redis - Prefect

Integrations to extend Prefect’s functionality with Redis.

Getting started

Install `prefect-redis`

The following command will install a version of prefect-redis compatible with your installed version of prefect. If you don’t already have prefect installed, it will install the newest version of prefect as well.

pip install "prefect[redis]"

Upgrade to the latest versions of prefect and prefect-redis:

pip install -U "prefect[redis]"

Register newly installed block types

prefect block register -m prefect_redis

Connecting with a URL (including Redis Sentinel)

Every prefect-redis connection point — the server messaging client, the worker cleanup queue, the RedisLockManager, and the RedisDatabase block — accepts a full connection URL in addition to the individual host/port/db/username/password/ssl fields. When a URL is provided it is authoritative and the scalar fields are ignored. The following schemes are supported:

Scheme	Behavior
`redis://`	Single-node TCP connection
`rediss://`	Single-node TLS connection
`redis+sentinel://`	Redis Sentinel discovery (TCP data nodes)
`rediss+sentinel://`	Redis Sentinel discovery (TLS data nodes)

The Sentinel schemes accept a comma-separated list of Sentinel members and a master group name, and the resulting client resolves the current master through the Sentinel daemons and follows failover automatically:

redis+sentinel://[user:pass@]sentinel-a:26379,sentinel-b:26379/<master-name>[/<db>][?params]

The sentinel_username and sentinel_password query parameters configure authentication to the Sentinel daemons separately from the data nodes. All other query parameters are treated as standard redis-py connection options (socket_timeout, max_connections, health_check_interval, …) and apply to the data-node connections. On the rediss+sentinel:// scheme the ssl_* options also apply to the Sentinel daemon connections, so a single private CA works for the whole topology, e.g. rediss+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster?ssl_ca_certs=/etc/redis/ca.pem. Query options take precedence over the equivalent Prefect settings (for example PREFECT_REDIS_MESSAGING_SOCKET_TIMEOUT), just as they do with redis-py’s Redis.from_url.

Server messaging and worker cleanup queue

Point the Prefect server’s Redis messaging and its worker cleanup queue at a Sentinel topology with one URL per subsystem:

export PREFECT_REDIS_MESSAGING_URL="redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster"
export PREFECT_REDIS_WORKER_CLEANUP_QUEUE_URL="redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster"

Both settings also accept single-node redis:// / rediss:// URLs.

Lock manager

from prefect_redis import RedisLockManager

lock_manager = RedisLockManager(
    connection_url="redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster"
)

`RedisDatabase` block

from prefect_redis import RedisDatabase

block = RedisDatabase.from_connection_string(
    "redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster"
)
block.save("BLOCK_NAME")

Resources

Refer to the SDK documentation to explore all the capabilities of prefect-redis.

​Getting started

​Install prefect-redis

​Register newly installed block types

​Connecting with a URL (including Redis Sentinel)

​Server messaging and worker cleanup queue

​Lock manager

​RedisDatabase block

​Resources