Prefect Collections¶
Prefect Collections are groupings of pre-built tasks and flows used to quickly build data flows with Prefect.
Collections are grouped around the services with which they interact. For example, to download data from an S3 bucket, you could use the s3_download
task from the prefect-aws
collection, or if you want to send a Slack message as part of your flow you could use the send_message
task from the prefect-slack
collection.
By using Prefect Collections, you can reduce the amount of boilerplate code that you need to write for interacting with common services, and focus on outcome you're seeking to achieve.
Usage¶
To use a Prefect Collection, first install the collection via pip
. As an example, to use prefect-aws
:
pip install prefect-aws
The AWS tasks and flows in that collection can then be imported and called within your flow:
from prefect import flow
from prefect_aws import AwsCredentials
from prefect_aws.secrets_manager import read_secret
@flow
def connect_to_database():
aws_credentials = AwsCredentials(
aws_access_key_id="access_key_id",
aws_secret_access_key="secret_access_key"
)
secret_value = read_secret(
secret_name="db_password",
aws_credentials=aws_credentials
)
# Use secret_value to connect to a database
Available Collections¶
To see the list of available Prefect Collections and links to each collection's GitHub repository and documentation, please refer to the Collection Catalog in the Prefect documentation.
Contributing Collections¶
Anyone can create and share a Prefect Collection and we encourage anyone interested in creating a collection to do so!
Generating a project¶
To help you get started with your collection, we've created a template that gives the tools you need to create and publish your collection.
To generate a collection from the template, run the following:
# 1. Install cookiecutter
pip install cookiecutter
# 2. Generate a Prefect Collection project
cookiecutter https://github.com/PrefectHQ/prefect-collection-template
After your project has been generated, refer to the MAINTAINER.md in the generated project for information about developing your collection.
Listing in the Collections Catalog¶
To list your collection in the Prefect Collections Catalog, submit a PR to the Prefect repository adding a file to the docs/collections/catalog
directory with details about your collection. Please use TEMPLATE.yaml
in that folder as guide.