Introduction
Prefect is an open-source orchestration engine that turns your Python functions into production-grade data pipelines with minimal friction. You can build and schedule workflows in pure Python—no DSLs or complex config files—and run them anywhere. Prefect handles the heavy lifting for you out of the box: automatic state tracking, failure handling, real-time monitoring, and more.
Why? In 2018, we set out to solve a problem many of you know too well: orchestrating data workflows shouldn’t feel like pulling teeth (e.g. writing DAGs in YAML). Prefect was designed to handle the challenges that tools like Airflow struggled with: dynamic workflows, modern infrastructure, and the complexity of today’s data pipelines. Since then, it’s become a go-to tool for everyone from startups to Fortune 100 companies. Prefect helps you ship production-ready workflows without the headaches, so you can spend more time building and less time debugging.
Essential features
| Feature | Description |
|---|---|
| Pythonic | Write workflows in native Python—no DSLs, YAML, or special syntax. Full support for type hints, async/await, and modern Python patterns. Use your existing IDE, debugger, and testing tools. |
| State & Recovery | Robust state management that tracks success, failure, and retry states. Resume interrupted runs from the last successful point, and cache expensive computations to avoid unnecessary rework. |
| Flexible & Portable Execution | Start flows locally for easy development, then deploy them anywhere—from a single process to containers, Kubernetes, or cloud services—without locking into a vendor. Infrastructure is defined by code (not just configuration), making it simple to scale or change environments. |
| Event-Driven | Trigger flows on schedules, external events, or via API. Pause flows for human intervention or approval. Chain flows together based on states, conditions, or any custom logic. |
| Dynamic Runtime | Create tasks dynamically at runtime based on actual data or conditions. Easily spawn new tasks and branches during execution for truly data-driven workflows. |
| Modern UI | Real-time flow run monitoring, logging, and state tracking through an intuitive interface. View dependency graphs and DAGs automatically—just run your flow and open the UI. |
| CI/CD First | Test and simulate flows like normal Python code, giving you fast feedback during development. Integrate seamlessly into your existing CI/CD pipeline for automated testing and deployment. |
Get started
Quickstart
Quickly create your first deployable workflow tracked by Prefect.
Prefect Cloud
Supercharge Prefect with enhanced governance, security, and performance capabilities.
Upgrade to Prefect 3
Upgrade from Prefect 2 to Prefect 3 to get the latest features and performance enhancements.
Tutorials
Schedule a flow
Get a workflow off of your laptop and run it on remote infrastructure on a schedule.
Build a data pipeline
Build a resilient and performant data pipeline with Prefect’s primitives.
Extract data from websites
Handle data dependencies and use pagination to efficiently extract data from websites.
Set up a platform
Use Prefect as a platform for your teams’ data pipelines.
Debug a data pipeline
Learn how to debug flow runs that fail, crash, or hang.
Send alerts on failure
Set up an email alert to notify your teams about failures.
Start building
Develop
Write, run, configure, and observe workflows and their tasks, results, artifacts, and more.
Deploy
Run workflows in local processes or deploy them to dynamically provisioned infrastructure.
Automate
Enable workflows to react to their environment with events, automations, and webhooks.
Mini-history of Prefect
2018-2021: Our story begins in 2018, when we introduced the idea that workflow orchestration should be Pythonic. Inspired by distributed tools like Dask, and building on the experience of our founder, Jeremiah Lowin (a PMC member of Apache Airflow), we created a system based on simple Python decorators for tasks and flows. But what made Prefect truly special was our introduction of task mapping—a feature that would later become foundational to our dynamic execution capabilities (and eventually imitated by other orchestration SDKs).
2022: Prefect’s 2.0 release became inevitable once we recognized that real-world workflows don’t always fit into neat, pre-planned DAG structures: sometimes you need to update a job definition based on runtime information, for example by skipping a branch of your workflow. So we removed a key constraint that workflows be written explicitly as DAGs, fully embracing native Python control flow—if/else conditionals, while loops-everything that makes Python…Python.
2023-present: With our release of Prefect 3.0 in 2024, we fully embraced these dynamic patterns by open-sourcing our events and automations backend, allowing users to natively represent event-driven workflows and gain additional observability into their execution. Prefect 3.0 also unlocked a leap forward in performance, improving the runtime overhead of Prefect by up to 90%.
Join our community
Join Prefect’s vibrant community of nearly 30,000 engineers to learn with others and share your knowledge!
Was this page helpful?