Prefect Core is a new kind of workflow management system. We started with a simple premise:
Your code probably works. But sometimes it doesn't.
When your code works, you might not even need a workflow system. We call that positive engineering. It's only when things go wrong that a system like Prefect starts to be valuable. That's negative engineering: all the little details that guarantee your code either achieves its goal, or fails successfully. In this way, workflow systems are actually risk management tools, like insurance: there when you need them, invisible when you don't.
And yet, we don't see a single tool designed that way. Other workflow systems seem to believe that they're actually positive engineering tools, somehow enabling users to do things they couldn't do otherwise. As a result, they feel no shame in asking users to generate yet another config file, or contort code into a convoluted DAG structure. Prefect already knows you can write incredible code; it just wants to make sure it works.
Prefect takes your code and transforms it into a robust, distributed pipeline. You can continue to use your existing tools, languages, infrastructure, and scripts. Prefect builds a rich DAG structure, but in a way that respects positive engineering and doesn't inhibit it. You can use Prefect's functional API to transform scripts with minimal hooks; or you can access the deferred computational graph directly; or any combination thereof. It's up to you.
The most common thing we hear about negative engineering is: "This should be easy!"
Now it can be.
- The Prefect Team