Infrastructure Debugging

Before a flow run starts executing your code, it passes through several infrastructure stages: submission, scheduling, container startup, and process initialization. Previously, all of this was invisible; a run sat in Pending until it either started or failed.

Now, Prefect tracks every stage of the infrastructure lifecycle with dedicated states:

• Submitting: The worker is actively creating infrastructure (a Kubernetes Job, an ECS task, a local process)
• InfrastructurePending: Infrastructure exists while the flow run process is still waiting to start (for example, a pod is pulling images or waiting for node capacity)

These states appear in the UI timeline and are available through the API, so you always know whether a delay is caused by infrastructure provisioning or something else entirely.

When infrastructure fails, Prefect tells you that it failed, then shows you why and what to do about it.

Automatic failure diagnosis for Kubernetes and Amazon ECS inspects the actual infrastructure state and translates it into actionable guidance:

Wondering if your flow is running out of memory or maxing out CPU? Prefect collects CPU and memory metrics from flow run processes and displays them as real-time charts in the UI.

• Flow run detail page: Time-series charts for CPU utilization and memory usage appear in the Infrastructure panel, with peak values shown in the title bar for at-a-glance monitoring
• Deployment page: Summary cards show the highest CPU and memory usage across all recent flow runs, with a direct link to the run that produced the peak

Use these metrics to right-size your infrastructure, catch memory leaks early, and understand whether a failure was caused by resource exhaustion while staying in Prefect.

Some of the hardest failures to debug are the ones where a run crashes before it ever connects to Prefect. An OOM kill during import, a bad entrypoint, a missing dependency. In these cases, the flow run process never initializes its logging handler, so no logs reach Prefect.

When a Kubernetes pod or ECS task crashes before the flow run establishes connectivity, the observer automatically fetches the container’s stdout and stderr and forwards them as flow run logs. The Python traceback that explains what went wrong appears right alongside the rest of your run’s logs in the Prefect UI.

When flow runs queue up waiting for concurrency slots, it’s important to understand who is holding those slots and for how long. Prefect now surfaces concurrency utilization across the UI and CLI:

• Work pool and work queue pages show active slot counts so you can see utilization at a glance
• CLI commands (prefect work-pool slots, prefect work-queue slots) show which flow runs occupy each slot and how long they’ve been running
• Concurrency status endpoints provide programmatic access to slot occupancy data for custom dashboards and alerting

Enhanced lifecycle logging gives you a detailed narrative of what happens at every stage of a flow run’s execution:

• Workers report each step of infrastructure creation, from job submission to container startup, with timing information and error context
• Runners log process management details including subprocess creation, signal handling, and graceful shutdown sequences
• Pull steps log code retrieval progress and surface resolution hints when storage access fails (wrong credentials, missing buckets, network issues)

When something goes wrong, each component suggests concrete next steps. Instead of a generic “infrastructure exited with code 1,” you get a clear explanation and a path to resolution.