Configure logging
Configure logging on flows and tasks for monitoring, troubleshooting, and auditing.
When you run a flow or a task, Prefect automatically emits a standard set of logs that you can inspect in the UI, in the CLI, or through the API. Additionally, you can emit custom log messages during flow and task runs to capture specific events or information important to your workflow.
Prefect logging is configured automatically anytime you execute a flow or a task. The standard set of logs include:
- information about when runs are created or renamed
- information about run state changes
- the tracebacks of any errors that arise during execution
Prefect loggers
Prefect exposes a set of loggers that you can use to emit your own custom logs.
To use a Prefect logger, import get_run_logger
from the prefect.logging
module.
This function returns a logger instance that is aware of the current flow or task run context, allowing for more detailed and contextual logging.
This allows you to explore logs in the UI or API based on relevant run information such as run ID and run name.
get_run_logger()
can only be used in the context of a flow or task.
To use a normal Python logger anywhere with your same configuration, use get_logger()
from prefect.logging
.
The logger retrieved with get_logger()
will not send log records to the Prefect API.
Logging in flows and tasks
To log from a flow or a task, retrieve a logger instance with get_run_logger()
, then call the standard Python
logging methods:
Logging print statements
Prefect provides the log_prints
option on both flows and tasks to enable the automatic logging of print
statements.
When log_prints=True
for a given task or flow, the Python built-in print
is patched to redirect to the Prefect
logger for the scope of that task or flow.
These logs are emitted at the INFO
level.
By default, task runs and nested flow runs inherit the log_prints
setting from their parent flow run, unless opted out with their
own explicit log_prints
setting.
Outputs:
Using log_prints=False
at the task level outputs:
You can configure this behavior as the default for all Prefect flow and task runs through the
PREFECT_LOGGING_LOG_PRINTS
setting:
Logging configuration
Prefect relies on the standard Python implementation of logging configuration.
The full specification of the default logging configuration for any version of Prefect can always be inspected here.
The default logging level is INFO
.
Customize logging configuration
Prefect provides several settings to configure the logging level and individual loggers.
Any value in Prefect’s logging configuration file can be overridden through
a Prefect setting of the form PREFECT_LOGGING_[PATH]_[TO]_[KEY]=value
corresponding to the nested address of the field you are configuring.
For example, to change the default logging level for flow runs but not task runs, update your profile with:
or set the corresponding environment variable:
You can also configure the “root” Python logger. The root logger receives logs from all loggers unless they
explicitly opt out by disabling propagation. By default, the root logger is configured to output WARNING
level logs
to the console. As with other logging settings, you can override this from the environment or in the logging configuration
file. For example, you can change the level with the PREFECT_LOGGING_ROOT_LEVEL
environment variable.
In some situations you may want to completely overhaul the Prefect logging configuration by providing your own logging.yml
file.
You can create your own version of logging.yml
in one of two ways:
- Create a
logging.yml
file in yourPREFECT_HOME
directory (default is~/.prefect
). - Specify a custom path to your
logging.yml
file using thePREFECT_LOGGING_SETTINGS_PATH
setting.
If Prefect cannot find the logging.yml
file at the specified location, it will fall back to using the default logging configuration.
See the Python Logging configuration
documentation for more information about the configuration options and syntax used by logging.yml
.
As with all Prefect settings, logging settings are loaded at runtime. This means that to customize Prefect logging in a remote environment requires setting the appropriate environment variables and/or profile in that environment.
Formatters
Prefect log formatters specify the format of log messages.
The default formatting for task and flow run records is
"%(asctime)s.%(msecs)03d | %(levelname)-7s | Task run %(task_run_name)r - %(message)s"
for tasks and
similarly "%(asctime)s.%(msecs)03d | %(levelname)-7s | Flow run %(flow_run_name)r - %(message)s"
for flows.
The variables available to interpolate in log messages vary by logger. In addition to the run context, message string, and any keyword arguments, flow and task run loggers have access to additional variables.
The flow run logger has the following variables available for formatting:
flow_run_name
flow_run_id
flow_name
The task run logger has the following variables available for formatting:
task_run_id
flow_run_id
task_run_name
task_name
flow_run_name
flow_name
You can specify custom formatting by setting the relevant environment variable or by modifying the formatter in a custom logging.yml
file as
described earlier.
For example, the following changes the formatting for the flow runs formatter:
The resulting messages, using the flow run ID instead of name, look like this:
Styles
By default, Prefect highlights specific keywords in the console logs with a variety of colors.
You can toggle highlighting on/off with the PREFECT_LOGGING_COLORS
setting:
You can also change what gets highlighted and even adjust the colors by updating the styles - see the styles
section of the Prefect logging configuration file for available keys.
Note that these style settings only impact the display within a terminal, not the Prefect UI.
You can even build your own handler with a custom highlighter. For example, to additionally highlight emails:
- Copy and paste the following code into
my_package_or_module.py
(rename as needed) in the same directory as the flow run script; or ideally as part of a Python package so it’s available insite-packages
and accessible anywhere within your environment.
- Update
~/.prefect/logging.yml
to usemy_package_or_module.CustomConsoleHandler
and additionally reference the base_style and named expression:log.email
.
- On your next flow run, text that looks like an email is highlighted. For example,
my@email.com
is colored in magenta below:
Apply markup in logs
To use Rich’s markup in Prefect logs, first
configure PREFECT_LOGGING_MARKUP
:
The following will highlight “fancy” in red:
Inaccurate logs could result
If enabled, strings that contain square brackets may be
inaccurately interpreted and lead to incomplete output. For example, DROP TABLE [dbo].[SomeTable];"
outputs
DROP TABLE .[SomeTable];
.
Include logs from other libraries
By default, Prefect won’t capture log statements from libraries that your flows
and tasks use. You can tell Prefect to include logs from these libraries with
the PREFECT_LOGGING_EXTRA_LOGGERS
setting.
To use this setting, specify one or more Python library names to include, separated by commas. For example, if you want Prefect to capture Dask and SciPy logging statements with your flow and task run logs, use:
PREFECT_LOGGING_EXTRA_LOGGERS=dask,scipy
Configure this setting as an environment variable or in a profile. See Settings for more details about how to use settings.
Access logs from the command line
You can retrieve logs for a specific flow run ID using Prefect’s CLI:
This can be particularly helpful if you want to access the logs as a local file:
Was this page helpful?