Prefect Docs

# TaskRunner

# TaskRunner

class

prefect.engine.task_runner.TaskRunner

(task, state_handlers=None, flow_result=None)[source]

TaskRunners handle the execution of Tasks and determine the State of a Task before, during and after the Task is run.

In particular, through the TaskRunner you can specify the states of any upstream dependencies and what state the Task should be initialized with.

Args:

  • task (Task): the Task to be run / executed
  • state_handlers (Iterable[Callable], optional): A list of state change handlers that will be called whenever the task changes state, providing an opportunity to inspect or modify the new state. The handler will be passed the task runner instance, the old (prior) state, and the new (current) state, with the following signature: state_handler(TaskRunner, old_state, new_state) -> Optional[State]; If multiple functions are passed, then the new_state argument will be the result of the previous handler.
  • flow_result: the result instance configured for the flow (if any)

methods:                                                                                                                                                       

prefect.engine.task_runner.TaskRunner.cache_result

(state, inputs)[source]

Caches the result of a successful task, if appropriate. Alternatively, if the task is failed, caches the inputs.

Tasks are cached if: - task.cache_for is not None - the task state is Successful - the task state is not Skipped (which is a subclass of Successful)

Args:

  • state (State): the current state of this task
  • inputs (Dict[str, Result], optional): a dictionary of inputs whose keys correspond to the task's run() arguments.
Returns:
  • State: the state of the task after running the check

prefect.engine.task_runner.TaskRunner.call_runner_target_handlers

(old_state, new_state)[source]

A special state handler that the TaskRunner uses to call its task's state handlers. This method is called as part of the base Runner's handle_state_change() method.

Args:

  • old_state (State): the old (previous) state
  • new_state (State): the new (current) state
Returns:
  • State: the new state

prefect.engine.task_runner.TaskRunner.check_for_retry

(state, inputs)[source]

Checks to see if a FAILED task should be retried.

Args:

  • state (State): the current state of this task
  • inputs (Dict[str, Result]): a dictionary of inputs whose keys correspond to the task's run() arguments.
Returns:
  • State: the state of the task after running the check

prefect.engine.task_runner.TaskRunner.check_target

(state, inputs)[source]

Checks if a Result exists at the task's target.

Args:

  • state (State): the current state of this task
  • inputs (Dict[str, Result]): a dictionary of inputs whose keys correspond to the task's run() arguments.
Returns:
  • State: the state of the task after running the check

prefect.engine.task_runner.TaskRunner.check_task_is_cached

(state, inputs)[source]

Checks if task is cached and whether the cache is still valid.

Args:

  • state (State): the current state of this task
  • inputs (Dict[str, Result]): a dictionary of inputs whose keys correspond to the task's run() arguments.
Returns:
  • State: the state of the task after running the check
Raises:
  • ENDRUN: if the task is not ready to run

prefect.engine.task_runner.TaskRunner.check_task_is_looping

(state, inputs=None, upstream_states=None, context=None)[source]

Checks to see if the task is in a Looped state and if so, rerun the pipeline with an incremeneted loop_count.

Args:

  • state (State, optional): initial State to begin task run from; defaults to Pending()
  • inputs (Dict[str, Result], optional): a dictionary of inputs whose keys correspond to the task's run() arguments.
  • upstream_states (Dict[Edge, State]): a dictionary representing the states of any tasks upstream of this one. The keys of the dictionary should correspond to the edges leading to the task.
  • context (dict, optional): prefect Context to use for execution
Returns:
  • State object representing the final post-run state of the Task

prefect.engine.task_runner.TaskRunner.check_task_is_ready

(state)[source]

Checks to make sure the task is ready to run (Pending or Mapped).

Args:

  • state (State): the current state of this task
Returns:
  • State: the state of the task after running the check
Raises:
  • ENDRUN: if the task is not ready to run

prefect.engine.task_runner.TaskRunner.check_task_reached_start_time

(state)[source]

Checks if a task is in a Scheduled state and, if it is, ensures that the scheduled time has been reached. Note: Scheduled states include Retry states. Scheduled states with no start time (start_time = None) are never considered ready; they must be manually placed in another state.

Args:

  • state (State): the current state of this task
Returns:
  • State: the state of the task after performing the check
Raises:
  • ENDRUN: if the task is Scheduled with a future scheduled time

prefect.engine.task_runner.TaskRunner.check_task_ready_to_map

(state, upstream_states)[source]

Checks if the parent task is ready to proceed with mapping.

Args:

  • state (State): the current state of this task
  • upstream_states (Dict[Edge, Union[State, List[State]]]): the upstream states
Raises:
  • ENDRUN: either way, we dont continue past this point

prefect.engine.task_runner.TaskRunner.check_task_trigger

(state, upstream_states)[source]

Checks if the task's trigger function passes.

Args:

  • state (State): the current state of this task
  • upstream_states (Dict[Edge, Union[State, List[State]]]): the upstream states
Returns:
  • State: the state of the task after running the check
Raises:
  • ENDRUN: if the trigger raises an error

prefect.engine.task_runner.TaskRunner.check_upstream_finished

(state, upstream_states)[source]

Checks if the upstream tasks have all finshed.

Args:

  • state (State): the current state of this task
  • upstream_states (Dict[Edge, Union[State, List[State]]]): the upstream states
Returns:
  • State: the state of the task after running the check
Raises:
  • ENDRUN: if upstream tasks are not finished.

prefect.engine.task_runner.TaskRunner.check_upstream_skipped

(state, upstream_states)[source]

Checks if any of the upstream tasks have skipped.

Args:

  • state (State): the current state of this task
  • upstream_states (Dict[Edge, State]): the upstream states
Returns:
  • State: the state of the task after running the check

prefect.engine.task_runner.TaskRunner.get_task_inputs

(state, upstream_states)[source]

Given the task's current state and upstream states, generates the inputs for this task. Upstream state result values are used.

Args:

  • state (State): the task's current state.
  • upstream_states (Dict[Edge, State]): the upstream state_handlers
Returns:
  • Dict[str, Result]: the task inputs

prefect.engine.task_runner.TaskRunner.get_task_run_state

(state, inputs)[source]

Runs the task and traps any signals or errors it raises. Also checkpoints the result of a successful task, if task.checkpoint is True.

Args:

  • state (State): the current state of this task
  • inputs (Dict[str, Result], optional): a dictionary of inputs whose keys correspond to the task's run() arguments.
Returns:
  • State: the state of the task after running the check
Raises:
  • signals.PAUSE: if the task raises PAUSE
  • ENDRUN: if the task is not ready to run

prefect.engine.task_runner.TaskRunner.initialize_run

(state, context)[source]

Initializes the Task run by initializing state and context appropriately.

If the task is being retried, then we retrieve the run count from the initial Retry state. Otherwise, we assume the run count is 1. The run count is stored in context as task_run_count.

Also, if the task is being resumed through a Resume state, updates context to have resume=True.

Args:

  • state (Optional[State]): the initial state of the run
  • context (Dict[str, Any]): the context to be updated with relevant information
Returns:
  • tuple: a tuple of the updated state, context, upstream_states, and inputs objects

prefect.engine.task_runner.TaskRunner.load_results

(state, upstream_states)[source]

Given the task's current state and upstream states, populates all relevant result objects for this task run.

Args:

  • state (State): the task's current state.
  • upstream_states (Dict[Edge, State]): the upstream state_handlers
Returns:
  • Tuple[State, dict]: a tuple of (state, upstream_states)

prefect.engine.task_runner.TaskRunner.run

(state=None, upstream_states=None, context=None, is_mapped_parent=False)[source]

The main endpoint for TaskRunners. Calling this method will conditionally execute self.task.run with any provided inputs, assuming the upstream dependencies are in a state which allow this Task to run.

Args:

  • state (State, optional): initial State to begin task run from; defaults to Pending()
  • upstream_states (Dict[Edge, State]): a dictionary representing the states of any tasks upstream of this one. The keys of the dictionary should correspond to the edges leading to the task.
  • context (dict, optional): prefect Context to use for execution
  • is_mapped_parent (bool): a boolean indicating whether this task run is the run of a parent mapped task
Returns:
  • State object representing the final post-run state of the Task

prefect.engine.task_runner.TaskRunner.set_task_run_name

(task_inputs)[source]

Sets the name for this task run and adds to prefect.context

Args:

  • task_inputs (Dict[str, Result]): a dictionary of inputs whose keys correspond to the task's run() arguments.

prefect.engine.task_runner.TaskRunner.set_task_to_running

(state, inputs)[source]

Sets the task to running

Args:

  • state (State): the current state of this task
  • inputs (Dict[str, Result]): a dictionary of inputs whose keys correspond to the task's run() arguments.
Returns:
  • State: the state of the task after running the check
Raises:
  • ENDRUN: if the task is not ready to run


This documentation was auto-generated from commit ffa9a6c
on February 1, 2023 at 18:44 UTC

← Executors →