Execution Environments


Execution environments encapsulate the logic for where your Flow should execute in Prefect Cloud.

Currently, we recommend all users deploy their Flow using the RemoteEnvironment configured with the appropriate choice of executor.

DaskKubernetesEnvironment

class

prefect.environments.execution.dask.k8s.DaskKubernetesEnvironment

(min_workers=1, max_workers=2, private_registry=False, docker_secret=None)[source]

DaskKubernetesEnvironment is an environment which deploys your flow (stored in a Docker image) on Kubernetes by spinning up a temporary Dask Cluster (using dask-kubernetes) and running the Prefect DaskExecutor on this cluster.

If pulling from a private docker registry, setup will ensure the appropriate kubernetes secret exists; execute creates a single job that has the role of spinning up a dask executor and running the flow. The job created in the execute function does have the requirement in that it needs to have an identifier_label set with a UUID so resources can be cleaned up independently of other deployments.

Args:

  • min_workers (int, optional): the minimum allowed number of Dask worker pods; defaults to 1
  • max_workers (int, optional): the maximum allowed number of Dask worker pods; defaults to 1
  • private_registry (bool, optional): a boolean specifying whether your Flow's Docker container will be in a private Docker registry; if so, requires a Prefect Secret containing your docker credentials to be set. Defaults to False.
  • docker_secret (str, optional): the name of the Prefect Secret containing your Docker credentials; defaults to "DOCKER_REGISTRY_CREDENTIALS". This Secret should be a dictionary containing the following keys: "docker-server", "docker-username", "docker-password", and "docker-email".

methods:                                                                                                                                                       

prefect.environments.execution.dask.k8s.DaskKubernetesEnvironment.create_flow_run_job

(docker_name, flow_file_path)[source]

Creates a Kubernetes job to run the flow using the information stored on the Docker storage object.

Args:

  • docker_name (str): the full name of the docker image (registry/name:tag)
  • flow_file_path (str): location of the flow file in the image

prefect.environments.execution.dask.k8s.DaskKubernetesEnvironment.execute

(storage, flow_location, **kwargs)[source]

Create a single Kubernetes job that spins up a dask scheduler, dynamically creates worker pods, and runs the flow.

Args:

  • storage (Docker): the Docker storage object that contains information relating to the image which houses the flow
  • flow_location (str): the location of the Flow to execute
  • **kwargs (Any): additional keyword arguments to pass to the runner
Raises:
  • TypeError: if the storage is not Docker

prefect.environments.execution.dask.k8s.DaskKubernetesEnvironment.run_flow

()[source]

Run the flow from specified flow_file_path location using a Dask executor

prefect.environments.execution.dask.k8s.DaskKubernetesEnvironment.setup

(storage)[source]

Sets up any infrastructure needed for this environment

Args:

  • storage (Storage): the Storage object that contains the flow



LocalEnvironment

class

prefect.environments.execution.local.LocalEnvironment

()[source]

A LocalEnvironment class for executing a flow contained in Storage in the local process. Execution will first attempt to call get_flow on the storage object, and if that fails it will fall back to get_env_runner. If get_env_runner is used, the environment variables from this process will be passed.

methods:                                                                                                                                                       

prefect.environments.execution.local.LocalEnvironment.execute

(storage, flow_location, **kwargs)[source]

Executes the flow for this environment from the storage parameter, by calling get_flow on the storage; if that fails, get_env_runner will be used with the OS environment variables inherited from this process.

Args:

  • storage (Storage): the Storage object that contains the flow
  • flow_location (str): the location of the Flow to execute
  • **kwargs (Any): additional keyword arguments to pass to the runner



RemoteEnvironment

class

prefect.environments.execution.remote.RemoteEnvironment

(executor=None, executor_kwargs=None)[source]

RemoteEnvironment is an environment which takes in information about an executor and runs the flow in place using that executor.

Example:

# using a RemoteEnvironment w/ an existing Dask cluster

env = RemoteEnvironment(
    executor="prefect.engine.executors.DaskExecutor",
    executor_kwargs={"address": "tcp://dask_scheduler_address"}
)

f = Flow("dummy flow", environment=env)

Args:

  • executor (str, optional): an importable string to an executor class; defaults to prefect.config.engine.executor.default_class
  • executor_kwargs (dict, optional): a dictionary of kwargs to be passed to the executor; defaults to an empty dictionary

methods:                                                                                                                                                       

prefect.environments.execution.remote.RemoteEnvironment.execute

(storage, flow_location, **kwargs)[source]

Run a flow from the flow_location here using the specified executor and executor kwargs.

Args:

  • storage (Storage): the storage object that contains information relating to where and how the flow is stored
  • flow_location (str): the location of the Flow to execute
  • **kwargs (Any): additional keyword arguments to pass to the runner



This documentation was auto-generated from commit 3c91159
on August 24, 2019 at 18:41 UTC