Storage


The Prefect Storage interface encapsulates logic for storing, serializing and even running Flows. Each storage unit is able to store multiple flows (possibly with the constraint of name uniqueness within a given unit), and exposes the following methods and attributes:

  • a name attribute
  • a flows attribute that is a dictionary of Flows -> location
  • an add_flow(flow: Flow) -> str method for adding flows to Storage, and that will return the location of the given flow in the Storage unit
  • the __contains__(self, obj) -> bool special method for determining whether the Storage contains a given Flow
  • one of get_flow(flow_location: str) or get_env_runner(flow_location: str) for retrieving a way of interfacing with either flow.run or a FlowRunner for the flow; get_env_runner is intended for situations where flow execution can only be interacted with via environment variables
  • a build() -> Storage method for "building" the storage
  • a serialize() -> dict method for serializing the relevant information about this Storage for later re-use.

Docker storage

Note that currently all environments that are compatible with Prefect Cloud require your Flow to use Docker storage.

Docker

class

prefect.environments.storage.docker.Docker

(registry_url=None, base_image=None, python_dependencies=None, image_name=None, image_tag=None, env_vars=None, files=None, base_url="unix://var/run/docker.sock", prefect_version=None, local_image=False)[source]

Docker storage provides a mechanism for storing Prefect flows in Docker images and optionally pushing them to a registry.

A user specifies a registry_url, base_image and other optional dependencies (e.g., python_dependencies) and build() will create a temporary Dockerfile that is used to build the image.

Note that the base_image must be capable of pip installing. Note that registry behavior with respect to image names can differ between providers - for example, Google's GCR registry allows for registry URLs of the form gcr.io/my-registry/subdir/my-image-name whereas DockerHub requires the registry URL to be separate from the image name.

Args:

  • registry_url (str, optional): URL of a registry to push the image to; image will not be pushed if not provided
  • base_image (str, optional): the base image for this environment (e.g. python:3.6), defaults to python:3.6
  • python_dependencies (List[str], optional): list of pip installable dependencies for the image
  • image_name (str, optional): name of the image to use when building, populated with a UUID after build
  • image_tag (str, optional): tag of the image to use when building, populated with a UUID after build
  • env_vars (dict, optional): a dictionary of environment variables to use when building
  • files (dict, optional): a dictionary of files to copy into the image when building
  • base_url: (str, optional): a URL of a Docker daemon to use when for Docker related functionality
  • prefect_version (str, optional): an optional branch, tag, or commit specifying the version of prefect you want installed into the container; defaults to the version you are currently using or "master" if your version is ahead of the latest tag
  • local_image(bool, optional): an optional flag whether or not to use a local docker image, if True then a pull will not be attempted

methods:                                                                                                                                                       

prefect.environments.storage.docker.Docker.add_flow

(flow)[source]

Method for adding a new flow to this Storage object.

Args:

  • flow (Flow): a Prefect Flow to add
Returns:
  • str: the location of the newly added flow in this Storage object

prefect.environments.storage.docker.Docker.build

(push=True)[source]

Build the Docker storage object. If image name and tag are not set, they will be autogenerated.

Args:

  • push (bool, optional): Whether or not to push the built Docker image, this requires the registry_url to be set
Returns:
  • Docker: a new Docker storage object that contains information about how and where the flow is stored. Image name and tag are generated during the build process.
Raises:
  • InterruptedError: if either pushing or pulling the image fails

prefect.environments.storage.docker.Docker.create_dockerfile_object

(directory=None)[source]

Writes a dockerfile to the provided directory using the specified arguments on this Docker storage object.

In order for the docker python library to build a container it needs a Dockerfile that it can use to define the container. This function takes the specified arguments then writes them to a temporary file called Dockerfile.

Note: if files are added to this container, they will be copied to this directory as well.

Args:

  • directory (str, optional): A directory where the Dockerfile will be created, if no directory is specified is will be created in the current working directory

prefect.environments.storage.docker.Docker.get_env_runner

(flow_location)[source]

Given a flow_location within this Storage object, returns something with a run() method which accepts the standard runner kwargs and can run the flow.

Args:

  • flow_location (str): the location of a flow within this Storage
Returns:
  • a runner interface (something with a run() method for running the flow)

prefect.environments.storage.docker.Docker.pull_image

()[source]

Pull the image specified so it can be built.

In order for the docker python library to use a base image it must be pulled from either the main docker registry or a separate registry that must be set as registry_url on this class.

Raises:

  • InterruptedError: if either pulling the image fails

prefect.environments.storage.docker.Docker.push_image

(image_name, image_tag)[source]

Push this environment to a registry

Args:

  • image_name (str): Name for the image
  • image_tag (str): Tag for the image
Raises:
  • InterruptedError: if either pushing the image fails



Local

class

prefect.environments.storage.local.Local

(directory="~/.prefect/flows")[source]

Local storage class. This class represents the Storage interface for Flows stored as bytes in the local filesystem.

Args:

  • directory (str, optional): the directory the flows will be stored in; defaults to ~/.prefect/flows. If it doesn't already exist, it will be created for you.

methods:                                                                                                                                                       

prefect.environments.storage.local.Local.add_flow

(flow)[source]

Method for storing a new flow as bytes in the local filesytem.

Args:

  • flow (Flow): a Prefect Flow to add
Returns:
  • str: the location of the newly added flow in this Storage object
Raises:
  • ValueError: if a flow with the same name is already contained in this storage

prefect.environments.storage.local.Local.build

()[source]

Build the Storage object.

Returns:

  • Storage: a Storage object that contains information about how and where each flow is stored

prefect.environments.storage.local.Local.get_flow

(flow_location)[source]

Given a flow_location within this Storage object, returns the underlying Flow (if possible).

Args:

  • flow_location (str): the location of a flow within this Storage; in this case, a file path where a Flow has been serialized to
Returns:
  • Flow: the requested flow
Raises:
  • ValueError: if the flow is not contained in this storage



Memory

class

prefect.environments.storage.memory.Memory

()[source]

Memory Storage class, mainly used for testing. This class represents the Storage interface for Flows contained in memory, i.e., flows are simply stored as attributes of this class.

methods:                                                                                                                                                       

prefect.environments.storage.memory.Memory.add_flow

(flow)[source]

Method for adding a new flow to this Storage object.

Args:

  • flow (Flow): a Prefect Flow to add
Returns:
  • str: the location of the newly added flow in this Storage object
Raises:
  • ValueError: if a flow with the same name is already contained in this storage

prefect.environments.storage.memory.Memory.build

()[source]

Build the Storage object.

Returns:

  • Storage: a Storage object that contains information about how and where each flow is stored

prefect.environments.storage.memory.Memory.get_flow

(flow_location)[source]

Given a flow_location within this Storage object, returns the underlying Flow (if possible).

Args:

  • flow_location (str): the location of a flow within this Storage
Returns:
  • Flow: the requested flow
Raises:
  • ValueError: if the flow is not contained in this storage



Bytes

class

prefect.environments.storage.bytes.Bytes

()[source]

Bytes Storage class, mainly used for testing. This class represents the Storage interface for Flows stored directly as bytes.

The main difference between this class and Memory storage is that Bytes can be serialized and deserialized while preserving all relevant information.

methods:                                                                                                                                                       

prefect.environments.storage.bytes.Bytes.add_flow

(flow)[source]

Method for adding a new flow to this Storage object.

Args:

  • flow (Flow): a Prefect Flow to add
Returns:
  • str: the location of the newly added flow in this Storage object
Raises:
  • ValueError: if a flow with the same name is already contained in this storage

prefect.environments.storage.bytes.Bytes.build

()[source]

Build the Storage object.

Returns:

  • Storage: a Storage object that contains information about how and where each flow is stored

prefect.environments.storage.bytes.Bytes.get_flow

(flow_location)[source]

Given a flow_location within this Storage object, returns the underlying Flow (if possible).

Args:

  • flow_location (str): the location of a flow within this Storage; in this case, a flow location is simply a Flow's name
Returns:
  • Flow: the requested flow
Raises:
  • ValueError: if the flow is not contained in this storage



This documentation was auto-generated from commit 3c91159
on August 24, 2019 at 18:41 UTC