# Kubernetes Agent

Looking for the latest Prefect 2 release? Prefect 2 and Prefect Cloud 2 have been released for General Availability. See https://docs.prefect.io/ for details.

The Kubernetes Agent deploys flow runs as Kubernetes Jobs. It can be run both in-cluster (recommended for production deployments) as well as external to the cluster.

# Requirements

If running locally, note that the required dependencies for the Kubernetes Agent aren't installed by default. If you're a pip user you'll need to add the kubernetes extra. Likewise, with conda you'll need to install the extra kubernetes package:

If you're deploying the Kubernetes Agent in-cluster, you won't need to worry about this.

Prefect Server

In order to use this agent with Prefect Server the server's GraphQL API endpoint must be accessible. This may require changes to your Prefect Server deployment and/or configuring the Prefect API address on the agent.

# Flow Configuration

The Kubernetes Agent will deploy flows using either a UniversalRun (the default) or KubernetesRun run_config. Using a KubernetesRun object lets you customize the deployment environment for a flow (exposing env, image, cpu_limit, etc...):

from prefect.run_configs import KubernetesRun

# Configure extra environment variables for this flow,
# and set a custom image
flow.run_config = KubernetesRun(
    env={"SOME_VAR": "VALUE"},
    image="my-custom-image"
)

See the KubernetesRun documentation for more information.

# Agent Configuration

The Kubernetes agent can be started from the Prefect CLI as

prefect agent kubernetes start

API Keys Cloud

When using Prefect Cloud, this will require a service account API key, see here for more information.

Below we cover a few common configuration options, see the CLI docs for a full list of options.

# Authentication

If running external to the cluster, you'll need to ensure your active kubeconfig is valid and for the proper cluster.

When running in-cluster, the Agent pod will need the proper RBAC credentials. This is covered in more detail below.

# Namespace

By default the agent will deploy flow run jobs into the default namespace. You can use the --namespace option to change this:

prefect agent kubernetes start --namespace my-namespace

# Service Account

You can use the --service-account-name option to configure a service account to use for all Jobs deployed by the agent.

prefect agent kubernetes start --service-account-name my-account

Flows can override this agent default by passing the service_account_name option to their respective KubernetesRun run_config.

# Image Pull Secrets

Likewise, the --image-pull-secrets option can be used to configure image pull secrets for all Jobs deployed by the agent. Multiple secrets can be provided as a comma-separated list. Note that the secrets must already exist within the same namespace as the agent.

prefect agent kubernetes start --image-pull-secrets secret-1,secret-2

Flows can override this agent default by passing the image_pull_secrets option to their respective KubernetesRun run_config.

# Custom Job Template

For deeper customization of the Kubernetes Job created by the agent, you may want to make use of a custom job template. This template can either be configured per-flow (on the KubernetesRun run_config), or on the Agent as a default for flows that don't provide their own template.

As an example, providing a custom job template will allow you to customize the Kubernetes Job name prefix from prefect-job to something more descriptive for each flow and its associated workload. The prefix is specified in the job template's metadata.name field.

For reference, the default template packaged with Prefect can be found here.

To provide your own job template, you can use the --job-template flag. This takes a path to a job template YAML file. The path can be local to the agent, or stored in cloud storage on either GCS or S3 (note that in this case, the agent will need credentials for S3/GCS access).

# Using a local file
prefect agent kubernetes start --job-template /path/to/my_template.yaml

# Stored on S3
prefect agent kubernetes start --job-template s3://bucket/path/to/my_template.yaml

# Running In-Cluster

For production deployments, we recommend running the Kubernetes Agent inside the Kubernetes Cluster as a Deployment. This helps ensure your agent is active and healthy.

To make this easier, the Prefect CLI provides a command for generating an example Kubernetes Manifest you can use for creating such a deployment.

prefect agent kubernetes install

The install command accepts many of the same options as the start command - see the CLI documentation for all available options.

The generated manifest can be piped to kubectl apply, or manually edited to further customize the deployment.

prefect agent kubernetes install -k API_KEY | kubectl apply --namespace=my-namespace -f -

Once created, you should be able to see the agent deployment running in your cluster:

$ kubectl get deploy
NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
prefect-agent   1         1         1            0           2s

$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
prefect-agent-845798bb59-s7wxg   1/1     Running   0          5s

# RBAC

The Kubernetes Agent requires certain RBAC permissions to work with jobs in its namespace. The proper RBAC configuration can be generated by providing the --rbac flag to prefect agent kubernetes install- for reference we provide it below as well.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: prefect-agent-rbac
rules:
  - apiGroups: ['batch', 'extensions']
    resources: ['jobs']
    verbs: ['*']
  - apiGroups: ['']
    resources: ['events', 'pods']
    verbs: ['*']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: default
  name: prefect-agent-rbac
subjects:
  - kind: ServiceAccount
    name: default
roleRef:
  kind: Role
  name: prefect-agent-rbac
  apiGroup: rbac.authorization.k8s.io

# Additional Permissions

If you are using Amazon EKS for your kubernetes deployment and you need S3 access, note that S3 is not accessible by default to Amazon EKS. To enable S3 access by your kubernetes cluster on EKS, add the necessary permissions (AmazonS3FullAccess or AmazonS3ReadOnlyAccess) directly to the NodeInstanceRole used by aws-auth-cm.yaml after launching worker nodes and before applying aws-auth-cm.yaml with kubectl.