airflow.contrib.operators.kubernetes_pod_operator

Executes task in a Kubernetes POD

Module Contents

class airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator(namespace=None, image=None, name=None, cmds=None, arguments=None, ports=None, volume_mounts=None, volumes=None, env_vars=None, secrets=None, in_cluster=None, cluster_context=None, labels=None, reattach_on_restart=True, startup_timeout_seconds=120, get_logs=True, image_pull_policy='IfNotPresent', annotations=None, resources=None, affinity=None, config_file=None, node_selectors=None, image_pull_secrets=None, service_account_name='default', is_delete_operator_pod=False, hostnetwork=False, tolerations=None, configmaps=None, security_context=None, pod_runtime_info_envs=None, dnspolicy=None, schedulername=None, full_pod_spec=None, init_containers=None, log_events_on_failure=False, do_xcom_push=False, pod_template_file=None, priority_class_name=None, termination_grace_period=None, *args, **kwargs)[source]

Bases: airflow.models.BaseOperator

Execute a task in a Kubernetes Pod

Note

If you use Google Kubernetes Engine, use GKEPodOperator, which simplifies the authorization process.

Parameters

  • image (str) – Docker image you wish to launch. Defaults to hub.docker.com, but fully qualified URLS will point to custom repositories.

  • name (str) – name of the pod in which the task will run, will be used (plus a random suffix) to generate a pod id (DNS-1123 subdomain, containing only [a-z0-9.-]).

  • cmds (list[str]) – entrypoint of the container. (templated) The docker images’s entrypoint is used if this is not provided.

  • arguments (list[str]) – arguments of the entrypoint. (templated) The docker image’s CMD is used if this is not provided.

  • image_pull_policy (str) – Specify a policy to cache or always pull an image.

  • image_pull_secrets (str) – Any image pull secrets to be given to the pod. If more than one secret is required, provide a comma separated list: secret_a,secret_b

  • ports (list[airflow.kubernetes.pod.Port]) – ports for launched pod.

  • volume_mounts (list[airflow.kubernetes.volume_mount.VolumeMount]) – volumeMounts for launched pod.

  • volumes (list[airflow.kubernetes.volume.Volume]) – volumes for launched pod. Includes ConfigMaps and PersistentVolumes.

  • labels (dict) – labels to apply to the Pod.

  • startup_timeout_seconds (int) – timeout in seconds to startup the pod.

  • name – name of the pod in which the task will run, will be used to generate a pod id (DNS-1123 subdomain, containing only [a-z0-9.-]).

  • env_vars (dict) – Environment variables initialized in the container. (templated)

  • secrets (list[airflow.kubernetes.secret.Secret]) – Kubernetes secrets to inject in the container. They can be exposed as environment vars or files in a volume.

  • in_cluster (bool) – run kubernetes client with in_cluster configuration.

  • cluster_context (str) – context that points to kubernetes cluster. Ignored when in_cluster is True. If None, current-context is used.

  • reattach_on_restart (bool) – if the scheduler dies while the pod is running, reattach and monitor

  • labels – labels to apply to the Pod.

  • startup_timeout_seconds – timeout in seconds to startup the pod.

  • get_logs (bool) – get the stdout of the container as logs of the tasks.

  • annotations (dict) – non-identifying metadata you can attach to the Pod. Can be a large range of data, and can include characters that are not permitted by labels.

  • resources (dict) – A dict containing resources requests and limits. Possible keys are request_memory, request_cpu, limit_memory, limit_cpu, and limit_gpu, which will be used to generate airflow.kubernetes.pod.Resources. See also kubernetes.io/docs/concepts/configuration/manage-compute-resources-container

  • affinity (dict) – A dict containing a group of affinity scheduling rules.

  • node_selectors (dict) – A dict containing a group of scheduling rules.

  • config_file (str) – The path to the Kubernetes config file. (templated)

  • config_file – The path to the Kubernetes config file. (templated) If not specified, default value is ~/.kube/config

  • do_xcom_push (bool) – If do_xcom_push is True, the content of the file /airflow/xcom/return.json in the container will also be pushed to an XCom when the container completes.

  • is_delete_operator_pod (bool) – What to do when the pod reaches its final state, or the execution is interrupted. If False (default): do nothing, If True: delete the pod

  • hostnetwork (bool) – If True enable host networking on the pod.

  • tolerations (list tolerations) – A list of kubernetes tolerations.

  • configmaps (list[str]) – A list of configmap names objects that we want mount as env variables.

  • pod_runtime_info_envs (list[airflow.kubernetes.pod_runtime_info_env.PodRuntimeInfoEnv]) – environment variables about pod runtime information (ip, namespace, nodeName, podName).

  • security_context (dict) – security options the pod should run with (PodSecurityContext).

  • dnspolicy (str) – dnspolicy for the pod.

  • schedulername (str) – Specify a schedulername for the pod

  • full_pod_spec (kubernetes.client.models.V1Pod) – The complete podSpec

  • init_containers (list[kubernetes.client.models.V1Container]) – init container for the launched Pod

  • log_events_on_failure (bool) – Log the pod’s events if a failure occurs

  • do_xcom_push – If True, the content of the file /airflow/xcom/return.json in the container will also be pushed to an XCom when the container completes.

  • pod_template_file (str) – path to pod template file

  • priority_class_name (str) – priority class name for the launched Pod

  • termination_grace_period (int) – Termination grace period if task killed in UI, defaults to kubernetes default

template_fields = ['image', 'cmds', 'arguments', 'env_vars', 'config_file', 'pod_template_file'][source]
static create_labels_for_pod(context)[source]

Generate labels for the pod to track the pod in case of Operator crash

Parameters

context – task context provided by airflow DAG

Returns

dict

execute(self, context)[source]
handle_pod_overlap(self, labels, try_numbers_match, launcher, pod)[source]

In cases where the Scheduler restarts while a KubernetsPodOperator task is running, this function will either continue to monitor the existing pod or launch a new pod based on the reattach_on_restart parameter. :param labels: labels used to determine if a pod is repeated :type labels: dict :param try_numbers_match: do the try numbers match? Only needed for logging purposes :type try_numbers_match: bool :param launcher: PodLauncher :param pod: Pod found

static _get_pod_identifying_label_string(labels)[source]
static _try_numbers_match(context, pod)[source]
static _set_resources(resources)[source]
_set_name(self, name)[source]
create_pod_request_obj(self)[source]

Creates a V1Pod based on user parameters. Note that a pod or pod_template_file will supersede all other values.

create_new_pod_for_operator(self, labels, launcher)[source]

Creates a new pod and monitors for duration of task

@param labels: labels used to track pod @param launcher: pod launcher that will manage launching and monitoring pods @return:

patch_already_checked(self, pod)[source]

Add an “already tried annotation to ensure we only retry once

monitor_launched_pod(self, launcher, pod)[source]

Monitors a pod to completion that was created by a previous KubernetesPodOperator

Parameters

  • launcher – pod launcher that will manage launching and monitoring pods

  • pod – podspec used to find pod using k8s API

Returns

on_kill(self)[source]

Was this entry helpful?