airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils¶

Classes¶

`ResourceVersion`	Singleton for tracking resourceVersion from Kubernetes.
`KubernetesJobWatcher`	Watches for Kubernetes jobs.
`AirflowKubernetesScheduler`	Airflow Scheduler for Kubernetes.

Functions¶

get_base_pod_from_template(pod_template_file, kube_config)

Get base pod from template.

Module Contents¶

class airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.ResourceVersion[source]¶

Singleton for tracking resourceVersion from Kubernetes.

resource_version: dict[str, str][source]¶

class airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher(namespace, watcher_queue, resource_version, scheduler_job_id, kube_config)[source]¶

Bases: multiprocessing.Process, airflow.utils.log.logging_mixin.LoggingMixin

Watches for Kubernetes jobs.

namespace[source]¶

scheduler_job_id[source]¶

watcher_queue[source]¶

resource_version[source]¶

kube_config[source]¶

run()[source]¶

Perform watching.

process_error(event)[source]¶

Process error response.

process_status(pod_name, namespace, status, annotations, resource_version, event)[source]¶

Process status response.

class airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.AirflowKubernetesScheduler(kube_config, result_queue, kube_client, scheduler_job_id)[source]¶

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Airflow Scheduler for Kubernetes.

kube_config[source]¶

result_queue[source]¶

namespace[source]¶

kube_client[source]¶

watcher_queue[source]¶

scheduler_job_id[source]¶

kube_watchers[source]¶

run_pod_async(pod, **kwargs)[source]¶

Run POD asynchronously.

run_next(next_job)[source]¶

Receives the next job to run, builds the pod, and creates it.

delete_pod(pod_name, namespace)[source]¶

Delete Pod from a namespace; does not raise if it does not exist.

patch_pod_revoked(*, pod_name, namespace)[source]¶

Patch the pod with a label that ensures it’s ignored by the kubernetes watcher.

patch_pod_executor_done(*, pod_name, namespace)[source]¶

Add a “done” annotation to ensure we don’t continually adopt pods.

sync()[source]¶

Check the status of all currently running kubernetes jobs.

If a job is completed, its status is placed in the result queue to be sent back to the scheduler.

process_watcher_task(task)[source]¶

Process the task by watcher.

terminate()[source]¶

Terminates the watcher.

airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.get_base_pod_from_template(pod_template_file, kube_config)[source]¶

Get base pod from template.

Reads either the pod_template_file set in the executor_config or the base pod_template_file set in the airflow.cfg to craft a “base pod” that will be used by the KubernetesExecutor

Parameters:

pod_template_file (str | None) – absolute path to a pod_template_file.yaml or None
kube_config (Any) – The KubeConfig class generated by airflow that contains all kube metadata

Returns:

a V1Pod that can be used as the base pod for k8s tasks

Return type:

kubernetes.client.models.V1Pod