airflow.executors.kubernetes_executor

KubernetesExecutor

See also

For more information on how the KubernetesExecutor works, take a look at the guide: Kubernetes Executor

Module Contents

Classes

ResourceVersion

Singleton for tracking resourceVersion from Kubernetes

KubernetesJobWatcher

Watches for Kubernetes jobs

AirflowKubernetesScheduler

Airflow Scheduler for Kubernetes

KubernetesExecutor

Executor for Kubernetes

Functions

get_base_pod_from_template(pod_template_file, kube_config)

Reads either the pod_template_file set in the executor_config or the base pod_template_file

Attributes

KubernetesJobType

KubernetesResultsType

KubernetesWatchType

airflow.executors.kubernetes_executor.KubernetesJobType[source]
airflow.executors.kubernetes_executor.KubernetesResultsType[source]
airflow.executors.kubernetes_executor.KubernetesWatchType[source]
class airflow.executors.kubernetes_executor.ResourceVersion[source]

Singleton for tracking resourceVersion from Kubernetes

resource_version = 0[source]
class airflow.executors.kubernetes_executor.KubernetesJobWatcher(namespace, multi_namespace_mode, watcher_queue, resource_version, scheduler_job_id, kube_config)[source]

Bases: multiprocessing.Process, airflow.utils.log.logging_mixin.LoggingMixin

Watches for Kubernetes jobs

run()[source]

Performs watching

process_error(event)[source]

Process error response

process_status(pod_id, namespace, status, annotations, resource_version, event)[source]

Process status response

class airflow.executors.kubernetes_executor.AirflowKubernetesScheduler(kube_config, task_queue, result_queue, kube_client, scheduler_job_id)[source]

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Airflow Scheduler for Kubernetes

run_pod_async(pod, **kwargs)[source]

Runs POD asynchronously

run_next(next_job)[source]

The run_next command will check the task_queue for any un-run jobs. It will then create a unique job-id, launch that job in the cluster, and store relevant info in the current_jobs map so we can track the job’s status

delete_pod(pod_id, namespace)[source]

Deletes POD

sync()[source]

The sync function checks the status of all currently running kubernetes jobs. If a job is completed, its status is placed in the result queue to be sent back to the scheduler.

Returns

Return type

None

process_watcher_task(task)[source]

Process the task by watcher.

terminate()[source]

Terminates the watcher.

airflow.executors.kubernetes_executor.get_base_pod_from_template(pod_template_file, kube_config)[source]

Reads either the pod_template_file set in the executor_config or the base pod_template_file set in the airflow.cfg to craft a “base pod” that will be used by the KubernetesExecutor

Parameters
  • pod_template_file (Optional[str]) – absolute path to a pod_template_file.yaml or None

  • kube_config (Any) – The KubeConfig class generated by airflow that contains all kube metadata

Returns

a V1Pod that can be used as the base pod for k8s tasks

Return type

kubernetes.client.models.V1Pod

class airflow.executors.kubernetes_executor.KubernetesExecutor[source]

Bases: airflow.executors.base_executor.BaseExecutor

Executor for Kubernetes

supports_ad_hoc_ti_run :bool = True[source]
clear_not_launched_queued_tasks(session=None)[source]

Tasks can end up in a “Queued” state through either the executor being abruptly shut down (leaving a non-empty task_queue on this executor) or when a rescheduled/deferred operator comes back up for execution (with the same try_number) before the pod of its previous incarnation has been fully removed (we think).

This method checks each of those tasks to see if the corresponding pod is around, and if not, and there’s no matching entry in our own task_queue, marks it for re-execution.

start()[source]

Starts the executor

execute_async(key, command, queue=None, executor_config=None)[source]

Executes task asynchronously

sync()[source]

Synchronize task state.

try_adopt_task_instances(tis)[source]

Try to adopt running task instances that have been abandoned by a SchedulerJob dying.

Anything that is not adopted will be cleared by the scheduler (and then become eligible for re-scheduling)

Returns

any TaskInstances that were unable to be adopted

Return type

list[airflow.models.TaskInstance]

adopt_launched_task(kube_client, pod, pod_ids)[source]

Patch existing pod so that the current KubernetesJobWatcher can monitor it via label selectors

Parameters
  • kube_client (kubernetes.client.CoreV1Api) – kubernetes client for speaking to kube API

  • pod (kubernetes.client.models.V1Pod) – V1Pod spec that we will patch with new label

  • pod_ids (Dict[airflow.models.taskinstance.TaskInstanceKey, kubernetes.client.models.V1Pod]) – pod_ids we expect to patch.

end()[source]

Called when the executor shuts down

terminate()[source]

Terminate the executor is not doing anything.

Was this entry helpful?