airflow.executors.kubernetes_executor
¶
KubernetesExecutor
See also
For more information on how the KubernetesExecutor works, take a look at the guide: Kubernetes Executor
Module Contents¶
Classes¶
Singleton for tracking resourceVersion from Kubernetes |
|
Watches for Kubernetes jobs |
|
Airflow Scheduler for Kubernetes |
|
Executor for Kubernetes |
Functions¶
|
Reads either the pod_template_file set in the executor_config or the base pod_template_file |
Attributes¶
- class airflow.executors.kubernetes_executor.ResourceVersion[source]¶
Singleton for tracking resourceVersion from Kubernetes
- class airflow.executors.kubernetes_executor.KubernetesJobWatcher(namespace, multi_namespace_mode, watcher_queue, resource_version, scheduler_job_id, kube_config)[source]¶
Bases:
multiprocessing.Process
,airflow.utils.log.logging_mixin.LoggingMixin
Watches for Kubernetes jobs
- class airflow.executors.kubernetes_executor.AirflowKubernetesScheduler(kube_config, task_queue, result_queue, kube_client, scheduler_job_id)[source]¶
Bases:
airflow.utils.log.logging_mixin.LoggingMixin
Airflow Scheduler for Kubernetes
- run_next(next_job)[source]¶
The run_next command will check the task_queue for any un-run jobs. It will then create a unique job-id, launch that job in the cluster, and store relevant info in the current_jobs map so we can track the job’s status
- airflow.executors.kubernetes_executor.get_base_pod_from_template(pod_template_file, kube_config)[source]¶
Reads either the pod_template_file set in the executor_config or the base pod_template_file set in the airflow.cfg to craft a “base pod” that will be used by the KubernetesExecutor
- Parameters
pod_template_file (Optional[str]) – absolute path to a pod_template_file.yaml or None
kube_config (Any) – The KubeConfig class generated by airflow that contains all kube metadata
- Returns
a V1Pod that can be used as the base pod for k8s tasks
- Return type
kubernetes.client.models.V1Pod
- class airflow.executors.kubernetes_executor.KubernetesExecutor[source]¶
Bases:
airflow.executors.base_executor.BaseExecutor
Executor for Kubernetes
- clear_not_launched_queued_tasks(session=None)[source]¶
Tasks can end up in a “Queued” state through either the executor being abruptly shut down (leaving a non-empty task_queue on this executor) or when a rescheduled/deferred operator comes back up for execution (with the same try_number) before the pod of its previous incarnation has been fully removed (we think).
This method checks each of those tasks to see if the corresponding pod is around, and if not, and there’s no matching entry in our own task_queue, marks it for re-execution.
- try_adopt_task_instances(tis)[source]¶
Try to adopt running task instances that have been abandoned by a SchedulerJob dying.
Anything that is not adopted will be cleared by the scheduler (and then become eligible for re-scheduling)
- Returns
any TaskInstances that were unable to be adopted
- Return type
list[airflow.models.TaskInstance]
- adopt_launched_task(kube_client, pod, pod_ids)[source]¶
Patch existing pod so that the current KubernetesJobWatcher can monitor it via label selectors
- Parameters
kube_client (kubernetes.client.CoreV1Api) – kubernetes client for speaking to kube API
pod (kubernetes.client.models.V1Pod) – V1Pod spec that we will patch with new label
pod_ids (Dict[airflow.models.taskinstance.TaskInstanceKey, kubernetes.client.models.V1Pod]) – pod_ids we expect to patch.