airflow.executors.celery_kubernetes_executor
¶
Module Contents¶
-
class
airflow.executors.celery_kubernetes_executor.
CeleryKubernetesExecutor
(celery_executor, kubernetes_executor)[source]¶ Bases:
airflow.utils.log.logging_mixin.LoggingMixin
CeleryKubernetesExecutor consists of CeleryExecutor and KubernetesExecutor. It chooses an executor to use based on the queue defined on the task. When the queue is kubernetes, KubernetesExecutor is selected to run the task, otherwise, CeleryExecutor is used.
-
queue_command
(self, task_instance: TaskInstance, command: CommandType, priority: int = 1, queue: Optional[str] = None)[source]¶ Queues command via celery or kubernetes executor
-
queue_task_instance
(self, task_instance: TaskInstance, mark_success: bool = False, pickle_id: Optional[str] = None, ignore_all_deps: bool = False, ignore_depends_on_past: bool = False, ignore_task_deps: bool = False, ignore_ti_state: bool = False, pool: Optional[str] = None, cfg_path: Optional[str] = None)[source]¶ Queues task instance via celery or kubernetes executor
-
has_task
(self, task_instance: TaskInstance)[source]¶ Checks if a task is either queued or running in either celery or kubernetes executor.
- Parameters
task_instance -- TaskInstance
- Returns
True if the task is known to this executor
-
get_event_buffer
(self, dag_ids=None)[source]¶ Returns and flush the event buffer from celery and kubernetes executor
- Parameters
dag_ids -- to dag_ids to return events for, if None returns all
- Returns
a dict of events
-
try_adopt_task_instances
(self, tis: List[TaskInstance])[source]¶ Try to adopt running task instances that have been abandoned by a SchedulerJob dying.
Anything that is not adopted will be cleared by the scheduler (and then become eligible for re-scheduling)
- Returns
any TaskInstances that were unable to be adopted
- Return type
-
_router
(self, simple_task_instance: SimpleTaskInstance)[source]¶ Return either celery_executor or kubernetes_executor
- Parameters
simple_task_instance -- SimpleTaskInstance
- Returns
celery_executor or kubernetes_executor
- Return type
Union[CeleryExecutor, KubernetesExecutor]
-