airflow.executors.celery_kubernetes_executor

Module Contents

class airflow.executors.celery_kubernetes_executor.CeleryKubernetesExecutor(celery_executor, kubernetes_executor)[source]

Bases: airflow.utils.log.logging_mixin.LoggingMixin

CeleryKubernetesExecutor consists of CeleryExecutor and KubernetesExecutor. It chooses an executor to use based on the queue defined on the task. When the queue is kubernetes, KubernetesExecutor is selected to run the task, otherwise, CeleryExecutor is used.

KUBERNETES_QUEUE[source]
queued_tasks[source]

Return queued tasks from celery and kubernetes executor

running[source]

Return running tasks from celery and kubernetes executor

start(self)[source]

Start celery and kubernetes executor

queue_command(self, task_instance: TaskInstance, command: CommandType, priority: int = 1, queue: Optional[str] = None)[source]

Queues command via celery or kubernetes executor

queue_task_instance(self, task_instance: TaskInstance, mark_success: bool = False, pickle_id: Optional[str] = None, ignore_all_deps: bool = False, ignore_depends_on_past: bool = False, ignore_task_deps: bool = False, ignore_ti_state: bool = False, pool: Optional[str] = None, cfg_path: Optional[str] = None)[source]

Queues task instance via celery or kubernetes executor

has_task(self, task_instance: TaskInstance)[source]

Checks if a task is either queued or running in either celery or kubernetes executor.

Parameters

task_instance -- TaskInstance

Returns

True if the task is known to this executor

heartbeat(self)[source]

Heartbeat sent to trigger new jobs in celery and kubernetes executor

get_event_buffer(self, dag_ids=None)[source]

Returns and flush the event buffer from celery and kubernetes executor

Parameters

dag_ids -- to dag_ids to return events for, if None returns all

Returns

a dict of events

try_adopt_task_instances(self, tis: List[TaskInstance])[source]

Try to adopt running task instances that have been abandoned by a SchedulerJob dying.

Anything that is not adopted will be cleared by the scheduler (and then become eligible for re-scheduling)

Returns

any TaskInstances that were unable to be adopted

Return type

list[airflow.models.TaskInstance]

end(self)[source]

End celery and kubernetes executor

terminate(self)[source]

Terminate celery and kubernetes executor

_router(self, simple_task_instance: SimpleTaskInstance)[source]

Return either celery_executor or kubernetes_executor

Parameters

simple_task_instance -- SimpleTaskInstance

Returns

celery_executor or kubernetes_executor

Return type

Union[CeleryExecutor, KubernetesExecutor]

debug_dump(self)[source]

Called in response to SIGUSR2 by the scheduler

Was this entry helpful?