CeleryKubernetesExecutor consists of CeleryExecutor and KubernetesExecutor. It chooses an executor to use based on the queue defined on the task. When the queue is the value of
[celery_kubernetes_executor]of the configuration (default value: kubernetes), KubernetesExecutor is selected to run the task, otherwise, CeleryExecutor is used.
Return queued tasks from celery and kubernetes executor
Return running tasks from celery and kubernetes executor
This is a class attribute in BaseExecutor but since this is not really an executor, but a wrapper of executors we implement as property so we can have custom setter.
Number of new tasks this executor instance can accept
Start celery and kubernetes executor
queue_command(self, task_instance: TaskInstance, command: CommandType, priority: int = 1, queue: Optional[str] = None)¶
Queues command via celery or kubernetes executor
queue_task_instance(self, task_instance: TaskInstance, mark_success: bool = False, pickle_id: Optional[str] = None, ignore_all_deps: bool = False, ignore_depends_on_past: bool = False, ignore_task_deps: bool = False, ignore_ti_state: bool = False, pool: Optional[str] = None, cfg_path: Optional[str] = None)¶
Queues task instance via celery or kubernetes executor
has_task(self, task_instance: TaskInstance)¶
Checks if a task is either queued or running in either celery or kubernetes executor.
task_instance -- TaskInstance
True if the task is known to this executor
Heartbeat sent to trigger new jobs in celery and kubernetes executor
Returns and flush the event buffer from celery and kubernetes executor
dag_ids -- to dag_ids to return events for, if None returns all
a dict of events
try_adopt_task_instances(self, tis: List[TaskInstance])¶
Try to adopt running task instances that have been abandoned by a SchedulerJob dying.
Anything that is not adopted will be cleared by the scheduler (and then become eligible for re-scheduling)
End celery and kubernetes executor
Terminate celery and kubernetes executor
_router(self, simple_task_instance: SimpleTaskInstance)¶
Return either celery_executor or kubernetes_executor
Called in response to SIGUSR2 by the scheduler