airflow.providers.cncf.kubernetes.executors.local_kubernetes_executor
¶
Module Contents¶
Classes¶
Chooses between LocalExecutor and KubernetesExecutor based on the queue defined on the task. |
- class airflow.providers.cncf.kubernetes.executors.local_kubernetes_executor.LocalKubernetesExecutor(local_executor, kubernetes_executor)[source]¶
Bases:
airflow.executors.base_executor.BaseExecutor
Chooses between LocalExecutor and KubernetesExecutor based on the queue defined on the task.
When the task’s queue is the value of
kubernetes_queue
in section[local_kubernetes_executor]
of the configuration (default value: kubernetes), KubernetesExecutor is selected to run the task, otherwise, LocalExecutor is used.- property queued_tasks: dict[airflow.models.taskinstance.TaskInstanceKey, airflow.executors.base_executor.QueuedTaskInstanceType][source]¶
Return queued tasks from local and kubernetes executor.
- property running: set[airflow.models.taskinstance.TaskInstanceKey][source]¶
Return running tasks from local and kubernetes executor.
- property job_id: int | str | None[source]¶
Inherited attribute from BaseExecutor.
Since this is not really an executor, but a wrapper of executors we implemented it as property, so we can have custom setter.
- queue_command(task_instance, command, priority=1, queue=None)[source]¶
Queues command via local or kubernetes executor.
- queue_task_instance(task_instance, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, wait_for_past_depends_before_skipping=False, ignore_task_deps=False, ignore_ti_state=False, pool=None, cfg_path=None, **kwargs)[source]¶
Queues task instance via local or kubernetes executor.
- has_task(task_instance)[source]¶
Check if a task is either queued or running in either local or kubernetes executor.
- Parameters
task_instance (airflow.models.taskinstance.TaskInstance) – TaskInstance
- Returns
True if the task is known to this executor
- Return type
- get_event_buffer(dag_ids=None)[source]¶
Return and flush the event buffer from local and kubernetes executor.
- try_adopt_task_instances(tis)[source]¶
Try to adopt running task instances that have been abandoned by a SchedulerJob dying.
Anything that is not adopted will be cleared by the scheduler (and then become eligible for re-scheduling)
- Returns
any TaskInstances that were unable to be adopted
- Return type
collections.abc.Sequence[airflow.models.taskinstance.TaskInstance]
- cleanup_stuck_queued_tasks(tis)[source]¶
Handle remnants of tasks that were failed because they were stuck in queued.
Tasks can get stuck in queued. If such a task is detected, it will be marked as UP_FOR_RETRY if the task instance has remaining retries or marked as FAILED if it doesn’t.
- Parameters
tis (list[airflow.models.taskinstance.TaskInstance]) – List of Task Instances to clean up
- Returns
List of readable task instances for a warning message
- Return type
- revoke_task(*, ti)[source]¶
Attempt to remove task from executor.
It should attempt to ensure that the task is no longer running on the worker, and ensure that it is cleared out from internal data structures.
It should not change the state of the task in airflow, or add any events to the event buffer.
It should not raise any error.
- Parameters
ti (airflow.models.taskinstance.TaskInstance) – Task instance to remove
- send_callback(request)[source]¶
Send callback for execution.
- Parameters
request (airflow.callbacks.callback_requests.CallbackRequest) – Callback request to be executed.
- static get_cli_commands()[source]¶
Vends CLI commands to be included in Airflow CLI.
Override this method to expose commands via Airflow CLI to manage this executor. This can be commands to setup/teardown the executor, inspect state, etc. Make sure to choose unique names for those commands, to avoid collisions.