airflow.executors.local_executor
¶
LocalExecutor
See also
For more information on how the LocalExecutor works, take a look at the guide: Local Executor
Module Contents¶
-
class
airflow.executors.local_executor.
LocalWorkerBase
(result_queue: ‘Queue[TaskInstanceStateType]’)[source]¶ Bases:
multiprocessing.Process
,airflow.utils.log.logging_mixin.LoggingMixin
LocalWorkerBase implementation to run airflow commands. Executes the given command and puts the result into a result queue when done, terminating execution.
- Parameters
result_queue – the queue to store result state
-
class
airflow.executors.local_executor.
LocalWorker
(result_queue: ‘Queue[TaskInstanceStateType]’, key: TaskInstanceKey, command: CommandType)[source]¶ Bases:
airflow.executors.local_executor.LocalWorkerBase
Local worker that executes the task.
- Parameters
result_queue – queue where results of the tasks are put.
key – key identifying task instance
command – Command to execute
-
class
airflow.executors.local_executor.
QueuedLocalWorker
(task_queue: ‘Queue[ExecutorWorkType]’, result_queue: ‘Queue[TaskInstanceStateType]’)[source]¶ Bases:
airflow.executors.local_executor.LocalWorkerBase
LocalWorker implementation that is waiting for tasks from a queue and will continue executing commands as they become available in the queue. It will terminate execution once the poison token is found.
- Parameters
task_queue – queue from which worker reads tasks
result_queue – queue where worker puts results after finishing tasks
-
class
airflow.executors.local_executor.
LocalExecutor
(parallelism: int = PARALLELISM)[source]¶ Bases:
airflow.executors.base_executor.BaseExecutor
LocalExecutor executes tasks locally in parallel. It uses the multiprocessing Python library and queues to parallelize the execution of tasks.
- Parameters
parallelism – how many parallel processes are run in the executor
-
class
UnlimitedParallelism
(executor: ‘LocalExecutor’)[source]¶ Implements LocalExecutor with unlimited parallelism, starting one process per each command to execute.
- Parameters
executor – the executor instance to implement.
-
execute_async
(self, key: TaskInstanceKey, command: CommandType, queue: Optional[str] = None, executor_config: Optional[Any] = None)[source]¶ Executes task asynchronously.
- Parameters
key – the key to identify the task instance
command – the command to execute
queue – Name of the queue
executor_config – configuration for the executor
-
class
LimitedParallelism
(executor: ‘LocalExecutor’)[source]¶ Implements LocalExecutor with limited parallelism using a task queue to coordinate work distribution.
- Parameters
executor – the executor instance to implement.
-
execute_async
(self, key: TaskInstanceKey, command: CommandType, queue: Optional[str] = None, executor_config: Optional[Any] = None)[source]¶ Executes task asynchronously.
- Parameters
key – the key to identify the task instance
command – the command to execute
queue – name of the queue
executor_config – configuration for the executor