airflow.executors.local_executor

LocalExecutor

See also

For more information on how the LocalExecutor works, take a look at the guide: Local Executor

Module Contents

Classes

LocalWorkerBase

LocalWorkerBase implementation to run airflow commands. Executes the given

LocalWorker

Local worker that executes the task.

QueuedLocalWorker

LocalWorker implementation that is waiting for tasks from a queue and will

LocalExecutor

LocalExecutor executes tasks locally in parallel.

Attributes

ExecutorWorkType

airflow.executors.local_executor.ExecutorWorkType[source]
class airflow.executors.local_executor.LocalWorkerBase(result_queue: Queue[TaskInstanceStateType])[source]

Bases: multiprocessing.Process, airflow.utils.log.logging_mixin.LoggingMixin

LocalWorkerBase implementation to run airflow commands. Executes the given command and puts the result into a result queue when done, terminating execution.

Parameters

result_queue – the queue to store result state

run(self)[source]

Method to be run in sub-process; can be overridden in sub-class

execute_work(self, key: airflow.models.taskinstance.TaskInstanceKey, command: airflow.executors.base_executor.CommandType) → None[source]

Executes command received and stores result state in queue.

Parameters

  • key – the key to identify the task instance

  • command – the command to execute

abstract do_work(self)[source]

Called in the subprocess and should then execute tasks

class airflow.executors.local_executor.LocalWorker(result_queue: Queue[TaskInstanceStateType], key: airflow.models.taskinstance.TaskInstanceKey, command: airflow.executors.base_executor.CommandType)[source]

Bases: LocalWorkerBase

Local worker that executes the task.

Parameters

  • result_queue – queue where results of the tasks are put.

  • key – key identifying task instance

  • command – Command to execute

do_work(self) → None[source]

Called in the subprocess and should then execute tasks

class airflow.executors.local_executor.QueuedLocalWorker(task_queue: Queue[ExecutorWorkType], result_queue: Queue[TaskInstanceStateType])[source]

Bases: LocalWorkerBase

LocalWorker implementation that is waiting for tasks from a queue and will continue executing commands as they become available in the queue. It will terminate execution once the poison token is found.

Parameters

  • task_queue – queue from which worker reads tasks

  • result_queue – queue where worker puts results after finishing tasks

do_work(self) → None[source]

Called in the subprocess and should then execute tasks

class airflow.executors.local_executor.LocalExecutor(parallelism: int = PARALLELISM)[source]

Bases: airflow.executors.base_executor.BaseExecutor

LocalExecutor executes tasks locally in parallel. It uses the multiprocessing Python library and queues to parallelize the execution of tasks.

Parameters

parallelism – how many parallel processes are run in the executor

class UnlimitedParallelism(executor: LocalExecutor)[source]

Implements LocalExecutor with unlimited parallelism, starting one process per each command to execute.

Parameters

executor – the executor instance to implement.

start(self) → None[source]

Starts the executor.

execute_async(self, key: airflow.models.taskinstance.TaskInstanceKey, command: airflow.executors.base_executor.CommandType, queue: Optional[str] = None, executor_config: Optional[Any] = None) → None[source]

Executes task asynchronously.

Parameters

  • key – the key to identify the task instance

  • command – the command to execute

  • queue – Name of the queue

  • executor_config – configuration for the executor

sync(self) → None[source]

Sync will get called periodically by the heartbeat method.

end(self) → None[source]

This method is called when the caller is done submitting job and wants to wait synchronously for the job submitted previously to be all done.

class LimitedParallelism(executor: LocalExecutor)[source]

Implements LocalExecutor with limited parallelism using a task queue to coordinate work distribution.

Parameters

executor – the executor instance to implement.

start(self) → None[source]

Starts limited parallelism implementation.

execute_async(self, key: airflow.models.taskinstance.TaskInstanceKey, command: airflow.executors.base_executor.CommandType, queue: Optional[str] = None, executor_config: Optional[Any] = None) → None[source]

Executes task asynchronously.

Parameters

  • key – the key to identify the task instance

  • command – the command to execute

  • queue – name of the queue

  • executor_config – configuration for the executor

sync(self)[source]

Sync will get called periodically by the heartbeat method.

end(self)[source]

Ends the executor. Sends the poison pill to all workers.

start(self) → None[source]

Starts the executor

execute_async(self, key: airflow.models.taskinstance.TaskInstanceKey, command: airflow.executors.base_executor.CommandType, queue: Optional[str] = None, executor_config: Optional[Any] = None) → None[source]

Execute asynchronously.

sync(self) → None[source]

Sync will get called periodically by the heartbeat method.

end(self) → None[source]

Ends the executor. :return:

terminate(self)[source]

Terminate the executor is not doing anything.

Was this entry helpful?