LocalExecutor runs tasks by spawning processes in a controlled fashion in different modes.
Given that BaseExecutor has the option to receive a
parallelism parameter to limit the number of process spawned,
when this parameter is
0 the number of processes that LocalExecutor can spawn is unlimited.
The following strategies are implemented:
- Unlimited Parallelism (
self.parallelism == 0): In this strategy, LocalExecutor willspawn a process every time
execute_asyncis called, that is, every task submitted to the
LocalExecutorwill be executed in its own process. Once the task is executed and theresult stored in the
result_queue, the process terminates. There is no need for a
task_queuein this approach, since as soon as a task is received a new process will beallocated to the task. Processes used in this strategy are of class
- Limited Parallelism (
self.parallelism > 0): In this strategy, the
LocalExecutorspawnsthe number of processes equal to the value of
task_queueto coordinate the ingestion of tasks and the work distribution amongthe workers, which will take a task as soon as they are ready. During the lifecycle ofthe LocalExecutor, the worker processes are running waiting for tasks, once theLocalExecutor receives the call to shutdown the executor a poison token is sent to theworkers to terminate them. Processes used in this strategy are of class
SequentialExecutor could be thought as a
LocalExecutor with limited
parallelism of just 1 worker, i.e.
self.parallelism = 1.
This option could lead to the unification of the executor implementations, running
locally, into just one
LocalExecutor with multiple modes.