Executors are the mechanism by which task instances get run. They have a common API and are “pluggable”, meaning you can swap executors based on your installation needs.
Airflow can only have one executor configured at a time; this is set by the
executor option in the
section of the configuration file.
Built-in executors are referred to by name, for example:
executor = KubernetesExecutor
You can also write your own custom executors, and refer to them by their full path:
executor = my_company.executors.MyCustomExecutor
For more information on Airflow’s configuration, see Setting Configuration Options.
If you want to check which executor is currently set, you can use the
airflow config get-value core executor command:
$ airflow config get-value core executor
There are two types of executor - those that run tasks locally (inside the
scheduler process), and those that run their tasks remotely (usually via a pool of workers). Airflow comes configured with the
SequentialExecutor by default, which is a local executor, and the safest option for execution, but we strongly recommend you change this to
LocalExecutor for small, single-machine installations, or one of the remote executors for a multi-machine/cloud installation.
Something that often confuses new users of Airflow is that they don’t need to run a separate
executor process. This is because the executor’s logic runs inside the
scheduler process - if you’re running a scheduler, you’re running the executor.