Stable (2.10.5)2.10.52.10.42.10.32.10.22.10.12.10.02.9.32.9.22.9.12.9.02.8.42.8.32.8.22.8.12.8.02.7.32.7.22.7.12.7.02.6.32.6.22.6.12.6.02.5.32.5.22.5.12.5.02.4.32.4.22.4.12.4.02.3.42.3.32.3.22.3.12.3.02.2.52.2.42.2.32.2.22.2.12.2.02.1.42.1.32.1.22.1.12.1.02.0.32.0.22.0.12.0.01.10.151.10.141.10.131.10.121.10.111.10.101.10.91.10.81.10.71.10.61.10.51.10.41.10.31.10.21.10.1
Stable (2.10.5)2.10.52.10.42.10.32.10.22.10.12.10.02.9.32.9.22.9.12.9.02.8.42.8.32.8.22.8.12.8.02.7.32.7.22.7.12.7.02.6.32.6.22.6.12.6.02.5.32.5.22.5.12.5.02.4.32.4.22.4.12.4.02.3.42.3.32.3.22.3.12.3.02.2.52.2.42.2.32.2.22.2.12.2.02.1.42.1.32.1.22.1.12.1.02.0.32.0.22.0.12.0.01.10.151.10.141.10.131.10.121.10.111.10.101.10.91.10.81.10.71.10.61.10.51.10.41.10.31.10.21.10.1

Dask Executor

airflow.executors.dask_executor.DaskExecutor allows you to run Airflow tasks in a Dask Distributed cluster.

Dask clusters can be run on a single machine or on remote networks. For complete details, consult the Distributed documentation.

To create a cluster, first start a Scheduler:

# default settings for a local cluster
DASK_HOST=127.0.0.1
DASK_PORT=8786

dask-scheduler --host $DASK_HOST --port $DASK_PORT
Copy to clipboard

Next start at least one Worker on any machine that can connect to the host:

dask-worker $DASK_HOST:$DASK_PORT
Copy to clipboard

Edit your airflow.cfg to set your executor to airflow.executors.dask_executor.DaskExecutor and provide the Dask Scheduler address in the [dask] section.

Please note:

  • Each Dask worker must be able to import Airflow and any dependencies you require.

  • Dask does not support queues. If an Airflow task was created with a queue, a warning will be raised but the task will be submitted to the cluster.

Was this entry helpful?