Autoscaling with KEDA

This feature is still experimental.

KEDA stands for Kubernetes Event Driven Autoscaling. KEDA is a custom controller that allows users to create custom bindings to the Kubernetes Horizontal Pod Autoscaler. The autoscaler will adjust the number of active Celery workers based on the number of tasks in queued or running state.

helm repo add kedacore https://kedacore.github.io/charts

helm repo update

kubectl create namespace keda

helm install keda kedacore/keda \
    --namespace keda \
    --version "v2.0.0"

Enable for the airflow instance by setting workers.keda.enabled=true in your helm command or in the values.yaml.

kubectl create namespace airflow
helm repo add apache-airflow https://airflow.apache.org
helm install airflow apache-airflow/airflow \
    --namespace airflow \
    --set executor=CeleryExecutor \
    --set workers.keda.enabled=true

A ScaledObject and an hpa will be created in the airflow namespace.

KEDA will derive the desired number of Celery workers by querying Airflow metadata database:

SELECT
    ceil(COUNT(*)::decimal / {{ .Values.config.celery.worker_concurrency }})
FROM task_instance
WHERE state='running' OR state='queued'

Note

Set Celery worker concurrency through the Helm value config.celery.worker_concurrency (i.e. instead of airflow.cfg or environment variables) so that the KEDA trigger will be consistent with the worker concurrency setting.

Was this entry helpful?