Apache Airflow aims to be a very Kubernetes-friendly project, and many users run Airflow from within a Kubernetes cluster in order to take advantage of the increased stability and autoscaling options that Kubernetes provides.
Helm Chart for Kubernetes¶
We maintain an official Helm chart for Airflow that helps you define, install, and upgrade deployment. The Helm Chart uses official Docker image and Dockerfile that is also maintained and released by the community.
The Kubernetes Executor allows you to run all the Airflow tasks on Kubernetes as separate Pods.
Pod Mutation Hook¶
The Airflow local settings file (
airflow_local_settings.py) can define a
that has the ability to mutate pod objects before sending them to the Kubernetes client
for scheduling. It receives a single argument as a reference to pod objects, and
are expected to alter its attributes.
This could be used, for instance, to add sidecar or init containers to every worker pod launched by KubernetesExecutor or KubernetesPodOperator.
from kubernetes.client.models import V1Pod def pod_mutation_hook(pod: V1Pod): pod.metadata.annotations["airflow.apache.org/launched-by"] = "Tests"