Parameters reference¶
The following tables lists the configurable parameters of the Airflow chart and their default values.
Sections:
Common¶
Parameter |
Description |
Default |
---|---|---|
|
airflow_local_settings file as a string (can be templated). |
|
|
Airflow version (Used to make some decisions based on Airflow Version being deployed). |
|
|
Settings to go into the mounted airflow.cfg |
|
|
Default airflow repository. Overrides all the specific images below. |
|
|
Default airflow tag to deploy. |
|
|
Airflow executor. |
|
|
The Fernet key used to encrypt passwords (can only be set during install, not upgrade). |
|
|
Enable PgBouncer. |
|
|
Enable default user creation. |
|
|
This string (can be templated) will be mounted into the Airflow webserver as a custom webserver_config.py. You can bake a webserver_config.py in to your image instead. |
Examples: webserverConfig: |-
from airflow import configuration as conf
# The SQLAlchemy connection string.
SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
# Flask-WTF flag for CSRF
CSRF_ENABLED = True
|
|
The Flask secret key for Airflow Webserver to encrypt browser session. |
|
Airflow¶
Parameter |
Description |
Default |
---|---|---|
|
Airflow home directory. Used for mount paths. |
|
|
Whether various Airflow components launch pods. |
|
|
Git branch |
|
|
Git sync container name. |
|
|
Name of a Secret containing the repo GIT_SYNC_USERNAME and GIT_SYNC_PASSWORD. |
|
|
Repository depth. |
|
|
Enable Git sync. |
|
|
Environment variables for git sync container. |
Examples: env:
- name: GIT_SYNC_TIMEOUT
value: '60'
|
|
Mount additional volumes into git sync container. |
|
|
When using a ssh private key, the contents of your known_hosts file. |
Examples: knownHosts: |-
<host1>,<ip1> <key1>
<host2>,<ip2> <key2>
knownHosts: <host1>,<ip1> <key1>
|
|
The number of consecutive failures allowed before aborting. |
|
|
Git repository. |
|
|
Resources on workers git-sync sidecar |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
Git revision. |
|
|
Name of a Secret containing the repo sshKeySecret. |
|
|
Subpath within the repo where dags are located. |
|
|
Git sync container run as user parameter. |
|
|
Interval between git sync attempts in seconds. |
|
|
Access mode of the persistent volume. |
|
|
Enable persistent volume for storing dags. |
|
|
The name of an existing PVC to use. |
|
|
Volume size for dags. |
|
|
If using a custom StorageClass, pass name here. |
|
|
Elasticsearch connection configuration. |
Examples: connection:
email: '...'
host: '...'
pass: '...'
user: '...'
|
|
Enable Elasticsearch task logging. |
|
|
A secret containing the connection string. |
|
|
Environment variables for all Airflow containers. |
Examples: env:
- name: MYENVVAR
value: something_fun
|
|
Extra env ‘items’ that will be added to the definition of Airflow containers; a string is expected (can be templated). |
Examples: extraEnv: |-
- name: AIRFLOW__CORE__LOAD_EXAMPLES
value: True
|
|
Extra envFrom ‘items’ that will be added to the definition of Airflow containers; a string is expected (can be templated). |
Examples: extraEnvFrom: |-
- secretRef:
name: '{{ .Release.Name }}-airflow-connections'
extraEnvFrom: |-
- configMapRef:
name: '{{ .Release.Name }}-airflow-variables'
|
|
The Fernet key secret name. |
|
|
Group of airflow user. |
|
|
Enable persistent volume for storing logs. |
|
|
The name of an existing PVC to use. |
|
|
Volume size for logs. |
|
|
If using a custom StorageClass, pass name here. |
|
|
Whether the KubernetesExecutor can launch workers and pods in multiple namespaces. If true, it creates |
|
|
The contents of |
Examples: podTemplate: |-
apiVersion: v1
kind: Pod
metadata:
name: dummy-name
labels:
tier: airflow
component: worker
release: my-release
spec:
priorityClassName: high-priority
containers:
- name: base
...
|
|
Secrets for all Airflow containers. |
Examples: secret:
- envName: SecretEnvVar
secretKey: somekey
secretName: somesecret
|
|
User of airflow user. |
|
|
The Secret name containing Flask secret_key for the Webserver. |
|
Images¶
Parameter |
Description |
Default |
---|---|---|
|
The airflow image pull policy. |
|
|
The airflow image repository. |
|
|
The airflow image tag. |
|
|
The flower image pull policy. |
|
|
The flower image repository. |
|
|
The flower image tag. |
|
|
The gitSync image pull policy. |
|
|
The gitSync image repository. |
|
|
The gitSync image tag. |
|
|
The PgBouncer image pull policy. |
|
|
The PgBouncer image repository. |
|
|
The PgBouncer image tag. |
|
|
The PgBouncer exporter image pull policy. |
|
|
The PgBouncer exporter image repository. |
|
|
The PgBouncer exporter image tag. |
|
|
The pod_template image pull policy. |
|
|
The pod_template image repository. |
|
|
The pod_template image tag. |
|
|
The redis image pull policy. |
|
|
The redis image repository. |
|
|
The redis image tag. |
|
|
The statsd image pull policy. |
|
|
The statsd image repository. |
|
|
The statsd image tag. |
|
Ports¶
Parameter |
Description |
Default |
---|---|---|
|
Airflow UI port. |
|
|
Flower UI port. |
|
|
PgBouncer port. |
|
|
PgBouncer scrape port. |
|
|
Redis port. |
|
|
StatsD ingest port. |
|
|
StatsD scrape port. |
|
|
Worker logs port. |
|
Database¶
Parameter |
Description |
Default |
---|---|---|
|
The name of the database. |
|
|
The database host. |
|
|
The user’s password. |
|
|
The database port. |
|
|
The database protocol. |
|
|
The database SSL parameter. |
|
|
The database user. |
|
|
Metadata connection string secret. |
|
|
Result backend connection configuration. |
|
|
The name of the database. |
|
|
The database host. |
|
|
The database password. |
|
|
The database port. |
|
|
The database protocol. |
|
|
The database SSL parameter. |
|
|
The database user. |
|
|
Result backend connection string secret. |
|
|
Enable PostgreSQL subchart. |
|
|
PostgreSQL password. |
|
|
PostgreSQL username. |
|
PgBouncer¶
Parameter |
Description |
Default |
---|---|---|
|
Specify scheduling constraints for PgBouncer pods. |
|
|
The allowed ciphers, might be ‘fast’, ‘normal’ or list ciphers separated with ‘:’. |
|
|
The PgBouncer config Secret name. |
|
|
Add extra general PgBouncer ini configuration: https://www.pgbouncer.org/config.html |
|
|
Add extra metadata database specific PgBouncer ini configuration: https://www.pgbouncer.org/config.html#section-databases |
|
|
Add extra result backend database specific PgBouncer ini configuration: https://www.pgbouncer.org/config.html#section-databases |
|
|
Additional NetworkPolicies as needed. |
|
|
Log successful logins. |
|
|
Log disconnections with reasons. |
|
|
Maximum clients that can connect to PgBouncer (higher = more file descriptors). |
|
|
Metadata pool size. |
|
|
Resources for the PgBouncer metric exporter. |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
Select certain nodes for PgBouncer pods. |
|
|
Max unavailable pods for PgBouncer. |
|
|
Enabled PodDistributionBudget. |
|
|
Resources for the PgBouncer pods. |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
Result backend pool size. |
|
|
Extra annotations for the PgBouncer Service. |
|
|
Annotations to add to the worker Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Certificate Authority for server side |
|
|
Server Certificate for server side |
|
|
Private key used to authenticate with the server |
|
|
SSL mode for PgBouncer. |
|
|
Specify Tolerations for PgBouncer pods. |
|
|
PgBouncer run as user parameter. |
|
|
Increase PgBouncer verbosity. |
|
Scheduler¶
Parameter |
Description |
Default |
---|---|---|
|
Specify scheduling constraints for scheduler pods. |
|
|
Args to use when running the Airflow scheduler (templated). |
|
|
Command to use when running the Airflow scheduler (templated). |
|
|
Launch additional containers into scheduler. |
|
|
Add additional init containers into scheduler. |
|
|
Mount additional volumes into scheduler. |
|
|
Mount additional volumes into scheduler. |
|
|
Scheduler Liveness probe failure threshold. |
|
|
Scheduler Liveness probe initial delay. |
|
|
Scheduler Liveness probe period seconds. |
|
|
Scheduler Liveness probe timeout seconds. |
|
|
Args to use when running the Airflow scheduler log groomer sidecar (templated). |
|
|
Command to use when running the Airflow scheduler log groomer sidecar (templated). |
|
|
Whether to deploy the Airflow scheduler log groomer sidecar. |
|
|
Resources for log groomer sidecar. |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
Select certain nodes for scheduler pods. |
|
|
Max unavailable pods for scheduler. |
|
|
Enable pod disruption budget. |
|
|
Airflow 2.0 allows users to run multiple schedulers. This feature is only recommended for MySQL 8+ and PostgreSQL |
|
|
Resources for scheduler pods. |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
This setting tells Kubernetes that its ok to evict when it wants to scale a node down. |
|
|
Annotations to add to the scheduler Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Specifies the strategy used to replace old Pods by new ones when deployed as a Deployment (when not using LocalExecutor and workers.persistence). |
|
|
Specify Tolerations for scheduler pods. |
|
|
Specifies the strategy used to replace old Pods by new ones when deployed as a StatefulSet (when using LocalExecutor and workers.persistence). |
|
Webserver¶
Parameter |
Description |
Default |
---|---|---|
|
Specify scheduling constraints for webserver pods. |
|
|
Allow webserver to read k8s pod logs. Useful when you don’t have an external log store. |
|
|
Args to use when running the Airflow webserver (templated). |
|
|
Command to use when running the Airflow webserver (templated). |
|
|
Default user email address. |
|
|
Default user firstname. |
|
|
Default user lastname. |
|
|
Default user password. |
|
|
Default user role. |
|
|
Default user username. |
|
|
Launch additional containers into webserver. |
|
|
Add additional init containers into webserver. |
|
|
Additional NetworkPolicies as needed (Deprecated - renamed to webserver.networkPolicy.ingress.from). |
|
|
Mount additional volumes into webserver. |
|
|
Mount additional volumes into webserver. |
|
|
Webserver Liveness probe failure threshold. |
|
|
Webserver Liveness probe initial delay. |
|
|
Webserver Liveness probe period seconds. |
|
|
Webserver Liveness probe timeout seconds. |
|
|
Peers for webserver NetworkPolicy ingress. |
|
|
Ports for webserver NetworkPolicy ingress (if from is set). |
Examples: ports:
- port: sidecar
|
|
Select certain nodes for webserver pods. |
|
|
Webserver Readiness probe failure threshold. |
|
|
Webserver Readiness probe initial delay. |
|
|
Webserver Readiness probe period seconds. |
|
|
Webserver Readiness probe timeout seconds. |
|
|
How many Airflow webserver replicas should run. |
|
|
Resources for webserver pods. |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
Annotations for the webserver Service. |
|
|
Webserver Service loadBalancerIP. |
|
|
Ports for the webserver Service. |
Examples: ports:
- name: airflow-ui
port: 80
targetPort: airflow-ui
ports:
- name: only_sidecar
port: 80
targetPort: 8888
|
|
Webserver Service type. |
|
|
Annotations to add to the webserver Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Specifies the strategy used to replace old Pods by new ones. |
|
|
Specify Tolerations for webserver pods. |
|
Workers¶
Parameter |
Description |
Default |
---|---|---|
|
Specify scheduling constraints for worker pods. |
|
|
Args to use when running Airflow workers (templated). |
|
|
Command to use when running Airflow workers (templated). |
|
|
Launch additional containers into workers. |
|
|
Add additional init containers into workers. |
|
|
Mount additional volumes into workers. |
|
|
Mount additional volumes into workers. |
|
|
Specify HostAliases for workers. |
Examples: hostAliases:
- hostnames:
- test.hostname.one
ip: 127.0.0.2
hostAliases:
- hostnames:
- test.hostname.two
ip: 127.0.0.3
|
|
How many seconds KEDA will wait before scaling to zero. |
|
|
Allow KEDA autoscaling. Persistence.enabled must be set to false to use KEDA. |
|
|
Maximum number of workers created by KEDA. |
|
|
Minimum number of workers created by KEDA. |
|
|
Labels used in matchLabels for namespace in the PgBouncer NetworkPolicy. |
|
|
How often KEDA polls the airflow DB to report new scale requests to the HPA. |
|
|
Enable Kerberos sidecar for the worker. |
|
|
Resources on workers kerberos sidecar |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
Args to use when running the Airflow workers log groomer sidecar (templated). |
|
|
Command to use when running the Airflow workers log groomer sidecar (templated). |
|
|
Resources for Airflow workers log groomer sidecar. |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
Select certain nodes for worker pods. |
|
|
Enable persistent volumes. |
|
|
Execute init container to chown log directory. This is currently only needed in kind, due to usage of local-path provisioner. |
|
|
Volume size for worker StatefulSet. |
|
|
If using a custom StorageClass, pass name ref to all StatefulSets here. |
|
|
Number of Airflow Celery workers in StatefulSet. |
|
|
Resources on workers |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
This setting tells Kubernetes that it’s ok to evict when it wants to scale a node down. |
|
|
Annotations to add to the worker Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Specifies the strategy used to replace old Pods by new ones when deployed as a Deployment. |
|
|
Grace period for tasks to finish after SIGTERM is sent from Kubernetes. |
|
|
Specify Tolerations for worker pods. |
|
|
Specifies the strategy used to replace old Pods by new ones when deployed as a StatefulSet. |
|
Flower¶
Parameter |
Description |
Default |
---|---|---|
|
Specify scheduling constraints for Flower pods. |
|
|
Args to use when running flower (templated). |
|
|
Command to use when running flower (templated). |
|
|
Enable Flower. |
|
|
Launch additional containers into the flower pods. |
|
|
Additional NetworkPolicies as needed (Deprecated - renamed to flower.networkPolicy.ingress.from). |
|
|
Mount additional volumes into the flower pods. |
|
|
Peers for flower NetworkPolicy ingress. |
|
|
Ports for flower NetworkPolicy ingress (if from is set). |
Examples: ports:
- port: sidecar
|
|
Select certain nodes for Flower pods. |
|
|
Password use to access Flower. |
|
|
Resources for Flower pods. |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
A secret containing the user and password pair. |
|
|
Annotations for the flower Service. |
|
|
Flower Service loadBalancerIP. |
|
|
Ports for the flower Service. |
Examples: ports:
- name: flower-ui
port: 8080
targetPort: flower-ui
|
|
Flower Service type. |
|
|
Annotations to add to the worker Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Specify Tolerations for Flower pods. |
|
|
Username use to access Flower. |
|
Redis¶
Parameter |
Description |
Default |
---|---|---|
|
Direct url to the redis broker (when using an external redis instance) (can only be set during install, not upgrade). |
|
|
Redis broker URL secret. |
|
|
Specify scheduling constraints for Redis pods. |
|
|
Enable the Redis provisioned by the chart (you can also use an external Redis instance with data.brokerUrl or data.brokerUrlSecretName). |
|
|
Select certain nodes for Redis pods. |
|
|
If password is set, create secret with it, else generate a new one on install (can only be set during install, not upgrade). |
|
|
Redis password secret. |
|
|
Enable persistent volumes. |
|
|
Volume size for Redis StatefulSet. |
|
|
If using a custom StorageClass, pass name ref to all StatefulSets here. |
|
|
Resources for the Redis pods |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
This setting tells Kubernetes that its ok to evict when it wants to scale a node down. |
|
|
Annotations to add to the worker Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Grace period for Redis to exit after SIGTERM is sent from Kubernetes. |
|
|
Specify Tolerations for Redis pods. |
|
Statsd¶
Parameter |
Description |
Default |
---|---|---|
|
Specify scheduling constraints for StatsD pods. |
|
|
Enable StatsD. |
|
|
Additional mappings for StatsD exporter. |
|
|
Additional NetworkPolicies as needed. |
|
|
Select certain nodes for StatsD pods. |
|
|
Resources for StatsD pods. |
Examples: resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
|
|
Extra annotations for the StatsD Service. |
|
|
Annotations to add to the StatsD Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Specify Tolerations for StatsD pods. |
|
|
StatsD run as user parameter. |
|
Jobs¶
Parameter |
Description |
Default |
---|---|---|
|
Specify scheduling constraints for cleanup pods. |
|
|
Enable cleanup. |
|
|
Select certain nodes for cleanup pods. |
|
|
Cleanup schedule. |
|
|
Annotations to add to the cleanup CronJob Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Specify Tolerations for cleanup pods. |
|
|
Specify scheduling constraints for the create user job pod. |
|
|
Annotations to add to the create user job pod. |
|
|
Annotations to add to the create user job job. |
|
|
Select certain nodes for the create user job pod. |
|
|
Annotations to add to the create user job Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Specify Tolerations for the create user job pod. |
|
|
Specify scheduling constraints for the migrate database job pod. |
|
|
Annotations to add to the migrate database job pod. |
|
|
Annotations to add to the migrate database job. |
|
|
Select certain nodes for the migrate database job pod. |
|
|
Annotations to add to the migrate database job Kubernetes ServiceAccount. |
|
|
Specifies whether a ServiceAccount should be created. |
|
|
The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
Specify Tolerations for the migrate database job pod. |
|
Kubernetes¶
Parameter |
Description |
Default |
---|---|---|
|
Specify scheduling constraints for all pods. |
|
|
Extra annotations to apply to the main Airflow configmap. |
|
|
Extra annotations to apply to all Airflow pods. |
|
|
Extra ConfigMaps that will be managed by the chart. |
Examples: extraConfigMaps:
'{{ .Release.Name }}-airflow-variables':
data: |-
AIRFLOW_VAR_HELLO_MESSAGE: 'Hi!'
AIRFLOW_VAR_KUBERNETES_NAMESPACE: '{{ .Release.Namespace }}'
|
|
Extra secrets that will be managed by the chart. |
Examples: extraSecrets:
'{{ .Release.Name }}-airflow-connections':
data: |-
AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string'
AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string'
stringData: 'AIRFLOW_CONN_OTHER: ''other_conn'''
|
|
Add common labels to all objects and pods defined in this chart. |
|
|
Define default/max/min values for pods and containers in namespace. |
|
|
Enabled network policies. |
|
|
Select certain nodes for all pods. |
|
|
Define any ResourceQuotas for namespace. |
|
|
Specifies whether RBAC resources should be created. |
|
|
Registry connection configuration. |
Examples: connection:
email: '...'
host: '...'
pass: '...'
user: '...'
|
|
Registry connection string secret. |
|
|
Specify Tolerations for all pods. |
|
Ingress¶
Parameter |
Description |
Default |
---|---|---|
|
Enable ingress resource. |
|
|
Annotations for the flower Ingress. |
|
|
The hostname for the flower Ingress. |
|
|
The path for the flower Ingress. |
|
|
HTTP paths to add to the flower Ingress before the default path. |
|
|
HTTP paths to add to the flower Ingress after the default path. |
|
|
Enable TLS termination for the flower Ingress. |
|
|
The name of a pre-created Secret containing a TLS private key and certificate. |
|
|
Annotations for the web Ingress. |
|
|
The hostname for the web Ingress. |
|
|
The path for the web Ingress. |
|
|
HTTP paths to add to the web Ingress before the default path. |
|
|
HTTP paths to add to the web Ingress after the default path. |
|
|
Enable TLS termination for the web Ingress. |
|
|
The name of a pre-created Secret containing a TLS private key and certificate. |
|
Kerberos¶
Parameter |
Description |
Default |
---|---|---|
|
Name for kerberos credentials cache file. |
|
|
Path to mount shared volume for kerberos credentials cache. |
|
|
Contents of krb5.conf. |
|
|
Path to mount krb5.conf kerberos configuration file. |
|
|
Enable kerberos. |
|
|
Path to mount the keytab for refreshing credentials in the kerberos sidecar. |
|
|
Principal to use when refreshing kerberos credentials. |
|
|
How often (in seconds) airflow kerberos will reinitialize the credentials cache. |
|
Specify each parameter using the --set key=value[,key=value]
argument to helm install
. For example,
helm install my-release apache-airflow/airflow \
--set executor=CeleryExecutor \
--set enablePodLaunching=false .