Production Guide¶

The following are things to consider when using this Helm chart in a production environment.

Database¶

You will want to use an external database instead of the one deployed with the chart by default. Both PostgresSQL and MySQL are supported. Supported versions can be found on the Set up a Database Backend page.

# Don't deploy postgres
postgre:
  enabled: false

# Use an external database
data:
  metadataConnection:
    user: ...
    pass: ...
    protocol: postgresql  # or 'mysql'
    host: ...
    port: ...
    db: ...

      
    

PgBouncer¶

If you are using PostgresSQL as your database, you will likely want to enable PgBouncer as well. Airflow can open a lot of database connections due to its distributed nature and using a connection pooler can significantly reduce the number of open connections on the database.

pgbouncer:
  enabled: true

Depending on the size of you Airflow instance, you may want to adjust the following as well (defaults are shown):

pgbouncer:
  # The maximum number of connections to PgBouncer
  maxClientConn: 100
  # The maximum number of server connections to the metadata database from PgBouncer
  metadataPoolSize: 10
  # The maximum number of server connections to the result backend database from PgBouncer
  resultBackendPoolSize: 5

      
    

DAG Files¶

See Manage DAGs files.

knownHosts¶

If you are using dags.gitSync.sshKeySecret, you should also set dags.gitSync.knownHosts. Here we will show the process for GitHub, but the same can be done for any provider:

Grab GitHub’s public key:

ssh-keyscan -t rsa github.com > github_public_key

Next, print the fingerprint for the public key:

ssh-keygen -lf github_public_key

Compare that output with GitHub’s SSH key fingerprints.

They match, right? Good. Now, add the public key to your values. It’ll look something like this:

dags:
  gitSync:
    knownHosts: |
      github.com ssh-rsa AAAA...FAaQ==

Accessing the Airflow UI¶

How you access the Airflow UI will depend on your environment, however the chart does support various options:

Ingress¶

You can create and configure Ingress objects. See the Ingress chart parameters. For more information on Ingress, see the Kubernetes Ingress documentation.

LoadBalancer Service¶

You can change the Service type for the webserver to be LoadBalancer, and set any necessary annotations:

webserver:
  service: LoadBalancer
  annotations: {}

For more information on LoadBalancer Services, see the Kubernetes LoadBalancer Service Documentation.

Logging¶

Depending on your choice of executor, task logs may not work out of the box. All logging choices can be found at Manage logs.

Metrics¶

The chart can support sending metrics to an existing StatsD instance or provide a Prometheus endpoint.

Prometheus¶

The metrics endpoint is available at svc/{{ .Release.Name }}-statsd:9102/metrics.

External StatsD¶

To use an external StatsD instance:

statsd:
  enabled: false
config:
  metrics:  # or 'scheduler' for Airflow 1
    statsd_on: true
    statsd_host: ...
    statsd_port: ...

      
    

Celery Backend¶

If you are using CeleryExecutor or CeleryKubernetesExecutor, you can bring your own Celery backend.

By default, the chart will deploy Redis. However, you can use any supported Celery backend instead:

redis:
  enabled: false
data:
  brokerUrl: redis://redis-user:password@redis-host:6379/0

For more information about setting up a Celery broker, refer to the exhaustive Celery documentation on the topic.