Secrets backend

New in version 1.10.10.

In addition to retrieving connections & variables from environment variables or the metastore database, you can enable an alternative secrets backend to retrieve Airflow connections or Airflow variables, such as Google Cloud Secret Manager, Hashicorp Vault Secrets or you can roll your own.

Note

The Airflow UI only shows connections and variables stored in the Metadata DB and not via any other method. If you use an alternative secrets backend, check inside your backend to view the values of your variables and connections.

You can also get Airflow configurations with sensitive data from the Secrets Backend. See Setting Configuration Options for more details.

Search path

When looking up a connection/variable, by default Airflow will search environment variables first and metastore database second.

If you enable an alternative secrets backend, it will be searched first, followed by environment variables, then metastore. This search ordering is not configurable.

Configuration

The [secrets] section has the following options:

[secrets]
backend =
backend_kwargs =

Set backend to the fully qualified class name of the backend you want to enable.

You can provide backend_kwargs with json and it will be passed as kwargs to the __init__ method of your secrets backend.

If you want to check which secret backend is currently set, you can use airflow config get-value secrets backend command as in the example below.

$ airflow config get-value secrets backend
airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend

Roll your own secrets backend

A secrets backend is a subclass of airflow.secrets.BaseSecretsBackend and must implement either get_connection() or get_conn_uri().

After writing your backend class, provide the fully qualified class name in the backend key in the [secrets] section of airflow.cfg.

Additional arguments to your SecretsBackend can be configured in airflow.cfg by supplying a JSON string to backend_kwargs, which will be passed to the __init__ of your SecretsBackend. See Configuration for more details, and SSM Parameter Store for an example.

Note

If you are rolling your own secrets backend, you don't strictly need to use airflow's URI format. But doing so makes it easier to switch between environment variables, the metastore, and your secrets backend.

Was this entry helpful?