New in version 1.10.10.
In addition to retrieving connections & variables from environment variables or the metastore database, you can also enable alternative secrets backend to retrieve Airflow connections or Airflow variables via Apache Airflow Community provided backends in Secret backends.
The Airflow UI only shows connections and variables stored in the Metadata DB and not via any other method. If you use an alternative secrets backend, check inside your backend to view the values of your variables and connections.
You can also get Airflow configurations with sensitive data from the Secrets Backend. See Setting Configuration Options for more details.
When looking up a connection/variable, by default Airflow will search environment variables first and metastore database second.
If you enable an alternative secrets backend, it will be searched first, followed by environment variables, then metastore. This search ordering is not configurable.
When using environment variables or an alternative secrets backend to store secrets or variables, it is possible to create key collisions. In the event of a duplicated key between backends, all write operations will update the value in the metastore, but all read operations will return the first match for the requested key starting with the custom backend, then the environment variables and finally the metastore.
[secrets] section has the following options:
[secrets] backend = backend_kwargs =
backend to the fully qualified class name of the backend you want to enable.
You can provide
backend_kwargs with json and it will be passed as kwargs to the
__init__ method of
your secrets backend.
If you want to check which secret backend is currently set, you can use
airflow config get-value secrets backend command as in
the example below.
$ airflow config get-value secrets backend airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
Apache Airflow Community provided secret backends¶
Apache Airflow Community also releases community developed providers (Provider packages) and some of them also provide handlers that extend secret backends capability of Apache Airflow. You can see all those providers in Secret backends.
Roll your own secrets backend¶
A secrets backend is a subclass of
airflow.secrets.BaseSecretsBackend and must implement either
After writing your backend class, provide the fully qualified class name in the
backend key in the
Additional arguments to your SecretsBackend can be configured in
airflow.cfg by supplying a JSON string to
backend_kwargs, which will be passed to the
__init__ of your SecretsBackend.
See Configuration for more details, and SSM Parameter Store for an example.
Adapt to non-Airflow compatible secret formats for connections¶
The default implementation of Secret backend requires use of an Airflow-specific format of storing secrets for connections. Currently most community provided implementations require the connections to be stored as JSON or the Airflow Connection URI format (see Secret backends). However some organizations may need to store the credentials (passwords/tokens etc) in some other way, for example if the same credentials store needs to be used for multiple data platforms, or if you are using a service with a built-in mechanism of rotating the credentials that does not work with the Airflow-specific format. In this case you will need to roll your own secret backend as described in the previous chapter, possibly extending an existing secrets backend and adapting it to the scheme used by your organization.