airflow.providers.amazon.aws.secrets.secrets_manager

Objects relating to sourcing secrets from AWS Secrets Manager

Module Contents

Classes

SecretsManagerBackend

Retrieves Connection or Variables from AWS Secrets Manager
class airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend(connections_prefix: str = 'airflow/connections', variables_prefix: str = 'airflow/variables', config_prefix: str = 'airflow/config', profile_name: Optional[str] = None, sep: str = '/', full_url_mode: bool = True, extra_conn_words: Optional[dict] = None, **kwargs)[source]

Bases: airflow.secrets.BaseSecretsBackend, airflow.utils.log.logging_mixin.LoggingMixin

Retrieves Connection or Variables from AWS Secrets Manager

Configurable via airflow.cfg like so:

[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix": "airflow/connections"}

For example, if secrets prefix is airflow/connections/smtp_default, this would be accessible if you provide {"connections_prefix": "airflow/connections"} and request conn_id smtp_default. If variables prefix is airflow/variables/hello, this would be accessible if you provide {"variables_prefix": "airflow/variables"} and request variable key hello. And if config_prefix is airflow/config/sql_alchemy_conn, this would be accessible if you provide {"config_prefix": "airflow/config"} and request config key sql_alchemy_conn.

You can also pass additional keyword arguments like aws_secret_access_key, aws_access_key_id or region_name to this class and they would be passed on to Boto3 client.

There are two ways of storing secrets in Secret Manager for using them with this operator: storing them as a conn URI in one field, or taking advantage of native approach of Secrets Manager and storing them in multiple fields. There are certain words that will be searched in the name of fields for trying to retrieve a connection part. Those words are:

possible_words_for_conn_fields = {
    "user": ["user", "username", "login", "user_name"],
    "password": ["password", "pass", "key"],
    "host": ["host", "remote_host", "server"],
    "port": ["port"],
    "schema": ["database", "schema"],
    "conn_type": ["conn_type", "conn_id", "connection_type", "engine"],
}

However, these lists can be extended using the configuration parameter extra_conn_words. Also, you can have a field named extra for extra parameters for the conn. Please note that this extra field must be a valid JSON.

Parameters

  • connections_prefix (str) -- Specifies the prefix of the secret to read to get Connections. If set to None (null value in the configuration), requests for connections will not be sent to AWS Secrets Manager. If you don't want a connections_prefix, set it as an empty string

  • variables_prefix (str) -- Specifies the prefix of the secret to read to get Variables. If set to None (null value in the configuration), requests for variables will not be sent to AWS Secrets Manager. If you don't want a variables_prefix, set it as an empty string

  • config_prefix (str) -- Specifies the prefix of the secret to read to get Configurations. If set to None (null value in the configuration), requests for configurations will not be sent to AWS Secrets Manager. If you don't want a config_prefix, set it as an empty string

  • profile_name (str) -- The name of a profile to use. If not given, then the default profile is used.

  • sep (str) -- separator used to concatenate secret_prefix and secret_id. Default: "/"

  • full_url_mode (bool) -- if True, the secrets must be stored as one conn URI in just one field per secret. If False (set it as false in backend_kwargs), you can store the secret using different fields (password, user...).

  • extra_conn_words (dict) -- for using just when you set full_url_mode as false and store the secrets in different fields of secrets manager. You can add more words for each connection part beyond the default ones. The extra words to be searched should be passed as a dict of lists, each list corresponding to a connection part. The optional keys of the dict must be: user, password, host, schema, conn_type.

client(self)[source]

Create a Secrets Manager client

get_uri_from_secret(self, secret)[source]
get_conn_uri(self, conn_id: str)[source]

Get Connection Value

Parameters

conn_id (str) -- connection id

get_variable(self, key: str) → Optional[str][source]

Get Airflow Variable from Environment Variable :param key: Variable Key :return: Variable Value

get_config(self, key: str) → Optional[str][source]

Get Airflow Configuration :param key: Configuration Option Key :return: Configuration Option Value

Was this entry helpful?