airflow.providers.amazon.aws.hooks.base_aws

This module contains Base AWS Hook.

See also

For more information on how to use this hook, take a look at the guide: Amazon Web Services Connection

Module Contents

Classes

BaseSessionFactory

Base AWS Session Factory class to handle synchronous and async boto session creation.

AwsGenericHook

Generic class for interact with AWS.

AwsBaseHook

Base class for interact with AWS.

BaseAsyncSessionFactory

Base AWS Session Factory class to handle aiobotocore session creation.

AwsBaseAsyncHook

Interacts with AWS using aiobotocore asynchronously.

Functions

resolve_session_factory()

Resolves custom SessionFactory class

Attributes

BaseAwsConnection

SessionFactory

airflow.providers.amazon.aws.hooks.base_aws.BaseAwsConnection[source]
class airflow.providers.amazon.aws.hooks.base_aws.BaseSessionFactory(conn, region_name=None, config=None)[source]

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Base AWS Session Factory class to handle synchronous and async boto session creation. It can handle most of the AWS supported authentication methods.

User can also derive from this class to have full control of boto3 session creation or to support custom federation.

Note: Not all features implemented for synchronous sessions are available for async sessions.

See also

property extra_config: dict[str, Any][source]

AWS Connection extra_config.

property region_name: str | None[source]

AWS Region Name read-only property.

property config: Config | None[source]

Configuration for botocore client read-only property.

property role_arn: str | None[source]

Assume Role ARN from AWS Connection

conn()[source]

Cached AWS Connection Wrapper.

basic_session()[source]

Cached property with basic boto3.session.Session.

get_async_session()[source]
create_session(deferrable=False)[source]

Create boto3 or aiobotocore Session from connection config.

class airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook(aws_conn_id=default_conn_name, verify=None, region_name=None, client_type=None, resource_type=None, config=None)[source]

Bases: airflow.hooks.base.BaseHook, Generic[BaseAwsConnection]

Generic class for interact with AWS. This class provide a thin wrapper around the boto3 python library.

Parameters
property service_config: dict[source]
property region_name: str | None[source]

AWS Region Name read-only property.

property config: botocore.config.Config[source]

Configuration for botocore client read-only property.

property verify: bool | str | None[source]

Verify or not SSL certificates boto3 client/resource read-only property.

property async_conn[source]

Get an aiobotocore client to use for async operations.

property conn_region_name: str[source]

Get actual AWS Region Name from Hook connection (cached).

property conn_partition: str[source]

Get associated AWS Region Partition from Hook connection (cached).

conn_name_attr = 'aws_conn_id'[source]
default_conn_name = 'aws_default'[source]
conn_type = 'aws'[source]
hook_name = 'Amazon Web Services'[source]
conn_config()[source]

Get the Airflow Connection object and wrap it in helper (cached).

get_session(region_name=None, deferrable=False)[source]

Get the underlying boto3.session.Session(region_name=region_name).

get_client_type(region_name=None, config=None, deferrable=False)[source]

Get the underlying boto3 client using boto3 session

get_resource_type(region_name=None, config=None)[source]

Get the underlying boto3 resource using boto3 session

conn()[source]

Get the underlying boto3 client/resource (cached)

Returns

boto3.client or boto3.resource

Return type

BaseAwsConnection

conn_client_meta()[source]

Get botocore client metadata from Hook connection (cached).

get_conn()[source]

Get the underlying boto3 client/resource (cached)

Implemented so that caching works as intended. It exists for compatibility with subclasses that rely on a super().get_conn() method.

Returns

boto3.client or boto3.resource

Return type

BaseAwsConnection

get_credentials(region_name=None)[source]

Get the underlying botocore.Credentials object.

This contains the following authentication attributes: access_key, secret_key and token. By use this method also secret_key and token will mask in tasks logs.

expand_role(role, region_name=None)[source]

If the IAM role is a role name, get the Amazon Resource Name (ARN) for the role. If IAM role is already an IAM role ARN, no change is made.

Parameters

  • role (str) – IAM role name or ARN

  • region_name (str | None) – Optional region name to get credentials for

Returns

IAM role ARN

Return type

str

static retry(should_retry)[source]

A decorator that provides a mechanism to repeat requests in response to exceeding a temporary quote limit.

static get_ui_field_behaviour()[source]

Returns custom UI field behaviour for AWS Connection.

test_connection()[source]

Tests the AWS connection by call AWS STS (Security Token Service) GetCallerIdentity API.

See also

https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html

waiter_path()[source]
get_waiter(waiter_name, parameters=None, deferrable=False, client=None)[source]

First checks if there is a custom waiter with the provided waiter_name and uses that if it exists, otherwise it will check the service client for a waiter that matches the name and pass that through.

If deferrable is True, the waiter will be an AIOWaiter, generated from the client that is passed as a parameter. If deferrable is True, client must be provided.

Parameters

  • waiter_name (str) – The name of the waiter. The name should exactly match the name of the key in the waiter model file (typically this is CamelCase).

  • parameters (dict[str, str] | None) – will scan the waiter config for the keys of that dict, and replace them with the corresponding value. If a custom waiter has such keys to be expanded, they need to be provided here.

  • deferrable (bool) – If True, the waiter is going to be an async custom waiter.

list_waiters()[source]

Returns a list containing the names of all waiters for the service, official and custom.

class airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook(aws_conn_id=default_conn_name, verify=None, region_name=None, client_type=None, resource_type=None, config=None)[source]

Bases: AwsGenericHook[Union[boto3.client, boto3.resource]]

Base class for interact with AWS. This class provide a thin wrapper around the boto3 python library.

Parameters
airflow.providers.amazon.aws.hooks.base_aws.resolve_session_factory()[source]

Resolves custom SessionFactory class

airflow.providers.amazon.aws.hooks.base_aws.SessionFactory[source]
class airflow.providers.amazon.aws.hooks.base_aws.BaseAsyncSessionFactory(conn, region_name=None, config=None)[source]

Bases: BaseSessionFactory

Base AWS Session Factory class to handle aiobotocore session creation.

It currently, handles ENV, AWS secret key and STS client method assume_role provided in Airflow connection

async get_role_credentials()[source]

Get the role_arn, method credentials from connection details and get the role credentials detail

create_session(deferrable=False)[source]

Create aiobotocore Session from connection and config.

class airflow.providers.amazon.aws.hooks.base_aws.AwsBaseAsyncHook(aws_conn_id=default_conn_name, verify=None, region_name=None, client_type=None, resource_type=None, config=None)[source]

Bases: AwsBaseHook

Interacts with AWS using aiobotocore asynchronously.

Parameters

  • aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default botocore behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default botocore configuration would be used (and must be maintained on each worker node).

  • verify (bool | str | None) – Whether to verify SSL certificates.

  • region_name (str | None) – AWS region_name. If not specified then the default boto3 behaviour is used.

  • client_type (str | None) – boto3.client client_type. Eg ‘s3’, ‘emr’ etc

  • resource_type (str | None) – boto3.resource resource_type. Eg ‘dynamodb’ etc

  • config (Config | None) – Configuration for botocore client.

get_async_session()[source]

Get the underlying aiobotocore.session.AioSession(…).

async get_client_async()[source]

Get the underlying aiobotocore client using aiobotocore session

Was this entry helpful?