Configuration Reference

This page contains the list of all available Airflow configurations for the apache-airflow-providers-amazon provider that can be set in the airflow.cfg file or using environment variables.

Note

The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.

Note

For more information see Setting Configuration Options.

[aws]

This section contains settings for Amazon Web Services (AWS) integration.

cloudwatch_task_handler_json_serializer

New in version 8.7.2.

By default, when logging non-string messages, all non-json objects are logged as null. Except datetime objects which are ISO formatted. Users can optionally use a repr serializer or provide their own JSON serializer for any non-JSON-serializable objects in the logged message.

  • airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize uses repr (be aware there is the potential of logging sensitive data depending on the repr method of logged objects)

  • airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize_legacy uses null.

If a custom serializer is provided, it must adhere to Callable[[Any], str | None], where None serializes to null (e.g. def my_serializer(o: Any) -> str | None). Since this is on the logging path and it’s possible there’s an exception being handled, special care should be taken to fail gracefully without raising a new exception inside of your serializer.

Type

string

Default

airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize_legacy

Environment Variable

AIRFLOW__AWS__CLOUDWATCH_TASK_HANDLER_JSON_SERIALIZER

Example

airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize

session_factory

New in version 3.1.1.

Full import path to the class which implements a custom session factory for boto3.session.Session. For more details please have a look at Session Factory.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS__SESSION_FACTORY

Example

my_company.aws.MyCustomSessionFactory

[aws_auth_manager]

This section only applies if you are using the AwsAuthManager. In other words, if you set [core] auth_manager = airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager in Airflow’s configuration.

avp_policy_store_id

New in version 8.12.0.

Amazon Verified Permissions’ policy store ID where all the policies defining user permissions in Airflow are stored. Required.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_AUTH_MANAGER__AVP_POLICY_STORE_ID

conn_id

New in version 8.12.0.

The Airflow connection (i.e. credentials) used by the AWS auth manager to make API calls to AWS Identity Center and Amazon Verified Permissions.

Type

string

Default

aws_default

Environment Variable

AIRFLOW__AWS_AUTH_MANAGER__CONN_ID

Example

aws_default

enable

New in version 8.12.0.

AWS auth manager is not ready to be used. Turn on this flag to use it anyway. Do that at your own risk since the AWS auth manager is not in an usable state.

Type

boolean

Default

False

Environment Variable

AIRFLOW__AWS_AUTH_MANAGER__ENABLE

Example

True

region_name

New in version 8.10.

The name of the AWS Region where Amazon Verified Permissions is configured. Required.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_AUTH_MANAGER__REGION_NAME

Example

us-east-1

saml_metadata_url

New in version 8.12.0.

SAML metadata XML file provided by AWS Identity Center. This URL can be found in the AWS Identity Center console. Required.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_AUTH_MANAGER__SAML_METADATA_URL

Example

https://portal.sso.<region>.amazonaws.com/saml/metadata/XXXXXXXXXX

[aws_ecs_executor]

This section only applies if you are using the AwsEcsExecutor in Airflow’s [core] configuration. For more information on any of these execution parameters, see the link below: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/client/run_task.html For boto3 credential management, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

assign_public_ip

New in version 8.10.

Whether to assign a public IP address to the containers launched by the ECS executor. For more info see url to Boto3 docs above.

Type

boolean

Default

False

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__ASSIGN_PUBLIC_IP

Example

True

capacity_provider_strategy

New in version 8.17.

The capacity provider strategy to use for the task.

If a Capacity Provider Strategy is specified, the Launch Type parameter must be omitted. If no Capacity Provider Strategy or Launch Type is specified, the Default CapacityProvider Strategy for the cluster is used, if present.

When you use cluster auto scaling, you must specify Capacity Provider Strategy and not Launch Type.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__CAPACITY_PROVIDER_STRATEGY

Example

[{'capacityProvider': 'cp1', 'weight': 5}, {'capacityProvider': 'cp2', 'weight': 1}]

check_health_on_startup

New in version 8.11.

Whether or not to check the ECS Executor health on startup.

Type

boolean

Default

True

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__CHECK_HEALTH_ON_STARTUP

Example

True

cluster

New in version 8.10.

Name of the Amazon ECS Cluster. Required.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__CLUSTER

Example

ecs_executor_cluster

conn_id

New in version 8.10.

The Airflow connection (i.e. credentials) used by the ECS executor to make API calls to AWS ECS.

Type

string

Default

aws_default

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__CONN_ID

Example

aws_default

container_name

New in version 8.10.

Name of the container that will be used to execute Airflow tasks via the ECS executor. The container should be specified in the ECS Task Definition and will receive an airflow CLI command as an additional parameter to its entrypoint. For more info see url to Boto3 docs above. Required.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__CONTAINER_NAME

Example

ecs_executor_container

launch_type

New in version 8.10.

Launch type can either be ‘FARGATE’ OR ‘EC2’. For more info see url to Boto3 docs above.

If a Launch Type is specified, the Capacity Provider Strategy parameter must be omitted. If no Capacity Provider Strategy or Launch Type is specified, the Default Capacity Provider Strategy for the cluster is used, if present.

If the launch type is EC2, the executor will attempt to place tasks on empty EC2 instances. If there are no EC2 instances available, no task is placed and this function will be called again in the next heart-beat.

If the launch type is FARGATE, this will run the tasks on new AWS Fargate instances.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__LAUNCH_TYPE

Example

FARGATE

max_run_task_attempts

New in version 8.10.

The maximum number of times the ECS Executor should attempt to run a task.

Type

integer

Default

3

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__MAX_RUN_TASK_ATTEMPTS

Example

3

platform_version

New in version 8.10.

The platform version the task uses. A platform version is only specified for tasks hosted on Fargate. If one isn’t specified, the LATEST platform version is used.

Type

string

Default

LATEST

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__PLATFORM_VERSION

Example

1.4.0

region_name

New in version 8.10.

The name of the AWS Region where Amazon ECS is configured. Required.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__REGION_NAME

Example

us-east-1

run_task_kwargs

New in version 8.10.

A JSON string containing arguments to provide the ECS run_task API (see url above).

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__RUN_TASK_KWARGS

Example

{"tags": {"key": "schema", "value": "1.0"}}

security_groups

New in version 8.10.

The comma-separated IDs of the security groups associated with the task. If you don’t specify a security group, the default security group for the VPC is used. There’s a limit of 5 security groups. For more info see url to Boto3 docs above.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__SECURITY_GROUPS

Example

sg-XXXX,sg-YYYY

subnets

New in version 8.10.

The comma-separated IDs of the subnets associated with the task or service. There’s a limit of 16 subnets. For more info see url to Boto3 docs above.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__SUBNETS

Example

subnet-XXXXXXXX,subnet-YYYYYYYY

task_definition

New in version 8.10.

The family and revision (family:revision) or full ARN of the task definition to run. If a revision isn’t specified, the latest ACTIVE revision is used. For more info see url to Boto3 docs above.

Type

string

Default

None

Environment Variable

AIRFLOW__AWS_ECS_EXECUTOR__TASK_DEFINITION

Example

executor_task_definition:LATEST

Was this entry helpful?