Configuration Reference¶

This page contains the list of all available Airflow configurations for the apache-airflow-providers-amazon provider that can be set in the airflow.cfg file or using environment variables.

Note

The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.

Note

For more information see Setting Configuration Options.

Sections:

[aws]
[aws_auth_manager]
[aws_ecs_executor]

[aws]¶

This section contains settings for Amazon Web Services (AWS) integration.

cloudwatch_task_handler_json_serializer¶

By default, when logging non-string messages, all non-json objects are logged as null. Except datetime objects which are ISO formatted. Users can optionally use a repr serializer or provide their own JSON serializer for any non-JSON-serializable objects in the logged message.

airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize uses repr (be aware there is the potential of logging sensitive data depending on the repr method of logged objects)
airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize_legacy uses null.

If a custom serializer is provided, it must adhere to Callable[[Any], str | None], where None serializes to null (e.g. def my_serializer(o: Any) -> str | None). Since this is on the logging path and it’s possible there’s an exception being handled, special care should be taken to fail gracefully without raising a new exception inside of your serializer.

Type: string
Default: airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize_legacy
Environment Variable: AIRFLOW__AWS__CLOUDWATCH_TASK_HANDLER_JSON_SERIALIZER
Example: airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize

session_factory¶

Full import path to the class which implements a custom session factory for boto3.session.Session. For more details please have a look at Session Factory.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS__SESSION_FACTORY
Example: my_company.aws.MyCustomSessionFactory

[aws_auth_manager]¶

This section only applies if you are using the AwsAuthManager. In other words, if you set [core] auth_manager = airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager in Airflow’s configuration.

avp_policy_store_id¶

Amazon Verified Permissions’ policy store ID where all the policies defining user permissions in Airflow are stored. Required.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_AUTH_MANAGER__AVP_POLICY_STORE_ID

conn_id¶

The Airflow connection (i.e. credentials) used by the AWS auth manager to make API calls to AWS Identity Center and Amazon Verified Permissions.

Type: string
Default: aws_default
Environment Variable: AIRFLOW__AWS_AUTH_MANAGER__CONN_ID
Example: aws_default

enable¶

AWS auth manager is not ready to be used. Turn on this flag to use it anyway. Do that at your own risk since the AWS auth manager is not in an usable state.

Type: boolean
Default: False
Environment Variable: AIRFLOW__AWS_AUTH_MANAGER__ENABLE
Example: True

region_name¶

The name of the AWS Region where Amazon Verified Permissions is configured. Required.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_AUTH_MANAGER__REGION_NAME
Example: us-east-1

saml_metadata_url¶

SAML metadata XML file provided by AWS Identity Center. This URL can be found in the AWS Identity Center console. Required.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_AUTH_MANAGER__SAML_METADATA_URL
Example: https://portal.sso.<region>.amazonaws.com/saml/metadata/XXXXXXXXXX

[aws_ecs_executor]¶

This section only applies if you are using the AwsEcsExecutor in Airflow’s [core] configuration. For more information on any of these execution parameters, see the link below: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/client/run_task.html For boto3 credential management, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

assign_public_ip¶

Whether to assign a public IP address to the containers launched by the ECS executor. For more info see url to Boto3 docs above.

Type: boolean
Default: False
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__ASSIGN_PUBLIC_IP
Example: True

capacity_provider_strategy¶

The capacity provider strategy to use for the task.

If a Capacity Provider Strategy is specified, the Launch Type parameter must be omitted. If no Capacity Provider Strategy or Launch Type is specified, the Default CapacityProvider Strategy for the cluster is used, if present.

When you use cluster auto scaling, you must specify Capacity Provider Strategy and not Launch Type.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__CAPACITY_PROVIDER_STRATEGY
Example: [{'capacityProvider': 'cp1', 'weight': 5}, {'capacityProvider': 'cp2', 'weight': 1}]

check_health_on_startup¶

Whether or not to check the ECS Executor health on startup.

Type: boolean
Default: True
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__CHECK_HEALTH_ON_STARTUP
Example: True

cluster¶

Name of the Amazon ECS Cluster. Required.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__CLUSTER
Example: ecs_executor_cluster

conn_id¶

The Airflow connection (i.e. credentials) used by the ECS executor to make API calls to AWS ECS.

Type: string
Default: aws_default
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__CONN_ID
Example: aws_default

container_name¶

Name of the container that will be used to execute Airflow tasks via the ECS executor. The container should be specified in the ECS Task Definition and will receive an airflow CLI command as an additional parameter to its entrypoint. For more info see url to Boto3 docs above. Required.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__CONTAINER_NAME
Example: ecs_executor_container

launch_type¶

Launch type can either be ‘FARGATE’ OR ‘EC2’. For more info see url to Boto3 docs above.

If a Launch Type is specified, the Capacity Provider Strategy parameter must be omitted. If no Capacity Provider Strategy or Launch Type is specified, the Default Capacity Provider Strategy for the cluster is used, if present.

If the launch type is EC2, the executor will attempt to place tasks on empty EC2 instances. If there are no EC2 instances available, no task is placed and this function will be called again in the next heart-beat.

If the launch type is FARGATE, this will run the tasks on new AWS Fargate instances.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__LAUNCH_TYPE
Example: FARGATE

max_run_task_attempts¶

The maximum number of times the ECS Executor should attempt to run a task.

Type: integer
Default: 3
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__MAX_RUN_TASK_ATTEMPTS
Example: 3

platform_version¶

The platform version the task uses. A platform version is only specified for tasks hosted on Fargate. If one isn’t specified, the LATEST platform version is used.

Type: string
Default: LATEST
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__PLATFORM_VERSION
Example: 1.4.0

region_name¶

The name of the AWS Region where Amazon ECS is configured. Required.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__REGION_NAME
Example: us-east-1

run_task_kwargs¶

A JSON string containing arguments to provide the ECS run_task API (see url above).

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__RUN_TASK_KWARGS
Example: {"tags": {"key": "schema", "value": "1.0"}}

security_groups¶

The comma-separated IDs of the security groups associated with the task. If you don’t specify a security group, the default security group for the VPC is used. There’s a limit of 5 security groups. For more info see url to Boto3 docs above.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__SECURITY_GROUPS
Example: sg-XXXX,sg-YYYY

subnets¶

The comma-separated IDs of the subnets associated with the task or service. There’s a limit of 16 subnets. For more info see url to Boto3 docs above.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__SUBNETS
Example: subnet-XXXXXXXX,subnet-YYYYYYYY

task_definition¶

The family and revision (family:revision) or full ARN of the task definition to run. If a revision isn’t specified, the latest ACTIVE revision is used. For more info see url to Boto3 docs above.

Type: string
Default: None
Environment Variable: AIRFLOW__AWS_ECS_EXECUTOR__TASK_DEFINITION
Example: executor_task_definition:LATEST