• Home
• airflow.providers.amazon
• airflow.providers.amazon.aws
• airflow.providers.amazon.aws.operators
• airflow.providers.amazon.aws.operators.ecs

airflow.providers.amazon.aws.operators.ecs

Module Contents

Classes

EcsProtocol	A structured Protocol for boto3.client('ecs'). This is used for type hints on
EcsTaskLogFetcher	Fetches Cloudwatch log events with specific interval as a thread
EcsOperator	Execute a task on AWS ECS (Elastic Container Service)
ECSOperator	This operator is deprecated.
ECSTaskLogFetcher	This class is deprecated.
ECSProtocol	This class is deprecated.

Functions

should_retry(exception)	Check if exception is related to ECS resource quota (CPU, MEM).
should_retry_eni(exception)	Check if exception is related to ENI (Elastic Network Interfaces).

airflow.providers.amazon.aws.operators.ecs.should_retry(exception)

Check if exception is related to ECS resource quota (CPU, MEM).

airflow.providers.amazon.aws.operators.ecs.should_retry_eni(exception)

Check if exception is related to ENI (Elastic Network Interfaces).

class airflow.providers.amazon.aws.operators.ecs.EcsProtocol

Bases: airflow.typing_compat.Protocol

A structured Protocol for boto3.client('ecs'). This is used for type hints on EcsOperator.client().

See also

• https://mypy.readthedocs.io/en/latest/protocols.html

• https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html

run_task(self, **kwargs)

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task

get_waiter(self, x)

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.get_waiter

describe_tasks(self, cluster, tasks)

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.describe_tasks

stop_task(self, cluster, task, reason)

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.stop_task

describe_task_definition(self, taskDefinition)

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.describe_task_definition

list_tasks(self, cluster, launchType, desiredStatus, family)

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.list_tasks

class airflow.providers.amazon.aws.operators.ecs.EcsTaskLogFetcher(*, aws_conn_id='aws_default', region_name=None, log_group, log_stream_name, fetch_interval, logger)

Bases: threading.Thread

Fetches Cloudwatch log events with specific interval as a thread and sends the log events to the info channel of the provided logger.

run(self)

Method representing the thread's activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object's constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

get_last_log_messages(self, number_messages)

get_last_log_message(self)

is_stopped(self)

stop(self)

class airflow.providers.amazon.aws.operators.ecs.EcsOperator(*, task_definition, cluster, overrides, aws_conn_id=None, region_name=None, launch_type='EC2', capacity_provider_strategy=None, group=None, placement_constraints=None, placement_strategy=None, platform_version=None, network_configuration=None, tags=None, awslogs_group=None, awslogs_region=None, awslogs_stream_prefix=None, awslogs_fetch_interval=timedelta(seconds=30), propagate_tags=None, quota_retry=None, reattach=False, number_logs_exception=10, **kwargs)

Bases: airflow.models.BaseOperator

Execute a task on AWS ECS (Elastic Container Service)

See also

For more information on how to use this operator, take a look at the guide: Overview

Parameters

• task_definition (str) -- the task definition name on Elastic Container Service

• cluster (str) -- the cluster name on Elastic Container Service

• overrides (dict) -- the same parameter that boto3 will receive (templated): https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task

• aws_conn_id (Optional[str]) -- connection id of AWS credentials / region name. If None, credential boto3 strategy will be used (http://boto3.readthedocs.io/en/latest/guide/configuration.html).

• region_name (Optional[str]) -- region name to use in AWS Hook. Override the region_name in connection (if provided)

• launch_type (str) -- the launch type on which to run your task ('EC2', 'EXTERNAL', or 'FARGATE')

• capacity_provider_strategy (Optional[list]) -- the capacity provider strategy to use for the task. When capacity_provider_strategy is specified, the launch_type parameter is omitted. If no capacity_provider_strategy or launch_type is specified, the default capacity provider strategy for the cluster is used.

• group (Optional[str]) -- the name of the task group associated with the task

• placement_constraints (Optional[list]) -- an array of placement constraint objects to use for the task

• placement_strategy (Optional[list]) -- an array of placement strategy objects to use for the task

• platform_version (Optional[str]) -- the platform version on which your task is running

• network_configuration (Optional[dict]) -- the network configuration for the task

• tags (Optional[dict]) -- a dictionary of tags in the form of {'tagKey': 'tagValue'}.

• awslogs_group (Optional[str]) -- the CloudWatch group where your ECS container logs are stored. Only required if you want logs to be shown in the Airflow UI after your job has finished.

• awslogs_region (Optional[str]) -- the region in which your CloudWatch logs are stored. If None, this is the same as the region_name parameter. If that is also None, this is the default AWS region based on your connection settings.

• awslogs_stream_prefix (Optional[str]) -- the stream prefix that is used for the CloudWatch logs. This is usually based on some custom name combined with the name of the container. Only required if you want logs to be shown in the Airflow UI after your job has finished.

• awslogs_fetch_interval (datetime.timedelta) -- the interval that the ECS task log fetcher should wait in between each Cloudwatch logs fetches.

• quota_retry (Optional[dict]) -- Config if and how to retry the launch of a new ECS task, to handle transient errors.

• reattach (bool) -- If set to True, will check if the task previously launched by the task_instance is already running. If so, the operator will attach to it instead of starting a new task. This is to avoid relaunching a new task when the connection drops between Airflow and ECS while the task is running (when the Airflow worker is restarted for example).

• number_logs_exception (int) -- Number of lines from the last Cloudwatch logs to return in the AirflowException if an ECS task is stopped (to receive Airflow alerts with the logs of what failed in the code running in ECS).

ui_color = #f0ede4

template_fields :Sequence[str] = ['overrides']

template_fields_renderers

REATTACH_XCOM_KEY = ecs_task_arn

REATTACH_XCOM_TASK_ID_TEMPLATE = {task_id}_task_arn

execute(self, context, session=None)

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_hook(self)

Create and return an AwsHook.

on_kill(self)

Override this method to cleanup subprocesses when a task instance gets killed. Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up or it will leave ghost processes behind.

class airflow.providers.amazon.aws.operators.ecs.ECSOperator(*args, **kwargs)

Bases: EcsOperator

This operator is deprecated. Please use airflow.providers.amazon.aws.operators.ecs.EcsOperator.

class airflow.providers.amazon.aws.operators.ecs.ECSTaskLogFetcher(*args, **kwargs)

Bases: EcsTaskLogFetcher

This class is deprecated. Please use airflow.providers.amazon.aws.operators.ecs.EcsTaskLogFetcher.

class airflow.providers.amazon.aws.operators.ecs.ECSProtocol

Bases: EcsProtocol

This class is deprecated. Please use airflow.providers.amazon.aws.operators.ecs.EcsProtocol.

Previous Next

Was this entry helpful?