ECS Operator¶

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation
Setup Connection.

Using Operator¶

Use the ECSOperator to run a task defined in AWS ECS.

In the following example, the task "hello_world" runs hello-world task in c cluster. It overrides the command in the hello-world-container container.

Before using ECSOperator, cluster and task definition need to be created.

airflow/providers/amazon/aws/example_dags/example_ecs_fargate.pyView Source

hello_world = ECSOperator(
    task_id="hello_world",
    dag=dag,
    aws_conn_id="aws_ecs",
    cluster="c",
    task_definition="hello-world",
    launch_type="FARGATE",
    overrides={
        "containerOverrides": [
            {
                "name": "hello-world-container",
                "command": ["echo", "hello", "world"],
            },
        ],
    },
    network_configuration={
        "awsvpcConfiguration": {
            "securityGroups": [os.environ.get("SECURITY_GROUP_ID", "sg-123abc")],
            "subnets": [os.environ.get("SUBNET_ID", "subnet-123456ab")],
        },
    },
    tags={
        "Customer": "X",
        "Project": "Y",
        "Application": "Z",
        "Version": "0.0.1",
        "Environment": "Development",
    },
    awslogs_group="/ecs/hello-world",
    awslogs_stream_prefix="prefix_b/hello-world-container",  # prefix with container name
)

More information¶

For further information, look at the documentation of run_task() method in boto3.