airflow.contrib.operators.awsbatch_operator

Module Contents

class airflow.contrib.operators.awsbatch_operator.AWSBatchOperator(job_name, job_definition, job_queue, overrides, max_retries=4200, aws_conn_id=None, region_name=None, **kwargs)

Bases: airflow.models.BaseOperator

Execute a job on AWS Batch Service

Parameters

• job_name (str) – the name for the job that will run on AWS Batch (templated)

• job_definition (str) – the job definition name on AWS Batch

• job_queue (str) – the queue name on AWS Batch

• overrides (dict) – the same parameter that boto3 will receive on containerOverrides (templated): http://boto3.readthedocs.io/en/latest/reference/services/batch.html#submit_job

• max_retries (int) – exponential backoff retries while waiter is not merged, 4200 = 48 hours

• aws_conn_id (str) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used (http://boto3.readthedocs.io/en/latest/guide/configuration.html).

• region_name (str) – region name to use in AWS Hook. Override the region_name in connection (if provided)

ui_color = #c3dae0

client

arn

template_fields = ['job_name', 'overrides']

execute(self, context)

_wait_for_task_ended(self)

Try to use a waiter from the below pull request

• https://github.com/boto/botocore/pull/1307

If the waiter is not available apply a exponential backoff

• docs.aws.amazon.com/general/latest/gr/api-retries.html

_check_success_task(self)

get_hook(self)

on_kill(self)