airflow.contrib.operators.awsbatch_operator¶
Module Contents¶
-
class
airflow.contrib.operators.awsbatch_operator.AWSBatchOperator(job_name, job_definition, job_queue, overrides, array_properties=None, parameters=None, max_retries=MAX_RETRIES, status_retries=STATUS_RETRIES, aws_conn_id=None, region_name=None, **kwargs)[source]¶ Bases:
airflow.models.BaseOperatorExecute a job on AWS Batch Service
- Parameters
job_name (str) – the name for the job that will run on AWS Batch (templated)
job_definition (str) – the job definition name on AWS Batch
job_queue (str) – the queue name on AWS Batch
overrides (dict) – the same parameter that boto3 will receive on containerOverrides (templated) http://boto3.readthedocs.io/en/latest/reference/services/batch.html#Batch.Client.submit_job
array_properties (dict) – the same parameter that boto3 will receive on arrayProperties http://boto3.readthedocs.io/en/latest/reference/services/batch.html#Batch.Client.submit_job
parameters (dict) – the same parameter that boto3 will receive on parameters (templated) http://boto3.readthedocs.io/en/latest/reference/services/batch.html#Batch.Client.submit_job
max_retries (int) – exponential backoff retries while waiter is not merged, 4200 = 48 hours
status_retries (int) – number of retries to get job description (status), 10
aws_conn_id (str) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used (http://boto3.readthedocs.io/en/latest/guide/configuration.html).
region_name (str) – region name to use in AWS Hook. Override the region_name in connection (if provided)
-
_wait_for_task_ended(self)[source]¶ Try to use a waiter from the below pull request
If the waiter is not available apply a exponential backoff
docs.aws.amazon.com/general/latest/gr/api-retries.html
-
_poll_for_task_ended(self)[source]¶ Poll for job status
docs.aws.amazon.com/general/latest/gr/api-retries.html