airflow.providers.amazon.aws.operators.batch
¶
An Airflow operator for AWS Batch services
See also
Module Contents¶
Classes¶
Execute a job on AWS Batch |
|
This operator is deprecated. |
- class airflow.providers.amazon.aws.operators.batch.BatchOperator(*, job_name, job_definition, job_queue, overrides, array_properties=None, parameters=None, job_id=None, waiters=None, max_retries=None, status_retries=None, aws_conn_id=None, region_name=None, tags=None, **kwargs)[source]¶
Bases:
airflow.models.BaseOperator
Execute a job on AWS Batch
See also
For more information on how to use this operator, take a look at the guide: Submit a new AWS Batch job
- Parameters
job_name (str) – the name for the job that will run on AWS Batch (templated)
job_definition (str) – the job definition name on AWS Batch
job_queue (str) – the queue name on AWS Batch
overrides (dict) – the containerOverrides parameter for boto3 (templated)
array_properties (Optional[dict]) – the arrayProperties parameter for boto3
parameters (Optional[dict]) – the parameters for boto3 (templated)
job_id (Optional[str]) – the job ID, usually unknown (None) until the submit_job operation gets the jobId defined by AWS Batch
waiters (Optional[Any]) – an
BatchWaiters
object (see note below); if None, polling is used with max_retries and status_retries.max_retries (Optional[int]) – exponential back-off retries, 4200 = 48 hours; polling is only used when waiters is None
status_retries (Optional[int]) – number of HTTP retries to get job status, 10; polling is only used when waiters is None
aws_conn_id (Optional[str]) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used.
region_name (Optional[str]) – region name to use in AWS Hook. Override the region_name in connection (if provided)
tags (Optional[dict]) – collection of tags to apply to the AWS Batch job submission if None, no tags are submitted
Note
Any custom waiters must return a waiter for these calls: .. code-block:: python
waiter = waiters.get_waiter(“JobExists”) waiter = waiters.get_waiter(“JobRunning”) waiter = waiters.get_waiter(“JobComplete”)
- on_kill(self)[source]¶
Override this method to cleanup subprocesses when a task instance gets killed. Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up or it will leave ghost processes behind.
- monitor_job(self, context)[source]¶
Monitor an AWS Batch job monitor_job can raise an exception or an AirflowTaskTimeout can be raised if execution_timeout is given while creating the task. These exceptions should be handled in taskinstance.py instead of here like it was previously done
- Raises
AirflowException
- class airflow.providers.amazon.aws.operators.batch.AwsBatchOperator(*args, **kwargs)[source]¶
Bases:
BatchOperator
This operator is deprecated. Please use
airflow.providers.amazon.aws.operators.batch.BatchOperator
.