airflow.providers.amazon.aws.executors.batch.utils¶

Attributes¶

`CommandType`
`ExecutorConfigType`
`CONFIG_GROUP_NAME`
`CONFIG_DEFAULTS`

Exceptions¶

BatchExecutorException

Thrown when something unexpected has occurred within the AWS Batch ecosystem.

Classes¶

`BatchQueuedJob`	Represents a Batch job that is queued. The job will be run in the next heartbeat.
`BatchJobInfo`	Contains information about a currently running Batch job.
`BatchJob`	Data Transfer Object for an AWS Batch Job.
`BatchJobCollection`	A collection to manage running Batch Jobs.
`BatchSubmitJobKwargsConfigKeys`	Keys loaded into the config which are valid Batch submit_job kwargs.
`AllBatchConfigKeys`	All keys loaded into the config which are related to the Batch Executor.

Module Contents¶

airflow.providers.amazon.aws.executors.batch.utils.CommandType[source]¶

airflow.providers.amazon.aws.executors.batch.utils.ExecutorConfigType[source]¶

airflow.providers.amazon.aws.executors.batch.utils.CONFIG_GROUP_NAME = 'aws_batch_executor'[source]¶

airflow.providers.amazon.aws.executors.batch.utils.CONFIG_DEFAULTS[source]¶

class airflow.providers.amazon.aws.executors.batch.utils.BatchQueuedJob[source]¶

Represents a Batch job that is queued. The job will be run in the next heartbeat.

key: airflow.models.taskinstance.TaskInstanceKey[source]¶

command: CommandType[source]¶

queue: str[source]¶

executor_config: ExecutorConfigType[source]¶

attempt_number: int[source]¶

next_attempt_time: datetime.datetime[source]¶

class airflow.providers.amazon.aws.executors.batch.utils.BatchJobInfo[source]¶

Contains information about a currently running Batch job.

cmd: CommandType[source]¶

queue: str[source]¶

config: ExecutorConfigType[source]¶

class airflow.providers.amazon.aws.executors.batch.utils.BatchJob(job_id, status, status_reason=None)[source]¶

Data Transfer Object for an AWS Batch Job.

STATE_MAPPINGS[source]¶

job_id[source]¶

status[source]¶

status_reason = None[source]¶

get_job_state()[source]¶

Return the state of the job.

__repr__()[source]¶: Return a visual representation of the Job status.

class airflow.providers.amazon.aws.executors.batch.utils.BatchJobCollection[source]¶

A collection to manage running Batch Jobs.

key_to_id: dict[airflow.models.taskinstance.TaskInstanceKey, str][source]¶

id_to_key: dict[str, airflow.models.taskinstance.TaskInstanceKey][source]¶

id_to_failure_counts: dict[str, int][source]¶

id_to_job_info: dict[str, BatchJobInfo][source]¶

add_job(job_id, airflow_task_key, airflow_cmd, queue, exec_config, attempt_number)[source]¶

Add a job to the collection.

pop_by_id(job_id)[source]¶

Delete job from collection based off of Batch Job ID.

failure_count_by_id(job_id)[source]¶

Get the number of times a job has failed given a Batch Job Id.

increment_failure_count(job_id)[source]¶

Increment the failure counter given a Batch Job Id.

get_all_jobs()[source]¶

Get all AWS ARNs in collection.

__len__()[source]¶: Return the number of jobs in collection.

class airflow.providers.amazon.aws.executors.batch.utils.BatchSubmitJobKwargsConfigKeys[source]¶

Bases: airflow.providers.amazon.aws.executors.utils.base_config_keys.BaseConfigKeys

Keys loaded into the config which are valid Batch submit_job kwargs.

JOB_NAME = 'job_name'[source]¶

JOB_QUEUE = 'job_queue'[source]¶

JOB_DEFINITION = 'job_definition'[source]¶

EKS_PROPERTIES_OVERRIDE = 'eks_properties_override'[source]¶

NODE_OVERRIDE = 'node_override'[source]¶

class airflow.providers.amazon.aws.executors.batch.utils.AllBatchConfigKeys[source]¶

Bases: BatchSubmitJobKwargsConfigKeys

All keys loaded into the config which are related to the Batch Executor.

MAX_SUBMIT_JOB_ATTEMPTS = 'max_submit_job_attempts'[source]¶

AWS_CONN_ID = 'conn_id'[source]¶

SUBMIT_JOB_KWARGS = 'submit_job_kwargs'[source]¶

REGION_NAME = 'region_name'[source]¶

CHECK_HEALTH_ON_STARTUP = 'check_health_on_startup'[source]¶

exception airflow.providers.amazon.aws.executors.batch.utils.BatchExecutorException[source]¶

Bases: Exception

Thrown when something unexpected has occurred within the AWS Batch ecosystem.