airflow.providers.amazon.aws.executors.aws_lambda.lambda_executor¶

Classes¶

AwsLambdaExecutor

An Airflow Executor that submits tasks to AWS Lambda asynchronously.

Module Contents¶

class airflow.providers.amazon.aws.executors.aws_lambda.lambda_executor.AwsLambdaExecutor(*args, **kwargs)[source]¶

Bases: airflow.executors.base_executor.BaseExecutor

An Airflow Executor that submits tasks to AWS Lambda asynchronously.

When execute_async() is called, the executor invokes a specified AWS Lambda function (asynchronously) with a payload that includes the task command and a unique task key.

The Lambda function writes its result directly to an SQS queue, which is then polled by this executor to update task state in Airflow.

queued_tasks: dict[airflow.models.taskinstancekey.TaskInstanceKey, airflow.executors.workloads.All][source]¶

pending_tasks: collections.deque[source]¶

running_tasks: dict[str, airflow.models.taskinstancekey.TaskInstanceKey][source]¶

lambda_function_name[source]¶

sqs_queue_url[source]¶

dlq_url[source]¶

qualifier[source]¶

max_invoke_attempts[source]¶

attempts_since_last_successful_connection = 0[source]¶

IS_BOTO_CONNECTION_HEALTHY = False[source]¶

start()[source]¶: Call this when the Executor is run for the first time by the scheduler.

check_health()[source]¶

Check the health of the Lambda and SQS connections.

For lambda: Use get_function to test if the lambda connection works and the function can be described. For SQS: Use get_queue_attributes is used as a close analog to describe to test if the SQS connection is working.

load_connections(check_connection=True)[source]¶

Retrieve the AWS connection via Hooks to leverage the Airflow connection system.

Parameters:: check_connection (bool) – If True, check the health of the connection after loading it.

sync()[source]¶

Sync the executor with the current state of tasks.

Check in on currently running tasks and attempt to run any new tasks that have been queued.

queue_workload(workload, session)[source]¶

execute_async(key, command, queue=None, executor_config=None)[source]¶

Save the task to be executed in the next sync by inserting the commands into a queue.

Parameters:

key (airflow.models.taskinstancekey.TaskInstanceKey) – A unique task key (typically a tuple identifying the task instance).
command (airflow.providers.amazon.aws.executors.aws_lambda.utils.CommandType) – The shell command string to execute.
executor_config – (Unused) to keep the same signature as the base.
queue – (Unused) to keep the same signature as the base.

attempt_task_runs()[source]¶

Attempt to run tasks that are queued in the pending_tasks.

Each task is submitted to AWS Lambda with a payload containing the task key and command. The task key is used to track the task’s state in Airflow.

sync_running_tasks()[source]¶

Poll the SQS queue for messages indicating task completion.

Each message is expected to contain a JSON payload with ‘task_key’ and ‘return_code’. Based on the return code, update the task state accordingly.

process_queue(queue_url)[source]¶

Poll the SQS queue for messages indicating task completion.

Each message is expected to contain a JSON payload with ‘task_key’ and ‘return_code’.

Based on the return code, update the task state accordingly.

try_adopt_task_instances(tis)[source]¶

Adopt task instances which have an external_executor_id (the serialized task key).

Anything that is not adopted will be cleared by the scheduler and becomes eligible for re-scheduling.

Parameters:: tis (collections.abc.Sequence[airflow.models.taskinstance.TaskInstance]) – The task instances to adopt.

end(heartbeat_interval=10)[source]¶

End execution. Poll until all outstanding tasks are marked as completed.

This is a blocking call and async Lambda tasks can not be cancelled, so this will wait until all tasks are either completed or the timeout is reached.

Parameters:: heartbeat_interval – The interval in seconds to wait between checks for task completion.

terminate()[source]¶: Get called when the daemon receives a SIGTERM.