airflow.providers.amazon.aws.triggers.emr¶

Classes¶

`EmrAddStepsTrigger`	Poll for the status of EMR steps until they reach terminal state.
`EmrCreateJobFlowTrigger`	Asynchronously poll the boto3 API and wait for the JobFlow to finish executing.
`EmrTerminateJobFlowTrigger`	Asynchronously poll the boto3 API and wait for the JobFlow to finish terminating.
`EmrContainerTrigger`	Poll for the status of EMR container until reaches terminal state.
`EmrStepSensorTrigger`	Poll for the status of EMR container until reaches terminal state.
`EmrServerlessCreateApplicationTrigger`	Poll an Emr Serverless application and wait for it to be created.
`EmrServerlessStartApplicationTrigger`	Poll an Emr Serverless application and wait for it to be started.
`EmrServerlessStopApplicationTrigger`	Poll an Emr Serverless application and wait for it to be stopped.
`EmrServerlessStartJobTrigger`	Poll an Emr Serverless job run and wait for it to be completed.
`EmrServerlessDeleteApplicationTrigger`	Poll an Emr Serverless application and wait for it to be deleted.
`EmrServerlessCancelJobsTrigger`	Trigger for canceling a list of jobs in an EMR Serverless application.

Module Contents¶

class airflow.providers.amazon.aws.triggers.emr.EmrAddStepsTrigger(job_flow_id, step_ids, waiter_delay, waiter_max_attempts, aws_conn_id='aws_default')[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Poll for the status of EMR steps until they reach terminal state.

Parameters:

job_flow_id (str) – job_flow_id which contains the steps to check the state of
step_ids (list[str]) – steps to check the state of
waiter_delay (int) – polling period in seconds to check for the status
waiter_max_attempts (int) – The maximum number of attempts to be made
aws_conn_id (str | None) – Reference to AWS connection id

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrCreateJobFlowTrigger(job_flow_id, aws_conn_id=None, waiter_delay=30, waiter_max_attempts=60)[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Asynchronously poll the boto3 API and wait for the JobFlow to finish executing.

Parameters:

job_flow_id (str) – The id of the job flow to wait for.
waiter_delay (int) – The amount of time in seconds to wait between attempts.
waiter_max_attempts (int) – The maximum number of attempts to be made.
aws_conn_id (str | None) – The Airflow connection used for AWS credentials.

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrTerminateJobFlowTrigger(job_flow_id, aws_conn_id=None, waiter_delay=30, waiter_max_attempts=60)[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Asynchronously poll the boto3 API and wait for the JobFlow to finish terminating.

Parameters:

job_flow_id (str) – ID of the EMR Job Flow to terminate
waiter_delay (int) – The amount of time in seconds to wait between attempts.
waiter_max_attempts (int) – The maximum number of attempts to be made.
aws_conn_id (str | None) – The Airflow connection used for AWS credentials.

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrContainerTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', waiter_delay=30, waiter_max_attempts=sys.maxsize)[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Poll for the status of EMR container until reaches terminal state.

Parameters:

virtual_cluster_id (str) – Reference Emr cluster id
job_id (str) – job_id to check the state
aws_conn_id (str | None) – Reference to AWS connection id
waiter_delay (int) – polling period in seconds to check for the status
waiter_max_attempts (int) – The maximum number of attempts to be made. Defaults to an infinite wait.

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrStepSensorTrigger(job_flow_id, step_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Poll for the status of EMR container until reaches terminal state.

Parameters:

job_flow_id (str) – job_flow_id which contains the step check the state of
step_id (str) – step to check the state of
waiter_delay (int) – polling period in seconds to check for the status
waiter_max_attempts (int) – The maximum number of attempts to be made
aws_conn_id (str | None) – Reference to AWS connection id

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrServerlessCreateApplicationTrigger(application_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Poll an Emr Serverless application and wait for it to be created.

Parameters:

application_id (str) – The ID of the application being polled.
waiter_max_attempts (int) – The maximum number of attempts to be made
aws_conn_id (str | None) – Reference to AWS connection id

Waiter_delay:

polling period in seconds to check for the status

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrServerlessStartApplicationTrigger(application_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Poll an Emr Serverless application and wait for it to be started.

Parameters:

application_id (str) – The ID of the application being polled.
waiter_max_attempts (int) – The maximum number of attempts to be made
aws_conn_id (str | None) – Reference to AWS connection id

Waiter_delay:

polling period in seconds to check for the status

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrServerlessStopApplicationTrigger(application_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Poll an Emr Serverless application and wait for it to be stopped.

Parameters:

application_id (str) – The ID of the application being polled.
waiter_max_attempts (int) – The maximum number of attempts to be made
aws_conn_id (str | None) – Reference to AWS connection id.

Waiter_delay:

polling period in seconds to check for the status

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrServerlessStartJobTrigger(application_id, job_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Poll an Emr Serverless job run and wait for it to be completed.

Parameters:

application_id (str) – The ID of the application the job in being run on.
job_id (str | None) – The ID of the job run.
waiter_max_attempts (int) – The maximum number of attempts to be made
aws_conn_id (str | None) – Reference to AWS connection id

Waiter_delay:

polling period in seconds to check for the status

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrServerlessDeleteApplicationTrigger(application_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Poll an Emr Serverless application and wait for it to be deleted.

Parameters:

application_id (str) – The ID of the application being polled.
waiter_max_attempts (int) – The maximum number of attempts to be made
aws_conn_id (str | None) – Reference to AWS connection id

Waiter_delay:

polling period in seconds to check for the status

hook()[source]¶

Override in subclasses to return the right hook.

class airflow.providers.amazon.aws.triggers.emr.EmrServerlessCancelJobsTrigger(application_id, aws_conn_id, waiter_delay, waiter_max_attempts)[source]¶

Bases: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

Trigger for canceling a list of jobs in an EMR Serverless application.

Parameters:

application_id (str) – EMR Serverless application ID
aws_conn_id (str | None) – Reference to AWS connection id
waiter_delay (int) – Delay in seconds between each attempt to check the status
waiter_max_attempts (int) – Maximum number of attempts to check the status

hook()[source]¶

Override in subclasses to return the right hook.

property hook_instance: airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook[source]¶

This property is added for backward compatibility.