airflow.providers.amazon.aws.sensors.emr_job_flow

Module Contents

class airflow.providers.amazon.aws.sensors.emr_job_flow.EmrJobFlowSensor(*, job_flow_id: str, target_states: Optional[Iterable[str]] = None, failed_states: Optional[Iterable[str]] = None, **kwargs)[source]

Bases: airflow.providers.amazon.aws.sensors.emr_base.EmrBaseSensor

Asks for the state of the EMR JobFlow (Cluster) until it reaches any of the target states. If it fails the sensor errors, failing the task.

With the default target states, sensor waits cluster to be terminated. When target_states is set to ['RUNNING', 'WAITING'] sensor waits until job flow to be ready (after 'STARTING' and 'BOOTSTRAPPING' states)

Parameters

  • job_flow_id (str) -- job_flow_id to check the state of

  • target_states (list[str]) -- the target states, sensor waits until job flow reaches any of these states

  • failed_states (list[str]) -- the failure states, sensor fails when job flow reaches any of these states

template_fields = ['job_flow_id', 'target_states', 'failed_states'][source]
template_ext = [][source]
get_emr_response(self)[source]

Make an API call with boto3 and get cluster-level details.

See also

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr.html#EMR.Client.describe_cluster

Returns

response

Return type

dict[str, Any]

static state_from_response(response: Dict[str, Any])[source]

Get state from response dictionary.

Parameters

response (dict[str, Any]) -- response from AWS API

Returns

current state of the cluster

Return type

str

static failure_message_from_response(response: Dict[str, Any])[source]

Get failure message from response dictionary.

Parameters

response (dict[str, Any]) -- response from AWS API

Returns

failure message

Return type

Optional[str]

Was this entry helpful?