airflow.providers.amazon.aws.hooks.emr

Module Contents

Classes

EmrHook

Interact with AWS EMR. emr_conn_id is only necessary for using the

EmrContainerHook

Interact with AWS EMR Virtual Cluster to run, poll jobs and return job status
class airflow.providers.amazon.aws.hooks.emr.EmrHook(emr_conn_id: Optional[str] = default_conn_name, *args, **kwargs)[source]

Bases: airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook

Interact with AWS EMR. emr_conn_id is only necessary for using the create_job_flow method.

Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHook.

See also

AwsBaseHook

conn_name_attr = emr_conn_id[source]
default_conn_name = emr_default[source]
conn_type = emr[source]
hook_name = Elastic MapReduce[source]
get_cluster_id_by_name(self, emr_cluster_name: str, cluster_states: List[str]) → Optional[str][source]

Fetch id of EMR cluster with given name and (optional) states. Will return only if single id is found.

Parameters

  • emr_cluster_name (str) -- Name of a cluster to find

  • cluster_states (list) -- State(s) of cluster to find

Returns

id of the EMR cluster

create_job_flow(self, job_flow_overrides: Dict[str, Any]) → Dict[str, Any][source]

Creates a job flow using the config from the EMR connection. Keys of the json extra hash may have the arguments of the boto3 run_job_flow method. Overrides for this config may be passed as the job_flow_overrides.

class airflow.providers.amazon.aws.hooks.emr.EmrContainerHook(*args: Any, virtual_cluster_id: str = None, **kwargs: Any)[source]

Bases: airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook

Interact with AWS EMR Virtual Cluster to run, poll jobs and return job status Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHook.

See also

AwsBaseHook

Parameters

virtual_cluster_id (str) -- Cluster ID of the EMR on EKS virtual cluster

INTERMEDIATE_STATES = ['PENDING', 'SUBMITTED', 'RUNNING'][source]
FAILURE_STATES = ['FAILED', 'CANCELLED', 'CANCEL_PENDING'][source]
SUCCESS_STATES = ['COMPLETED'][source]
submit_job(self, name: str, execution_role_arn: str, release_label: str, job_driver: dict, configuration_overrides: Optional[dict] = None, client_request_token: Optional[str] = None) → str[source]

Submit a job to the EMR Containers API and and return the job ID. A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query, that you submit to Amazon EMR on EKS. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr-containers.html#EMRContainers.Client.start_job_run # noqa: E501

Parameters

  • name (str) -- The name of the job run.

  • execution_role_arn (str) -- The IAM role ARN associated with the job run.

  • release_label (str) -- The Amazon EMR release version to use for the job run.

  • job_driver (dict) -- Job configuration details, e.g. the Spark job parameters.

  • configuration_overrides (dict) -- The configuration overrides for the job run, specifically either application configuration or monitoring configuration.

  • client_request_token (str) -- The client idempotency token of the job run request. Use this if you want to specify a unique ID to prevent two jobs from getting started.

Returns

Job ID

get_job_failure_reason(self, job_id: str) → Optional[str][source]

Fetch the reason for a job failure (e.g. error message). Returns None or reason string.

Parameters

job_id (str) -- Id of submitted job run

Returns

str

check_query_status(self, job_id: str) → Optional[str][source]

Fetch the status of submitted job run. Returns None or one of valid query states. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr-containers.html#EMRContainers.Client.describe_job_run # noqa: E501 :param job_id: Id of submitted job run :type job_id: str :return: str

poll_query_status(self, job_id: str, max_tries: Optional[int] = None, poll_interval: int = 30) → Optional[str][source]

Poll the status of submitted job run until query state reaches final state. Returns one of the final states.

Parameters

  • job_id (str) -- Id of submitted job run

  • max_tries (int) -- Number of times to poll for query state before function exits

  • poll_interval (int) -- Time (in seconds) to wait between calls to check query status on EMR

Returns

str

stop_query(self, job_id: str) → Dict[source]

Cancel the submitted job_run

Parameters

job_id (str) -- Id of submitted job_run

Returns

dict

Was this entry helpful?