airflow.providers.amazon.aws.hooks.emr

Module Contents

class airflow.providers.amazon.aws.hooks.emr.EmrHook(emr_conn_id: Optional[str] = default_conn_name, *args, **kwargs)[source]

Bases: airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook

Interact with AWS EMR. emr_conn_id is only necessary for using the create_job_flow method.

Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHook.

See also

AwsBaseHook

conn_name_attr = emr_conn_id[source]
default_conn_name = emr_default[source]
conn_type = emr[source]
hook_name = Elastic MapReduce[source]
get_cluster_id_by_name(self, emr_cluster_name: str, cluster_states: List[str])[source]

Fetch id of EMR cluster with given name and (optional) states. Will return only if single id is found.

Parameters

  • emr_cluster_name (str) -- Name of a cluster to find

  • cluster_states (list) -- State(s) of cluster to find

Returns

id of the EMR cluster

create_job_flow(self, job_flow_overrides: Dict[str, Any])[source]

Creates a job flow using the config from the EMR connection. Keys of the json extra hash may have the arguments of the boto3 run_job_flow method. Overrides for this config may be passed as the job_flow_overrides.

Was this entry helpful?