Module Contents



Interact with Amazon Elastic MapReduce Service (EMR).


Interact with Amazon EMR Serverless.


Interact with Amazon EMR Containers (Amazon EMR on EKS).

class, *args, **kwargs)[source]


Interact with Amazon Elastic MapReduce Service (EMR). Provide thick wrapper around boto3.client("emr").


emr_conn_id (str | None) – Amazon Elastic MapReduce Connection. This attribute is only necessary when using the

Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHook.

See also


conn_name_attr = 'emr_conn_id'[source]
default_conn_name = 'emr_default'[source]
conn_type = 'emr'[source]
hook_name = 'Amazon Elastic MapReduce'[source]
get_cluster_id_by_name(emr_cluster_name, cluster_states)[source]

Fetch id of EMR cluster with given name and (optional) states. Will return only if single id is found.

  • emr_cluster_name (str) – Name of a cluster to find

  • cluster_states (list[str]) – State(s) of cluster to find


id of the EMR cluster

Return type

str | None


Create and start running a new cluster (job flow).

This method uses EmrHook.emr_conn_id to receive the initial Amazon EMR cluster configuration. If EmrHook.emr_conn_id is empty or the connection does not exist, then an empty initial configuration is used.


job_flow_overrides (dict[str, Any]) – Is used to overwrite the parameters in the initial Amazon EMR configuration cluster. The resulting configuration will be used in the EMR.Client.run_job_flow().

add_job_flow_steps(job_flow_id, steps=None, wait_for_completion=False, waiter_delay=None, waiter_max_attempts=None, execution_role_arn=None)[source]

Add new steps to a running cluster.

  • job_flow_id (str) – The id of the job flow to which the steps are being added

  • steps (list[dict] | str | None) – A list of the steps to be executed by the job flow

  • wait_for_completion (bool) – If True, wait for the steps to be completed. Default is False

  • waiter_delay (int | None) – The amount of time in seconds to wait between attempts. Default is 5

  • waiter_max_attempts (int | None) – The maximum number of attempts to be made. Default is 100

  • execution_role_arn (str | None) – The ARN of the runtime role for a step on the cluster.


Return failed state for test Amazon Elastic MapReduce Connection (untestable).

We need to overwrite this method because this hook is based on AwsGenericHook, otherwise it will try to test connection to AWS STS by using the default boto3 credential strategy.

static get_ui_field_behaviour()[source]

Returns custom UI field behaviour for Amazon Elastic MapReduce Connection.

class*args, **kwargs)[source]


Interact with Amazon EMR Serverless. Provide thin wrapper around boto3.client("emr-serverless").

Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHook.


Get the underlying boto3 EmrServerlessAPIService client (cached)

waiter(get_state_callable, get_state_args, parse_response, desired_state, failure_states, object_type, action, countdown=25 * 60, check_interval_seconds=60)[source]

Will run the sensor until it turns True.

  • get_state_callable (Callable) – A callable to run until it returns True

  • get_state_args (dict) – Arguments to pass to get_state_callable

  • parse_response (list) – Dictionary keys to extract state from response of get_state_callable

  • desired_state (set) – Wait until the getter returns this value

  • failure_states (set) – A set of states which indicate failure and should throw an exception if any are reached before the desired_state

  • object_type (str) – Used for the reporting string. What are you waiting for? (application, job, etc)

  • action (str) – Used for the reporting string. What action are you waiting for? (created, deleted, etc)

  • countdown (int) – Total amount of time the waiter should wait for the desired state before timing out (in seconds). Defaults to 25 * 60 seconds.

  • check_interval_seconds (int) – Number of seconds waiter should wait before attempting to retry get_state_callable. Defaults to 60 seconds.

get_state(response, keys)[source]
class*args, virtual_cluster_id=None, **kwargs)[source]


Interact with Amazon EMR Containers (Amazon EMR on EKS). Provide thick wrapper around boto3.client("emr-containers").


virtual_cluster_id (str | None) – Cluster ID of the EMR on EKS virtual cluster

Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHook.

create_emr_on_eks_cluster(virtual_cluster_name, eks_cluster_name, eks_namespace, tags=None)[source]
submit_job(name, execution_role_arn, release_label, job_driver, configuration_overrides=None, client_request_token=None, tags=None)[source]

Submit a job to the EMR Containers API and return the job ID. A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query, that you submit to Amazon EMR on EKS.

  • name (str) – The name of the job run.

  • execution_role_arn (str) – The IAM role ARN associated with the job run.

  • release_label (str) – The Amazon EMR release version to use for the job run.

  • job_driver (dict) – Job configuration details, e.g. the Spark job parameters.

  • configuration_overrides (dict | None) – The configuration overrides for the job run, specifically either application configuration or monitoring configuration.

  • client_request_token (str | None) – The client idempotency token of the job run request. Use this if you want to specify a unique ID to prevent two jobs from getting started.

  • tags (dict | None) – The tags assigned to job runs.


The ID of the job run request.

Return type



Fetch the reason for a job failure (e.g. error message). Returns None or reason string.


job_id (str) – The ID of the job run request.


Fetch the status of submitted job run. Returns None or one of valid query states.


job_id (str) – The ID of the job run request.

poll_query_status(job_id, max_tries=None, poll_interval=30, max_polling_attempts=None)[source]

Poll the status of submitted job run until query state reaches final state. Returns one of the final states.

  • job_id (str) – The ID of the job run request.

  • max_tries (int | None) – Deprecated - Use max_polling_attempts instead

  • poll_interval (int) – Time (in seconds) to wait between calls to check query status on EMR

  • max_polling_attempts (int | None) – Number of times to poll for query state before function exits


Cancel the submitted job_run


job_id (str) – The ID of the job run to cancel.

Was this entry helpful?