airflow.contrib.operators.emr_add_steps_operator

Module Contents

class airflow.contrib.operators.emr_add_steps_operator.EmrAddStepsOperator(job_flow_id=None, job_flow_name=None, cluster_states=None, aws_conn_id='aws_default', steps=None, *args, **kwargs)[source]

Bases: airflow.models.BaseOperator

An operator that adds steps to an existing EMR job_flow.

Parameters

  • job_flow_id (str) – id of the JobFlow to add steps to. (templated)

  • job_flow_name (str) – name of the JobFlow to add steps to. Use as an alternative to passing job_flow_id. will search for id of JobFlow with matching name in one of the states in param cluster_states. Exactly one cluster like this should exist or will fail. (templated)

  • cluster_states (list) – Acceptable cluster states when searching for JobFlow id by job_flow_name. (templated)

  • aws_conn_id (str) – aws connection to uses

  • steps (list) – boto3 style steps to be added to the jobflow. (templated)

  • do_xcom_push (bool) – if True, job_flow_id is pushed to XCom with key job_flow_id.

template_fields = ['job_flow_id', 'job_flow_name', 'cluster_states', 'steps'][source]
template_ext = [][source]
ui_color = #f9c915[source]
execute(self, context)[source]

Was this entry helpful?