airflow.providers.amazon.aws.operators.emr_add_steps

Module Contents

class airflow.providers.amazon.aws.operators.emr_add_steps.EmrAddStepsOperator(*, job_flow_id: Optional[str] = None, job_flow_name: Optional[str] = None, cluster_states: Optional[List[str]] = None, aws_conn_id: str = 'aws_default', steps: Optional[Union[List[dict], str]] = None, **kwargs)[source]

Bases: airflow.models.BaseOperator

An operator that adds steps to an existing EMR job_flow.

Parameters

  • job_flow_id (Optional[str]) -- id of the JobFlow to add steps to. (templated)

  • job_flow_name (Optional[str]) -- name of the JobFlow to add steps to. Use as an alternative to passing job_flow_id. will search for id of JobFlow with matching name in one of the states in param cluster_states. Exactly one cluster like this should exist or will fail. (templated)

  • cluster_states (list) -- Acceptable cluster states when searching for JobFlow id by job_flow_name. (templated)

  • aws_conn_id (str) -- aws connection to uses

  • steps (list|str) -- boto3 style steps or reference to a steps file (must be '.json') to be added to the jobflow. (templated)

  • do_xcom_push (bool) -- if True, job_flow_id is pushed to XCom with key job_flow_id.

template_fields = ['job_flow_id', 'job_flow_name', 'cluster_states', 'steps'][source]
template_ext = ['.json'][source]
template_fields_renderers[source]
ui_color = #f9c915[source]
execute(self, context: Dict[str, Any])[source]

Was this entry helpful?