airflow.providers.microsoft.azure.operators.data_factory

Module Contents

class airflow.providers.microsoft.azure.operators.data_factory.AzureDataFactoryRunPipelineOperator(*, pipeline_name: str, azure_data_factory_conn_id: str = AzureDataFactoryHook.default_conn_name, wait_for_termination: bool = True, resource_group_name: Optional[str] = None, factory_name: Optional[str] = None, reference_pipeline_run_id: Optional[str] = None, is_recovery: Optional[bool] = None, start_activity_name: Optional[str] = None, start_from_failure: Optional[bool] = None, parameters: Optional[Dict[str, Any]] = None, timeout: Optional[int] = 60 * 60 * 24 * 7, check_interval: Optional[int] = 60, **kwargs)[source]

Bases: airflow.models.BaseOperator

Executes a data factory pipeline.

See also

For more information on how to use this operator, take a look at the guide: AzureDataFactoryRunPipelineOperator

Parameters
  • azure_data_factory_conn_id (str) -- The connection identifier for connecting to Azure Data Factory.

  • pipeline_name (str) -- The name of the pipeline to execute.

  • wait_for_termination (bool) -- Flag to wait on a pipeline run's termination. By default, this feature is enabled but could be disabled to perform an asynchronous wait for a long-running pipeline execution using the AzureDataFactoryPipelineRunSensor.

  • resource_group_name (str) -- The resource group name. If a value is not passed in to the operator, the AzureDataFactoryHook will attempt to use the resource group name provided in the corresponding connection.

  • factory_name (str) -- The data factory name. If a value is not passed in to the operator, the AzureDataFactoryHook will attempt to use the factory name name provided in the corresponding connection.

  • reference_pipeline_run_id (str) -- The pipeline run identifier. If this run ID is specified the parameters of the specified run will be used to create a new run.

  • is_recovery (bool) -- Recovery mode flag. If recovery mode is set to True, the specified referenced pipeline run and the new run will be grouped under the same groupId.

  • start_activity_name (str) -- In recovery mode, the rerun will start from this activity. If not specified, all activities will run.

  • start_from_failure (bool) -- In recovery mode, if set to true, the rerun will start from failed activities. The property will be used only if start_activity_name is not specified.

  • parameters (Dict[str, Any]) -- Parameters of the pipeline run. These parameters are referenced in a pipeline via @pipeline().parameters.parameterName and will be used only if the reference_pipeline_run_id is not specified.

  • timeout (int) -- Time in seconds to wait for a pipeline to reach a terminal status for non-asynchronous waits. Used only if wait_for_termination is True.

  • check_interval (int) -- Time in seconds to check on a pipeline run's status for non-asynchronous waits. Used only if wait_for_termination is True.

template_fields = ['azure_data_factory_conn_id', 'resource_group_name', 'factory_name', 'pipeline_name', 'reference_pipeline_run_id', 'parameters'][source]
template_fields_renderers[source]
ui_color = #0678d4[source]
execute(self, context: Dict)[source]
on_kill(self)[source]

Was this entry helpful?