Azure Data Factory Operators

Azure Data Factory is Azure’s cloud ETL service for scale-out serverless data integration and data transformation. It offers a code-free UI for intuitive authoring and single-pane-of-glass monitoring and management.

AzureDataFactoryRunPipelineOperator

Use the AzureDataFactoryRunPipelineOperator to execute a pipeline within a data factory. By default, the operator will periodically check on the status of the executed pipeline to terminate with a “Succeeded” status. This functionality can be disabled for an asynchronous wait – typically with the AzureDataFactoryPipelineRunSensor – by setting wait_for_termination to False.

Below is an example of using this operator to execute an Azure Data Factory pipeline.

tests/system/providers/microsoft/azure/example_adf_run_pipeline.py[source]

    run_pipeline1 = AzureDataFactoryRunPipelineOperator(
        task_id="run_pipeline1",
        pipeline_name="pipeline1",
        parameters={"myParam": "value"},
    )

Here is a different example of using this operator to execute a pipeline but coupled with the AzureDataFactoryPipelineRunSensor to perform an asynchronous wait.

tests/system/providers/microsoft/azure/example_adf_run_pipeline.py[source]

    run_pipeline2 = AzureDataFactoryRunPipelineOperator(
        task_id="run_pipeline2",
        pipeline_name="pipeline2",
        wait_for_termination=False,
    )

    pipeline_run_sensor = AzureDataFactoryPipelineRunStatusSensor(
        task_id="pipeline_run_sensor",
        run_id=cast(str, XComArg(run_pipeline2, key="run_id")),
    )

Reference

For further information, please refer to the Microsoft documentation:

Was this entry helpful?