airflow.providers.microsoft.azure.hooks.data_factory¶

Spelling exceptions.

Attributes¶

`Credentials`
`AsyncCredentials`
`T`

Exceptions¶

AzureDataFactoryPipelineRunException

An exception that indicates a pipeline run failed to complete.

Classes¶

`AzureDataFactoryPipelineRunStatus`	Azure Data Factory pipeline operation statuses.
`AzureDataFactoryHook`	A hook to interact with Azure Data Factory.
`AzureDataFactoryAsyncHook`	An Async Hook that connects to Azure DataFactory to perform pipeline operations.

Functions¶

`provide_targeted_factory`(func)	Provide the targeted factory to the decorated function in case it isn't specified.
`get_field`(extras, field_name[, strict])	Get field from extra, first checking short name, then for backcompat we check for prefixed name.
`provide_targeted_factory_async`(func)	Provide the targeted factory to the async decorated function in case it isn't specified.

Module Contents¶

airflow.providers.microsoft.azure.hooks.data_factory.Credentials[source]¶

airflow.providers.microsoft.azure.hooks.data_factory.AsyncCredentials[source]¶

airflow.providers.microsoft.azure.hooks.data_factory.T[source]¶

airflow.providers.microsoft.azure.hooks.data_factory.provide_targeted_factory(func)[source]¶

Provide the targeted factory to the decorated function in case it isn’t specified.

If resource_group_name or factory_name is not provided it defaults to the value specified in the connection extras.

class airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryPipelineRunStatus[source]¶

Azure Data Factory pipeline operation statuses.

QUEUED = 'Queued'[source]¶

IN_PROGRESS = 'InProgress'[source]¶

SUCCEEDED = 'Succeeded'[source]¶

FAILED = 'Failed'[source]¶

CANCELING = 'Canceling'[source]¶

CANCELLED = 'Cancelled'[source]¶

TERMINAL_STATUSES[source]¶

INTERMEDIATE_STATES[source]¶

FAILURE_STATES[source]¶

exception airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryPipelineRunException[source]¶

Bases: airflow.exceptions.AirflowException

An exception that indicates a pipeline run failed to complete.

airflow.providers.microsoft.azure.hooks.data_factory.get_field(extras, field_name, strict=False)[source]¶

Get field from extra, first checking short name, then for backcompat we check for prefixed name.

class airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryHook(azure_data_factory_conn_id=default_conn_name)[source]¶

Bases: airflow.providers.common.compat.sdk.BaseHook

A hook to interact with Azure Data Factory.

Parameters:: azure_data_factory_conn_id (str) – The Azure Data Factory connection id.

conn_type: str = 'azure_data_factory'[source]¶

conn_name_attr: str = 'azure_data_factory_conn_id'[source]¶

default_conn_name: str = 'azure_data_factory_default'[source]¶

hook_name: str = 'Azure Data Factory'[source]¶

classmethod get_connection_form_widgets()[source]¶

Return connection widgets to add to connection form.

classmethod get_ui_field_behaviour()[source]¶

Return custom field behaviour.

conn_id = 'azure_data_factory_default'[source]¶

get_conn()[source]¶

Return connection for the hook.

refresh_conn()[source]¶

get_factory(resource_group_name, factory_name, **config)[source]¶

Get the factory.

Parameters:

resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Returns:

The factory.

Return type:

azure.mgmt.datafactory.models.Factory | None

update_factory(factory, resource_group_name, factory_name, if_match=None, **config)[source]¶

Update the factory.

Parameters:

factory (azure.mgmt.datafactory.models.Factory) – The factory resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
if_match (str | None) – ETag of the factory entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update. Default value is None.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the factory does not exist.

Returns:

The factory.

Return type:

azure.mgmt.datafactory.models.Factory

create_factory(factory, resource_group_name, factory_name, **config)[source]¶

Create the factory.

Parameters:

factory (azure.mgmt.datafactory.models.Factory) – The factory resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the factory already exists.

Returns:

The factory.

Return type:

azure.mgmt.datafactory.models.Factory

delete_factory(resource_group_name, factory_name, **config)[source]¶

Delete the factory.

Parameters:

resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

get_linked_service(linked_service_name, resource_group_name, factory_name, if_none_match=None, **config)[source]¶

Get the linked service.

Parameters:

linked_service_name (str) – The linked service name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
if_none_match (str | None) – ETag of the linked service entity. Should only be specified for get. If the ETag matches the existing entity tag, or if * was provided, then no content will be returned. Default value is None.
config (Any) – Extra parameters for the ADF client.

Returns:

The linked service.

Return type:

azure.mgmt.datafactory.models.LinkedServiceResource | None

update_linked_service(linked_service_name, linked_service, resource_group_name, factory_name, **config)[source]¶

Update the linked service.

Parameters:

linked_service_name (str) – The linked service name.
linked_service (azure.mgmt.datafactory.models.LinkedServiceResource) – The linked service resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the linked service does not exist.

Returns:

The linked service.

Return type:

azure.mgmt.datafactory.models.LinkedServiceResource

create_linked_service(linked_service_name, linked_service, resource_group_name, factory_name, **config)[source]¶

Create the linked service.

Parameters:

linked_service_name (str) – The linked service name.
linked_service (azure.mgmt.datafactory.models.LinkedServiceResource) – The linked service resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the linked service already exists.

Returns:

The linked service.

Return type:

azure.mgmt.datafactory.models.LinkedServiceResource

delete_linked_service(linked_service_name, resource_group_name, factory_name, **config)[source]¶

Delete the linked service.

Parameters:

linked_service_name (str) – The linked service name.
resource_group_name (str) – The linked service name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

get_dataset(dataset_name, resource_group_name, factory_name, **config)[source]¶

Get the dataset.

Parameters:

dataset_name (str) – The dataset name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Returns:

The dataset.

Return type:

azure.mgmt.datafactory.models.DatasetResource | None

update_dataset(dataset_name, dataset, resource_group_name, factory_name, **config)[source]¶

Update the dataset.

Parameters:

dataset_name (str) – The dataset name.
dataset (azure.mgmt.datafactory.models.DatasetResource) – The dataset resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the dataset does not exist.

Returns:

The dataset.

Return type:

azure.mgmt.datafactory.models.DatasetResource

create_dataset(dataset_name, dataset, resource_group_name, factory_name, **config)[source]¶

Create the dataset.

Parameters:

dataset_name (str) – The dataset name.
dataset (azure.mgmt.datafactory.models.DatasetResource) – The dataset resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the dataset already exists.

Returns:

The dataset.

Return type:

azure.mgmt.datafactory.models.DatasetResource

delete_dataset(dataset_name, resource_group_name, factory_name, **config)[source]¶

Delete the dataset.

Parameters:

dataset_name (str) – The dataset name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

get_dataflow(dataflow_name, resource_group_name, factory_name, if_none_match=None, **config)[source]¶

Get the dataflow.

Parameters:

dataflow_name (str) – The dataflow name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
if_none_match (str | None) – ETag of the data flow entity. Should only be specified for get. If the ETag matches the existing entity tag, or if * was provided, then no content will be returned. Default value is None.
config (Any) – Extra parameters for the ADF client.

Returns:

The DataFlowResource.

Return type:

azure.mgmt.datafactory.models.DataFlowResource

update_dataflow(dataflow_name, dataflow, resource_group_name, factory_name, if_match=None, **config)[source]¶

Update the dataflow.

Parameters:

dataflow_name (str) – The dataflow name.
dataflow (azure.mgmt.datafactory.models.DataFlowResource | IO) – The dataflow resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
if_match (str | None) – ETag of the data flow entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update. Default value is None.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the dataset does not exist.

Returns:

DataFlowResource.

Return type:

azure.mgmt.datafactory.models.DataFlowResource

create_dataflow(dataflow_name, dataflow, resource_group_name, factory_name, if_match=None, **config)[source]¶

Create the dataflow.

Parameters:

dataflow_name (str) – The dataflow name.
dataflow (azure.mgmt.datafactory.models.DataFlowResource) – The dataflow resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
if_match (str | None) – ETag of the factory entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update. Default value is None.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the dataset already exists.

Returns:

The dataset.

Return type:

azure.mgmt.datafactory.models.DataFlowResource

delete_dataflow(dataflow_name, resource_group_name, factory_name, **config)[source]¶

Delete the dataflow.

Parameters:

dataflow_name (str) – The dataflow name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

get_pipeline(pipeline_name, resource_group_name, factory_name, **config)[source]¶

Get the pipeline.

Parameters:

pipeline_name (str) – The pipeline name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Returns:

The pipeline.

Return type:

azure.mgmt.datafactory.models.PipelineResource | None

update_pipeline(pipeline_name, pipeline, resource_group_name, factory_name, **config)[source]¶

Update the pipeline.

Parameters:

pipeline_name (str) – The pipeline name.
pipeline (azure.mgmt.datafactory.models.PipelineResource) – The pipeline resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the pipeline does not exist.

Returns:

The pipeline.

Return type:

azure.mgmt.datafactory.models.PipelineResource

create_pipeline(pipeline_name, pipeline, resource_group_name, factory_name, **config)[source]¶

Create the pipeline.

Parameters:

pipeline_name (str) – The pipeline name.
pipeline (azure.mgmt.datafactory.models.PipelineResource) – The pipeline resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the pipeline already exists.

Returns:

The pipeline.

Return type:

azure.mgmt.datafactory.models.PipelineResource

delete_pipeline(pipeline_name, resource_group_name, factory_name, **config)[source]¶

Delete the pipeline.

Parameters:

pipeline_name (str) – The pipeline name.
resource_group_name (str) – The pipeline name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

run_pipeline(pipeline_name, resource_group_name, factory_name, **config)[source]¶

Run a pipeline.

Parameters:

pipeline_name (str) – The pipeline name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Returns:

The pipeline run.

Return type:

azure.mgmt.datafactory.models.CreateRunResponse

get_pipeline_run(run_id, resource_group_name, factory_name, **config)[source]¶

Get the pipeline run.

Parameters:

run_id (str) – The pipeline run identifier.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Returns:

The pipeline run.

Return type:

azure.mgmt.datafactory.models.PipelineRun

get_pipeline_run_status(run_id, resource_group_name, factory_name)[source]¶

Get a pipeline run’s current status.

Parameters:

run_id (str) – The pipeline run identifier.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.

Returns:

The status of the pipeline run.

Return type:

str

wait_for_pipeline_run_status(run_id, expected_statuses, resource_group_name, factory_name, check_interval=60, timeout=60 * 60 * 24 * 7)[source]¶

Wait for a pipeline run to match an expected status.

Parameters:

run_id (str) – The pipeline run identifier.
expected_statuses (str | set[str]) – The desired status(es) to check against a pipeline run’s current status.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
check_interval (int) – Time in seconds to check on a pipeline run’s status.
timeout (int) – Time in seconds to wait for a pipeline to reach a terminal status or the expected status.

Returns:

Boolean indicating if the pipeline run has reached the expected_status.

Return type:

bool

cancel_pipeline_run(run_id, resource_group_name, factory_name, **config)[source]¶

Cancel the pipeline run.

Parameters:

run_id (str) – The pipeline run identifier.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

get_trigger(trigger_name, resource_group_name, factory_name, **config)[source]¶

Get the trigger.

Parameters:

trigger_name (str) – The trigger name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Returns:

The trigger.

Return type:

azure.mgmt.datafactory.models.TriggerResource | None

update_trigger(trigger_name, trigger, resource_group_name, factory_name, if_match=None, **config)[source]¶

Update the trigger.

Parameters:

trigger_name (str) – The trigger name.
trigger (azure.mgmt.datafactory.models.TriggerResource) – The trigger resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
if_match (str | None) – ETag of the trigger entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update. Default value is None.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the trigger does not exist.

Returns:

The trigger.

Return type:

azure.mgmt.datafactory.models.TriggerResource

create_trigger(trigger_name, trigger, resource_group_name, factory_name, **config)[source]¶

Create the trigger.

Parameters:

trigger_name (str) – The trigger name.
trigger (azure.mgmt.datafactory.models.TriggerResource) – The trigger resource definition.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Raises:

AirflowException – If the trigger already exists.

Returns:

The trigger.

Return type:

azure.mgmt.datafactory.models.TriggerResource

delete_trigger(trigger_name, resource_group_name, factory_name, **config)[source]¶

Delete the trigger.

Parameters:

trigger_name (str) – The trigger name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

start_trigger(trigger_name, resource_group_name, factory_name, **config)[source]¶

Start the trigger.

Parameters:

trigger_name (str) – The trigger name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Returns:

An Azure operation poller.

Return type:

azure.core.polling.LROPoller

stop_trigger(trigger_name, resource_group_name, factory_name, **config)[source]¶

Stop the trigger.

Parameters:

trigger_name (str) – The trigger name.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

Returns:

An Azure operation poller.

Return type:

azure.core.polling.LROPoller

rerun_trigger(trigger_name, run_id, resource_group_name, factory_name, **config)[source]¶

Rerun the trigger.

Parameters:

trigger_name (str) – The trigger name.
run_id (str) – The trigger run identifier.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

cancel_trigger(trigger_name, run_id, resource_group_name, factory_name, **config)[source]¶

Cancel the trigger.

Parameters:

trigger_name (str) – The trigger name.
run_id (str) – The trigger run identifier.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

test_connection()[source]¶

Test a configured Azure Data Factory connection.

airflow.providers.microsoft.azure.hooks.data_factory.provide_targeted_factory_async(func)[source]¶

Provide the targeted factory to the async decorated function in case it isn’t specified.

If resource_group_name or factory_name is not provided it defaults to the value specified in the connection extras.

class airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryAsyncHook(azure_data_factory_conn_id=default_conn_name)[source]¶

Bases: AzureDataFactoryHook

An Async Hook that connects to Azure DataFactory to perform pipeline operations.

Parameters:: azure_data_factory_conn_id (str) – The Azure Data Factory connection id.

default_conn_name: str = 'azure_data_factory_default'[source]¶

conn_id = 'azure_data_factory_default'[source]¶

async get_async_conn()[source]¶

Get async connection and connect to azure data factory.

async refresh_conn()[source]¶

async get_pipeline_run(run_id, resource_group_name, factory_name, **config)[source]¶

Connect to Azure Data Factory asynchronously to get the pipeline run details by run id.

Parameters:

run_id (str) – The pipeline run identifier.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.

async get_adf_pipeline_run_status(run_id, resource_group_name, factory_name)[source]¶

Connect to Azure Data Factory asynchronously and get the pipeline status by run_id.

Parameters:

run_id (str) – The pipeline run identifier.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.

async cancel_pipeline_run(run_id, resource_group_name, factory_name, **config)[source]¶

Cancel the pipeline run.

Parameters:

run_id (str) – The pipeline run identifier.
resource_group_name (str) – The resource group name.
factory_name (str) – The factory name.
config (Any) – Extra parameters for the ADF client.