airflow.providers.microsoft.azure.hooks.synapse

Module Contents

Classes

AzureSynapseSparkBatchRunStatus

Azure Synapse Spark Job operation statuses.

AzureSynapseHook

A hook to interact with Azure Synapse.

AzureSynapsePipelineRunStatus

Azure Synapse pipeline operation statuses.

BaseAzureSynapseHook

A base hook class to create session and connection to Azure Synapse using connection id.

AzureSynapsePipelineHook

A hook to interact with Azure Synapse Pipeline.

Attributes

Credentials

airflow.providers.microsoft.azure.hooks.synapse.Credentials[source]
class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapseSparkBatchRunStatus[source]

Azure Synapse Spark Job operation statuses.

NOT_STARTED = 'not_started'[source]
STARTING = 'starting'[source]
RUNNING = 'running'[source]
IDLE = 'idle'[source]
BUSY = 'busy'[source]
SHUTTING_DOWN = 'shutting_down'[source]
ERROR = 'error'[source]
DEAD = 'dead'[source]
KILLED = 'killed'[source]
SUCCESS = 'success'[source]
TERMINAL_STATUSES[source]
class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapseHook(azure_synapse_conn_id=default_conn_name, spark_pool='')[source]

Bases: airflow.hooks.base.BaseHook

A hook to interact with Azure Synapse.

Parameters
conn_type: str = 'azure_synapse'[source]
conn_name_attr: str = 'azure_synapse_conn_id'[source]
default_conn_name: str = 'azure_synapse_default'[source]
hook_name: str = 'Azure Synapse'[source]
classmethod get_connection_form_widgets()[source]

Return connection widgets to add to connection form.

classmethod get_ui_field_behaviour()[source]

Return custom field behaviour.

get_conn()[source]

Return connection for the hook.

run_spark_job(payload)[source]

Run a job in an Apache Spark pool.

Parameters

payload (azure.synapse.spark.models.SparkBatchJobOptions) – Livy compatible payload which represents the spark job that a user wants to submit.

get_job_run_status()[source]

Get the job run status.

wait_for_job_run_status(job_id, expected_statuses, check_interval=60, timeout=60 * 60 * 24 * 7)[source]

Wait for a job run to match an expected status.

Parameters

  • job_id (int | None) – The job run identifier.

  • expected_statuses (str | set[str]) – The desired status(es) to check against a job run’s current status.

  • check_interval (int) – Time in seconds to check on a job run’s status.

  • timeout (int) – Time in seconds to wait for a job to reach a terminal status or the expected status.

cancel_job_run(job_id)[source]

Cancel the spark job run.

Parameters

job_id (int) – The synapse spark job identifier.

class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapsePipelineRunStatus[source]

Azure Synapse pipeline operation statuses.

QUEUED = 'Queued'[source]
IN_PROGRESS = 'InProgress'[source]
SUCCEEDED = 'Succeeded'[source]
FAILED = 'Failed'[source]
CANCELING = 'Canceling'[source]
CANCELLED = 'Cancelled'[source]
TERMINAL_STATUSES[source]
INTERMEDIATE_STATES[source]
FAILURE_STATES[source]
exception airflow.providers.microsoft.azure.hooks.synapse.AzureSynapsePipelineRunException[source]

Bases: airflow.exceptions.AirflowException

An exception that indicates a pipeline run failed to complete.

class airflow.providers.microsoft.azure.hooks.synapse.BaseAzureSynapseHook(azure_synapse_conn_id=default_conn_name, **kwargs)[source]

Bases: airflow.hooks.base.BaseHook

A base hook class to create session and connection to Azure Synapse using connection id.

Parameters

azure_synapse_conn_id (str) – The Azure Synapse connection id.

conn_type: str = 'azure_synapse'[source]
conn_name_attr: str = 'azure_synapse_conn_id'[source]
default_conn_name: str = 'azure_synapse_default'[source]
hook_name: str = 'Azure Synapse'[source]
classmethod get_connection_form_widgets()[source]

Return connection widgets to add to connection form.

classmethod get_ui_field_behaviour()[source]

Return custom field behaviour.

class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapsePipelineHook(azure_synapse_workspace_dev_endpoint, azure_synapse_conn_id=default_conn_name, **kwargs)[source]

Bases: BaseAzureSynapseHook

A hook to interact with Azure Synapse Pipeline.

Parameters
default_conn_name: str = 'azure_synapse_connection'[source]
get_conn()[source]

Return connection for the hook.

run_pipeline(pipeline_name, **config)[source]

Run a Synapse pipeline.

Parameters

  • pipeline_name (str) – The pipeline name.

  • config (Any) – Extra parameters for the Synapse Artifact Client.

Returns

The pipeline run Id.

Return type

azure.synapse.artifacts.models.CreateRunResponse

get_pipeline_run(run_id)[source]

Get the pipeline run.

Parameters

run_id (str) – The pipeline run identifier.

Returns

The pipeline run.

Return type

azure.synapse.artifacts.models.PipelineRun

get_pipeline_run_status(run_id)[source]

Get a pipeline run’s current status.

Parameters

run_id (str) – The pipeline run identifier.

Returns

The status of the pipeline run.

Return type

str

refresh_conn()[source]
wait_for_pipeline_run_status(run_id, expected_statuses, check_interval=60, timeout=60 * 60 * 24 * 7)[source]

Wait for a pipeline run to match an expected status.

Parameters

  • run_id (str) – The pipeline run identifier.

  • expected_statuses (str | set[str]) – The desired status(es) to check against a pipeline run’s current status.

  • check_interval (int) – Time in seconds to check on a pipeline run’s status.

  • timeout (int) – Time in seconds to wait for a pipeline to reach a terminal status or the expected status.

Returns

Boolean indicating if the pipeline run has reached the expected_status.

Return type

bool

cancel_run_pipeline(run_id)[source]

Cancel the pipeline run.

Parameters

run_id (str) – The pipeline run identifier.

Was this entry helpful?