airflow.providers.databricks.plugins.databricks_workflow¶

Attributes¶

`REPAIR_WAIT_ATTEMPTS`
`REPAIR_WAIT_DELAY`

Classes¶

`WorkflowJobRunLink`	Constructs a link to monitor a Databricks Job Run.
`WorkflowJobRepairAllFailedLink`	Constructs a link to send a request to repair all failed tasks in the Databricks workflow.
`WorkflowJobRepairSingleTaskLink`	Construct a link to send a repair request for a single databricks task.
`DatabricksWorkflowPlugin`	Databricks Workflows plugin for Airflow.

Functions¶

`get_databricks_task_ids`(group_id, task_map, log)	Return a list of all Databricks task IDs for a dictionary of Airflow tasks.
`get_auth_decorator`()
`get_launch_task_id`(task_group)	Retrieve the launch task ID from the current task group or a parent task group, recursively.
`get_xcom_result`(ti_key, key)
`store_databricks_job_run_link`(context, metadata, logger)	Store the Databricks job run link in XCom during task execution.

Module Contents¶

airflow.providers.databricks.plugins.databricks_workflow.REPAIR_WAIT_ATTEMPTS[source]¶

airflow.providers.databricks.plugins.databricks_workflow.REPAIR_WAIT_DELAY[source]¶

airflow.providers.databricks.plugins.databricks_workflow.get_databricks_task_ids(group_id, task_map, log)[source]¶

Return a list of all Databricks task IDs for a dictionary of Airflow tasks.

Parameters:

group_id (str) – The task group ID.
task_map (dict[str, airflow.providers.databricks.operators.databricks.DatabricksTaskBaseOperator]) – A dictionary mapping task IDs to BaseOperator instances.
log (airflow.sdk.types.Logger) – The logger to use for logging.

Returns:

A list of Databricks task IDs for the given task group.

Return type:

list[str]

airflow.providers.databricks.plugins.databricks_workflow.get_auth_decorator()[source]¶

airflow.providers.databricks.plugins.databricks_workflow.get_launch_task_id(task_group)[source]¶

Retrieve the launch task ID from the current task group or a parent task group, recursively.

Parameters:: task_group (airflow.providers.common.compat.sdk.TaskGroup) – Task Group to be inspected
Returns:: launch Task ID
Return type:: str

airflow.providers.databricks.plugins.databricks_workflow.get_xcom_result(ti_key, key)[source]¶

class airflow.providers.databricks.plugins.databricks_workflow.WorkflowJobRunLink(context=None)[source]¶

Bases: airflow.providers.common.compat.sdk.BaseOperatorLink, airflow.utils.log.logging_mixin.LoggingMixin

Constructs a link to monitor a Databricks Job Run.

name = 'See Databricks Job Run'[source]¶: Name of the link. This will be the button name on the task UI.

property xcom_key: str[source]¶

XCom key where the link is stored during task execution.

get_link(operator, dttm=None, *, ti_key=None)[source]¶

Link to external system.

Parameters:

operator (airflow.models.BaseOperator) – The Airflow operator object this link is associated to.
ti_key (airflow.models.taskinstance.TaskInstanceKey | None) – TaskInstance ID to return link for.

Returns:

link to external system

Return type:

str

airflow.providers.databricks.plugins.databricks_workflow.store_databricks_job_run_link(context, metadata, logger)[source]¶

Store the Databricks job run link in XCom during task execution.

This should be called by Databricks operators during their execution.

class airflow.providers.databricks.plugins.databricks_workflow.WorkflowJobRepairAllFailedLink(context=None)[source]¶

Bases: airflow.providers.common.compat.sdk.BaseOperatorLink, airflow.utils.log.logging_mixin.LoggingMixin

Constructs a link to send a request to repair all failed tasks in the Databricks workflow.

name = 'Repair All Failed Tasks'[source]¶: Name of the link. This will be the button name on the task UI.

get_link(operator, dttm=None, *, ti_key=None)[source]¶

Link to external system.

Parameters:

operator – The Airflow operator object this link is associated to.
ti_key (airflow.models.taskinstance.TaskInstanceKey | None) – TaskInstance ID to return link for.

Returns:

link to external system

Return type:

str

classmethod get_task_group_children(task_group)[source]¶

Given a TaskGroup, return children which are Tasks, inspecting recursively any TaskGroups within.

Parameters:: task_group (airflow.providers.common.compat.sdk.TaskGroup) – An Airflow TaskGroup
Returns:: Dictionary that contains Task IDs as keys and Tasks as values.
Return type:: dict[str, airflow.models.BaseOperator]

get_tasks_to_run(ti_key, operator, log)[source]¶

class airflow.providers.databricks.plugins.databricks_workflow.WorkflowJobRepairSingleTaskLink(context=None)[source]¶

Bases: airflow.providers.common.compat.sdk.BaseOperatorLink, airflow.utils.log.logging_mixin.LoggingMixin

Construct a link to send a repair request for a single databricks task.

name = 'Repair a single task'[source]¶: Name of the link. This will be the button name on the task UI.

get_link(operator, dttm=None, *, ti_key=None)[source]¶

Link to external system.

Parameters:

operator – The Airflow operator object this link is associated to.
ti_key (airflow.models.taskinstance.TaskInstanceKey | None) – TaskInstance ID to return link for.

Returns:

link to external system

Return type:

str

class airflow.providers.databricks.plugins.databricks_workflow.DatabricksWorkflowPlugin[source]¶

Bases: airflow.plugins_manager.AirflowPlugin

Databricks Workflows plugin for Airflow.