airflow.providers.databricks.plugins.databricks_workflow¶

Attributes¶

`REPAIR_WAIT_ATTEMPTS`
`REPAIR_WAIT_DELAY`
`repair_databricks_view`
`repair_databricks_package`

Classes¶

`WorkflowJobRunLink`	Constructs a link to monitor a Databricks Job Run.
`WorkflowJobRepairAllFailedLink`	Constructs a link to send a request to repair all failed tasks in the Databricks workflow.
`WorkflowJobRepairSingleTaskLink`	Construct a link to send a repair request for a single databricks task.
`RepairDatabricksTasks`	Repair databricks tasks from Airflow.
`DatabricksWorkflowPlugin`	Databricks Workflows plugin for Airflow.

Functions¶

`get_auth_decorator`()
`get_databricks_task_ids`(group_id, task_map, log)	Return a list of all Databricks task IDs for a dictionary of Airflow tasks.
`get_launch_task_id`(task_group)	Retrieve the launch task ID from the current task group or a parent task group, recursively.
`get_task_instance`(operator, dttm[, session])
`get_xcom_result`(ti_key, key)

Module Contents¶

airflow.providers.databricks.plugins.databricks_workflow.REPAIR_WAIT_ATTEMPTS[source]¶

airflow.providers.databricks.plugins.databricks_workflow.REPAIR_WAIT_DELAY[source]¶

airflow.providers.databricks.plugins.databricks_workflow.get_auth_decorator()[source]¶

airflow.providers.databricks.plugins.databricks_workflow.get_databricks_task_ids(group_id, task_map, log)[source]¶

Return a list of all Databricks task IDs for a dictionary of Airflow tasks.

Parameters:

group_id (str) – The task group ID.
task_map (dict[str, airflow.providers.databricks.operators.databricks.DatabricksTaskBaseOperator]) – A dictionary mapping task IDs to BaseOperator instances.
log (logging.Logger) – The logger to use for logging.

Returns:

A list of Databricks task IDs for the given task group.

Return type:

list[str]

airflow.providers.databricks.plugins.databricks_workflow.get_launch_task_id(task_group)[source]¶

Retrieve the launch task ID from the current task group or a parent task group, recursively.

Parameters:: task_group (airflow.utils.task_group.TaskGroup) – Task Group to be inspected
Returns:: launch Task ID
Return type:: str

airflow.providers.databricks.plugins.databricks_workflow.get_task_instance(operator, dttm, session=NEW_SESSION)[source]¶

airflow.providers.databricks.plugins.databricks_workflow.get_xcom_result(ti_key, key)[source]¶

class airflow.providers.databricks.plugins.databricks_workflow.WorkflowJobRunLink(context=None)[source]¶

Bases: airflow.sdk.BaseOperatorLink, airflow.utils.log.logging_mixin.LoggingMixin

Constructs a link to monitor a Databricks Job Run.

name = 'See Databricks Job Run'[source]¶: Name of the link. This will be the button name on the task UI.

get_link(operator, dttm=None, *, ti_key=None)[source]¶

Link to external system.

Parameters:

operator (airflow.models.BaseOperator) – The Airflow operator object this link is associated to.
ti_key (airflow.models.taskinstance.TaskInstanceKey | None) – TaskInstance ID to return link for.

Returns:

link to external system

Return type:

str

class airflow.providers.databricks.plugins.databricks_workflow.WorkflowJobRepairAllFailedLink(context=None)[source]¶

Bases: airflow.sdk.BaseOperatorLink, airflow.utils.log.logging_mixin.LoggingMixin

Constructs a link to send a request to repair all failed tasks in the Databricks workflow.

name = 'Repair All Failed Tasks'[source]¶: Name of the link. This will be the button name on the task UI.

get_link(operator, dttm=None, *, ti_key=None)[source]¶

Link to external system.

Parameters:

operator – The Airflow operator object this link is associated to.
ti_key (airflow.models.taskinstance.TaskInstanceKey | None) – TaskInstance ID to return link for.

Returns:

link to external system

Return type:

str

classmethod get_task_group_children(task_group)[source]¶

Given a TaskGroup, return children which are Tasks, inspecting recursively any TaskGroups within.

Parameters:: task_group (airflow.utils.task_group.TaskGroup) – An Airflow TaskGroup
Returns:: Dictionary that contains Task IDs as keys and Tasks as values.
Return type:: dict[str, airflow.models.BaseOperator]

get_tasks_to_run(ti_key, operator, log)[source]¶

class airflow.providers.databricks.plugins.databricks_workflow.WorkflowJobRepairSingleTaskLink(context=None)[source]¶

Bases: airflow.sdk.BaseOperatorLink, airflow.utils.log.logging_mixin.LoggingMixin

Construct a link to send a repair request for a single databricks task.

name = 'Repair a single task'[source]¶: Name of the link. This will be the button name on the task UI.

get_link(operator, dttm=None, *, ti_key=None)[source]¶

Link to external system.

Parameters:

operator – The Airflow operator object this link is associated to.
ti_key (airflow.models.taskinstance.TaskInstanceKey | None) – TaskInstance ID to return link for.

Returns:

link to external system

Return type:

str

class airflow.providers.databricks.plugins.databricks_workflow.RepairDatabricksTasks[source]¶

Bases: flask_appbuilder.BaseView, airflow.utils.log.logging_mixin.LoggingMixin

Repair databricks tasks from Airflow.

default_view = 'repair'[source]¶: the default view for this BaseView, to be used with url_for (method name)

repair(dag_id, run_id)[source]¶

airflow.providers.databricks.plugins.databricks_workflow.repair_databricks_view[source]¶

airflow.providers.databricks.plugins.databricks_workflow.repair_databricks_package[source]¶

class airflow.providers.databricks.plugins.databricks_workflow.DatabricksWorkflowPlugin[source]¶

Bases: airflow.plugins_manager.AirflowPlugin

Databricks Workflows plugin for Airflow.