airflow.providers.databricks.utils.databricks¶

Functions¶

`normalise_json_content`(content[, json_path])	Normalize content or all values of content if it is a dict to a string.
`extract_failed_task_errors`(hook, run_info, run_state)	Extract error information from failed tasks in a Databricks run (synchronous version).
`extract_failed_task_errors_async`(hook, run_info, run_state)	Extract error information from failed tasks in a Databricks run (asynchronous version).
`validate_trigger_event`(event)	Validate correctness of the event received from DatabricksExecutionTrigger.

Module Contents¶

airflow.providers.databricks.utils.databricks.normalise_json_content(content, json_path='json')[source]¶

Normalize content or all values of content if it is a dict to a string.

The function will throw if content contains non-string or non-numeric non-boolean types. The reason why we have this function is because the self.json field must be a dict with only string values. This is because render_template will fail for numerical values.

The only one exception is when we have boolean values, they can not be converted to string type because databricks does not understand ‘True’ or ‘False’ values.

airflow.providers.databricks.utils.databricks.extract_failed_task_errors(hook, run_info, run_state)[source]¶

Extract error information from failed tasks in a Databricks run (synchronous version).

Parameters:

hook (airflow.providers.databricks.hooks.databricks.DatabricksHook) – Databricks hook instance for making API calls
run_info (dict) – Run information dictionary from Databricks API
run_state (airflow.providers.databricks.hooks.databricks.RunState) – Run state object

Returns:

List of failed task information with task_key, run_id, and error

Return type:

list[dict[str, str | int]]

async airflow.providers.databricks.utils.databricks.extract_failed_task_errors_async(hook, run_info, run_state)[source]¶

Extract error information from failed tasks in a Databricks run (asynchronous version).

Parameters:

hook (airflow.providers.databricks.hooks.databricks.DatabricksHook) – Databricks hook instance for making API calls
run_info (dict) – Run information dictionary from Databricks API
run_state (airflow.providers.databricks.hooks.databricks.RunState) – Run state object

Returns:

List of failed task information with task_key, run_id, and error

Return type:

list[dict[str, str | int]]

airflow.providers.databricks.utils.databricks.validate_trigger_event(event)[source]¶

Validate correctness of the event received from DatabricksExecutionTrigger.

See: DatabricksExecutionTrigger.