airflow.providers.databricks.hooks.databricks_sql

Attributes

LIST_SQL_ENDPOINTS_ENDPOINT

T

Classes

DatabricksSqlHook

Hook to interact with Databricks SQL.

Functions

create_timeout_thread(cur, execution_timeout)

Module Contents

airflow.providers.databricks.hooks.databricks_sql.LIST_SQL_ENDPOINTS_ENDPOINT = ('GET', 'api/2.0/sql/endpoints')[source]
airflow.providers.databricks.hooks.databricks_sql.T[source]
airflow.providers.databricks.hooks.databricks_sql.create_timeout_thread(cur, execution_timeout)[source]
class airflow.providers.databricks.hooks.databricks_sql.DatabricksSqlHook(databricks_conn_id=BaseDatabricksHook.default_conn_name, http_path=None, sql_endpoint_name=None, session_configuration=None, http_headers=None, catalog=None, schema=None, caller='DatabricksSqlHook', **kwargs)[source]

Bases: airflow.providers.databricks.hooks.databricks_base.BaseDatabricksHook, airflow.providers.common.sql.hooks.sql.DbApiHook

Hook to interact with Databricks SQL.

Parameters:

  • databricks_conn_id (str) – Reference to the Databricks connection.

  • http_path (str | None) – Optional string specifying HTTP path of Databricks SQL Endpoint or cluster. If not specified, it should be either specified in the Databricks connection’s extra parameters, or sql_endpoint_name must be specified.

  • sql_endpoint_name (str | None) – Optional name of Databricks SQL Endpoint. If not specified, http_path must be provided as described above.

  • session_configuration (dict[str, str] | None) – An optional dictionary of Spark session parameters. Defaults to None. If not specified, it could be specified in the Databricks connection’s extra parameters.

  • http_headers (list[tuple[str, str]] | None) – An optional list of (k, v) pairs that will be set as HTTP headers on every request

  • catalog (str | None) – An optional initial catalog to use. Requires DBR version 9.0+

  • schema (str | None) – An optional initial schema to use. Requires DBR version 9.0+

  • kwargs – Additional parameters internal to Databricks SQL Connector parameters

hook_name = 'Databricks SQL'[source]
supports_autocommit = True[source]
session_config = None[source]
http_headers = None[source]
catalog = None[source]
schema = None[source]
additional_params[source]
get_conn()[source]

Return a Databricks SQL connection object.

run(sql: str | collections.abc.Iterable[str], autocommit: bool = ..., parameters: collections.abc.Iterable | collections.abc.Mapping[str, Any] | None = ..., handler: None = ..., split_statements: bool = ..., return_last: bool = ..., execution_timeout: datetime.timedelta | None = None) → None[source]
run(sql: str | collections.abc.Iterable[str], autocommit: bool = ..., parameters: collections.abc.Iterable | collections.abc.Mapping[str, Any] | None = ..., handler: Callable[[Any], T] = ..., split_statements: bool = ..., return_last: bool = ..., execution_timeout: datetime.timedelta | None = None) → tuple | list[tuple] | list[list[tuple] | tuple] | None

Run a command or a list of commands.

Pass a list of SQL statements to the SQL parameter to get them to execute sequentially.

Parameters:

  • sql – the sql statement to be executed (str) or a list of sql statements to execute

  • autocommit – What to set the connection’s autocommit setting to before executing the query. Note that currently there is no commit functionality in Databricks SQL so this flag has no effect.

  • parameters – The parameters to render the SQL query with.

  • handler – The result handler which is called with the result of each statement.

  • split_statements – Whether to split a single SQL string into statements and run separately

  • return_last – Whether to return result for only last statement or for all after split

  • execution_timeout – max time allowed for the execution of this task instance, if it goes beyond it will raise and fail.

Returns:

return only result of the LAST SQL expression if handler was provided unless return_last is set to False.

abstract bulk_dump(table, tmp_file)[source]

Dump a database table into a tab-delimited file.

Parameters:

  • table – The name of the source table

  • tmp_file – The path of the target file

abstract bulk_load(table, tmp_file)[source]

Load a tab-delimited file into a database table.

Parameters:

  • table – The name of the target table

  • tmp_file – The path of the file to load into the table

Was this entry helpful?