airflow.providers.databricks.hooks.databricks_sql

Module Contents

Classes

DatabricksSqlHook

Hook to interact with Databricks SQL.

Attributes

LIST_SQL_ENDPOINTS_ENDPOINT

airflow.providers.databricks.hooks.databricks_sql.LIST_SQL_ENDPOINTS_ENDPOINT = ['GET', 'api/2.0/sql/endpoints'][source]
class airflow.providers.databricks.hooks.databricks_sql.DatabricksSqlHook(databricks_conn_id=BaseDatabricksHook.default_conn_name, http_path=None, sql_endpoint_name=None, session_configuration=None, http_headers=None, catalog=None, schema=None, caller='DatabricksSqlHook', **kwargs)[source]

Bases: airflow.providers.databricks.hooks.databricks_base.BaseDatabricksHook, airflow.providers.common.sql.hooks.sql.DbApiHook

Hook to interact with Databricks SQL.

Parameters

  • databricks_conn_id (str) – Reference to the Databricks connection.

  • http_path (Optional[str]) – Optional string specifying HTTP path of Databricks SQL Endpoint or cluster. If not specified, it should be either specified in the Databricks connection’s extra parameters, or sql_endpoint_name must be specified.

  • sql_endpoint_name (Optional[str]) – Optional name of Databricks SQL Endpoint. If not specified, http_path must be provided as described above.

  • session_configuration (Optional[Dict[str, str]]) – An optional dictionary of Spark session parameters. Defaults to None. If not specified, it could be specified in the Databricks connection’s extra parameters.

  • http_headers (Optional[List[Tuple[str, str]]]) – An optional list of (k, v) pairs that will be set as HTTP headers on every request

  • catalog (Optional[str]) – An optional initial catalog to use. Requires DBR version 9.0+

  • schema (Optional[str]) – An optional initial schema to use. Requires DBR version 9.0+

  • kwargs – Additional parameters internal to Databricks SQL Connector parameters

hook_name = Databricks SQL[source]
get_conn()[source]

Returns a Databricks SQL connection object

run(sql, autocommit=False, parameters=None, handler=None, split_statements=True, return_last=True)[source]

Runs a command or a list of commands. Pass a list of sql statements to the sql parameter to get them to execute sequentially

Parameters

  • sql (Union[str, Iterable[str]]) – the sql statement to be executed (str) or a list of sql statements to execute

  • autocommit (bool) – What to set the connection’s autocommit setting to before executing the query.

  • parameters (Optional[Union[Iterable, Mapping]]) – The parameters to render the SQL query with.

  • handler (Optional[Callable]) – The result handler which is called with the result of each statement.

  • split_statements (bool) – Whether to split a single SQL string into statements and run separately

  • return_last (bool) – Whether to return result for only last statement or for all after split

Returns

return only result of the LAST SQL expression if handler was provided.

Return type

Optional[Union[Tuple[str, Any], List[Tuple[str, Any]]]]

test_connection()[source]

Test the Databricks SQL connection by running a simple query.

abstract bulk_dump(table, tmp_file)[source]

Dumps a database table into a tab-delimited file

Parameters

  • table – The name of the source table

  • tmp_file – The path of the target file

abstract bulk_load(table, tmp_file)[source]

Loads a tab-delimited file into a database table

Parameters

  • table – The name of the target table

  • tmp_file – The path of the file to load into the table

Was this entry helpful?