airflow.providers.databricks.hooks.databricks_base

Databricks hook.

This hook enable the submitting and running of jobs to the Databricks platform. Internally the operators talk to the api/2.0/jobs/runs/submit endpoint.

Module Contents

Classes

BaseDatabricksHook

Base for interaction with Databricks.

BearerAuth

aiohttp only ships BasicAuth, for Bearer auth we need a subclass of BasicAuth.

Attributes

USER_AGENT_HEADER

AZURE_DEFAULT_AD_ENDPOINT

AZURE_TOKEN_SERVICE_URL

AZURE_METADATA_SERVICE_TOKEN_URL

AZURE_METADATA_SERVICE_INSTANCE_URL

TOKEN_REFRESH_LEAD_TIME

AZURE_MANAGEMENT_ENDPOINT

DEFAULT_DATABRICKS_SCOPE

airflow.providers.databricks.hooks.databricks_base.USER_AGENT_HEADER[source]
airflow.providers.databricks.hooks.databricks_base.AZURE_DEFAULT_AD_ENDPOINT = https://login.microsoftonline.com[source]
airflow.providers.databricks.hooks.databricks_base.AZURE_TOKEN_SERVICE_URL = {}/{}/oauth2/token[source]
airflow.providers.databricks.hooks.databricks_base.AZURE_METADATA_SERVICE_TOKEN_URL = http://169.254.169.254/metadata/identity/oauth2/token[source]
airflow.providers.databricks.hooks.databricks_base.AZURE_METADATA_SERVICE_INSTANCE_URL = http://169.254.169.254/metadata/instance[source]
airflow.providers.databricks.hooks.databricks_base.TOKEN_REFRESH_LEAD_TIME = 120[source]
airflow.providers.databricks.hooks.databricks_base.AZURE_MANAGEMENT_ENDPOINT = https://management.core.windows.net/[source]
airflow.providers.databricks.hooks.databricks_base.DEFAULT_DATABRICKS_SCOPE = 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d[source]
class airflow.providers.databricks.hooks.databricks_base.BaseDatabricksHook(databricks_conn_id=default_conn_name, timeout_seconds=180, retry_limit=3, retry_delay=1.0, retry_args=None)[source]

Bases: airflow.hooks.base.BaseHook

Base for interaction with Databricks.

Parameters

  • databricks_conn_id (str) -- Reference to the Databricks connection.

  • timeout_seconds (int) -- The amount of time in seconds the requests library will wait before timing-out.

  • retry_limit (int) -- The number of times to retry the connection in case of service outages.

  • retry_delay (float) -- The number of seconds to wait between retries (it might be a floating point number).

  • retry_args (Optional[Dict[Any, Any]]) -- An optional dictionary with arguments passed to tenacity.Retrying class.

conn_name_attr = databricks_conn_id[source]
default_conn_name = databricks_default[source]
conn_type = databricks[source]
extra_parameters = ['token', 'host', 'use_azure_managed_identity', 'azure_ad_endpoint', 'azure_resource_id',...[source]
databricks_conn(self)[source]
get_conn(self)[source]

Returns connection for the hook.

host(self)[source]
async __aenter__(self)[source]
async __aexit__(self, *err)[source]
class airflow.providers.databricks.hooks.databricks_base.BearerAuth(token)[source]

Bases: aiohttp.BasicAuth

aiohttp only ships BasicAuth, for Bearer auth we need a subclass of BasicAuth.

encode(self)[source]

Encode credentials.

Was this entry helpful?