airflow.providers.databricks.hooks.databricks_base

Databricks hook.

This hook enable the submitting and running of jobs to the Databricks platform. Internally the operators talk to the api/2.0/jobs/runs/submit endpoint.

Module Contents

Classes

BaseDatabricksHook

Base for interaction with Databricks.

BearerAuth

aiohttp only ships BasicAuth, for Bearer auth we need a subclass of BasicAuth.

Attributes

AZURE_DEFAULT_AD_ENDPOINT

AZURE_TOKEN_SERVICE_URL

AZURE_METADATA_SERVICE_TOKEN_URL

AZURE_METADATA_SERVICE_INSTANCE_URL

TOKEN_REFRESH_LEAD_TIME

AZURE_MANAGEMENT_ENDPOINT

DEFAULT_DATABRICKS_SCOPE

OIDC_TOKEN_SERVICE_URL

airflow.providers.databricks.hooks.databricks_base.AZURE_DEFAULT_AD_ENDPOINT = 'https://login.microsoftonline.com'[source]
airflow.providers.databricks.hooks.databricks_base.AZURE_TOKEN_SERVICE_URL = '{}/{}/oauth2/token'[source]
airflow.providers.databricks.hooks.databricks_base.AZURE_METADATA_SERVICE_TOKEN_URL = 'http://169.254.169.254/metadata/identity/oauth2/token'[source]
airflow.providers.databricks.hooks.databricks_base.AZURE_METADATA_SERVICE_INSTANCE_URL = 'http://169.254.169.254/metadata/instance'[source]
airflow.providers.databricks.hooks.databricks_base.TOKEN_REFRESH_LEAD_TIME = 120[source]
airflow.providers.databricks.hooks.databricks_base.AZURE_MANAGEMENT_ENDPOINT = 'https://management.core.windows.net/'[source]
airflow.providers.databricks.hooks.databricks_base.DEFAULT_DATABRICKS_SCOPE = '2ff814a6-3304-4ab8-85cb-cd0e6f879c1d'[source]
airflow.providers.databricks.hooks.databricks_base.OIDC_TOKEN_SERVICE_URL = '{}/oidc/v1/token'[source]
class airflow.providers.databricks.hooks.databricks_base.BaseDatabricksHook(databricks_conn_id=default_conn_name, timeout_seconds=180, retry_limit=3, retry_delay=1.0, retry_args=None, caller='Unknown')[source]

Bases: airflow.hooks.base.BaseHook

Base for interaction with Databricks.

Parameters

  • databricks_conn_id (str) – Reference to the Databricks connection.

  • timeout_seconds (int) – The amount of time in seconds the requests library will wait before timing-out.

  • retry_limit (int) – The number of times to retry the connection in case of service outages.

  • retry_delay (float) – The number of seconds to wait between retries (it might be a floating point number).

  • retry_args (dict[Any, Any] | None) – An optional dictionary with arguments passed to tenacity.Retrying class.

conn_name_attr: str = 'databricks_conn_id'[source]
default_conn_name = 'databricks_default'[source]
conn_type = 'databricks'[source]
extra_parameters = ['token', 'host', 'use_azure_managed_identity', 'azure_ad_endpoint', 'azure_resource_id',...[source]
databricks_conn()[source]
get_conn()[source]

Return connection for the hook.

user_agent_header()[source]
user_agent_value()[source]
host()[source]
async __aenter__()[source]
async __aexit__(*err)[source]
class airflow.providers.databricks.hooks.databricks_base.BearerAuth(token)[source]

Bases: aiohttp.BasicAuth

aiohttp only ships BasicAuth, for Bearer auth we need a subclass of BasicAuth.

encode()[source]

Encode credentials.

Was this entry helpful?