Module Contents



Connection to Druid overlord for ingestion


Interact with Druid broker

class airflow.providers.apache.druid.hooks.druid.DruidHook(druid_ingest_conn_id='druid_ingest_default', timeout=1, max_ingestion_time=None)[source]

Bases: airflow.hooks.base.BaseHook

Connection to Druid overlord for ingestion

To connect to a Druid cluster that is secured with the druid-basic-security extension, add the username and password to the druid ingestion connection.

  • druid_ingest_conn_id (str) – The connection id to the Druid overlord machine which accepts index jobs

  • timeout (int) – The interval between polling the Druid job for the status of the ingestion job. Must be greater than or equal to 1

  • max_ingestion_time (Optional[int]) – The maximum ingestion time before assuming the job failed


Get Druid connection url


Return username and password from connections tab as requests.auth.HTTPBasicAuth object.

If these details have not been set then returns None.

submit_indexing_job(self, json_index_spec)[source]

Submit Druid ingestion job

class airflow.providers.apache.druid.hooks.druid.DruidDbApiHook(*args, schema=None, **kwargs)[source]

Bases: airflow.hooks.dbapi.DbApiHook

Interact with Druid broker

This hook is purely for users to query druid broker. For ingestion, please use druidHook.

conn_name_attr = druid_broker_conn_id[source]
default_conn_name = druid_broker_default[source]
conn_type = druid[source]
hook_name = Druid[source]
supports_autocommit = False[source]

Establish a connection to druid broker.


Get the connection uri for druid broker.

e.g: druid://localhost:8082/druid/v2/sql/

abstract set_autocommit(self, conn, autocommit)[source]

Sets the autocommit flag on the connection

abstract insert_rows(self, table, rows, target_fields=None, commit_every=1000, replace=False, **kwargs)[source]

A generic way to insert a set of tuples into a table, a new transaction is created every commit_every rows

  • table (str) – Name of the target table

  • rows (Iterable[Tuple[str]]) – The rows to insert into the table

  • target_fields (Optional[Iterable[str]]) – The names of the columns to fill in the table

  • commit_every (int) – The maximum number of rows to insert in one transaction. Set to 0 to insert all rows in one transaction.

  • replace (bool) – Whether to replace instead of insert

Was this entry helpful?