airflow.providers.duckdb.hooks.duckdb

Attributes

IN_MEMORY_DATABASE

Classes

DuckDBHook

Interact with an in-process DuckDB database.

Module Contents

airflow.providers.duckdb.hooks.duckdb.IN_MEMORY_DATABASE = ':memory:'[source]
class airflow.providers.duckdb.hooks.duckdb.DuckDBHook(*args, duckdb_conn_id=None, database=None, extensions=None, extension_directory=None, autoinstall_extensions=None, autoload_extensions=None, allow_community_extensions=None, memory_limit=None, threads=None, temp_directory=None, read_only=None, settings=None, **kwargs)[source]

Bases: airflow.providers.common.sql.hooks.sql.DbApiHook

Interact with an in-process DuckDB database.

The hook opens a DuckDB database — in memory, backed by a local file, or hosted by MotherDuck — applies resource limits, and loads the requested extensions, so a Dag author only supplies SQL.

The Airflow connection is optional. With no duckdb_default connection configured the hook opens an in-memory database, which is the right default for a stateless, task-scoped analytical query. Supplying a connection id other than the default asserts that it exists, a missing connection raises rather than quietly falling back to an in-memory database.

Extensions are not downloaded at task runtime by default. DuckDB fetches them from its extension repository on first use, which a deployment without outbound internet access cannot do and which costs every worker the download. Pre-populate an extension directory, point extension_directory at it, and set autoinstall_extensions=True only if downloading on demand is acceptable.

Parameters:

  • duckdb_conn_id (str | None) – reference to a DuckDB connection. The default connection need not exist; when it is missing an in-memory database is used. Any other id must exist, and raises if it does not.

  • database (str | None) – database to open. Overrides the connection. :memory: (the default) opens a transient database that is discarded when the task finishes.

  • extensions (collections.abc.Sequence[str] | None) – extensions to load on connect, for example ["httpfs", "iceberg"].

  • extension_directory (str | None) – directory DuckDB loads extensions from and installs them into. Point this at a pre-populated directory to avoid downloading extensions at task runtime.

  • autoinstall_extensions (bool | None) – whether an extension that is not installed locally may be downloaded from DuckDB’s extension repository. Defaults to False: a locked-down environment usually cannot reach that repository, and downloading native code at task runtime is a decision worth making explicitly. Set to True to allow it.

  • autoload_extensions (bool | None) – whether DuckDB may load an already-installed extension implicitly, so that for example querying an s3:// path pulls in httpfs without it being listed in extensions. Defaults to True; this involves no download and no new code beyond what is already present.

  • allow_community_extensions (bool | None) – whether DuckDB may load community (third-party) extensions. Community extensions are native code from outside the DuckDB project, so this defaults to False.

  • memory_limit (str | None) – memory DuckDB may use, for example "2GB". DuckDB otherwise sizes itself from the memory it detects on the host, which over-commits inside a container that has a smaller limit than the host it runs on.

  • threads (int | None) – number of threads DuckDB may use. Defaults to the cores DuckDB detects, which is subject to the same container caveat as memory_limit.

  • temp_directory (str | None) – directory DuckDB spills to when a query exceeds memory_limit.

  • read_only (bool | None) – open the database read-only. Not valid for an in-memory database.

  • settings (dict[str, Any] | None) – additional DuckDB configuration options, passed through verbatim.

Every parameter above except duckdb_conn_id may also be set in the connection extra, in which case an explicit argument wins. This mirrors what BaseSQLOperator does when it merges connection extras into hook keyword arguments — a path the connection-optional operator has to bypass.

conn_name_attr = 'duckdb_conn_id'[source]
default_conn_name = 'duckdb_default'[source]
conn_type = 'duckdb'[source]
hook_name = 'DuckDB'[source]
placeholder = '?'[source]

Return SQL placeholder.

supports_autocommit = False[source]
required_extensions: tuple[str, ...] = ()[source]
database = None[source]
resolve_parameter(name, explicit, default=None)[source]

Return the explicit argument if given, else the connection extra, else the default.

property extension_directory: str | None[source]
property autoinstall_extensions: bool[source]
property autoload_extensions: bool[source]
property allow_community_extensions: bool[source]
property memory_limit: str | None[source]
property threads: int | None[source]
property temp_directory: str | None[source]
property read_only: bool[source]
property settings: dict[str, Any][source]
classmethod get_ui_field_behaviour()[source]

Return custom UI field behaviour for the DuckDB connection.

property airflow_connection: airflow.sdk.Connection | None[source]

Return the configured Airflow connection, or None when the default one does not exist.

A missing connection is only tolerated for the default connection id, which is what makes the hook usable with no configuration at all. An explicitly supplied id asserts that a particular connection exists, so its absence is a misconfiguration: falling back there would let a task that meant to write to a real database silently write to one that is discarded when it ends.

property connection_extra: dict[str, Any][source]

Return the connection’s extra, or an empty mapping when there is no connection.

get_database()[source]

Return the DuckDB database to open.

Precedence: the database argument, the connection extra database, a MotherDuck database built from the connection’s token, the connection host, then an in-memory database.

get_extensions()[source]

Return the extensions to load, in order, without duplicates.

get_connect_config()[source]

Return the DuckDB configuration applied when the database is opened.

These are passed to duckdb.connect(config=...) rather than issued as SET statements so that options which are only settable at startup are honored.

get_conn()[source]

Return a DuckDB connection with configuration, extensions and secrets applied.

load_extensions(conn)[source]

Load every configured extension, installing it only if it is not already available.

configure_secrets(conn)[source]

Create DuckDB secrets on a freshly opened connection.

A no-op for a plain DuckDB database. Subclasses that broker credentials for remote storage override this to issue CREATE SECRET.

get_uri()[source]

Return a SQLAlchemy URI for the database, for get_sqlalchemy_engine.

property dialect_name: str[source]

Return the SQL dialect name, without asking SQLAlchemy to resolve it.

DbApiHook derives this by handing get_uri() to SQLAlchemy and asking for a matching dialect, which only resolves when the third-party duckdb-engine package happens to be installed. Leaving it inferred would make dialect selection — and so the SQL generated for things like insert_rows — differ between deployments that are otherwise identical. Naming it here also keeps the MotherDuck token that get_uri() embeds out of SQLAlchemy’s URL parser.

Note this does not by itself populate reserved_words: that looks for a sqlalchemy.dialects.duckdb module, which does not exist. Choosing a keyword source is a separate decision.

Was this entry helpful?