airflow.providers.duckdb.operators.duckdb

Classes

DuckDBExecuteQueryOperator

Run SQL against an in-process DuckDB database.

Module Contents

class airflow.providers.duckdb.operators.duckdb.DuckDBExecuteQueryOperator(*, conn_id=DuckDBHook.default_conn_name, **kwargs)[source]

Bases: airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator

Run SQL against an in-process DuckDB database.

Unlike the other SQLExecuteQueryOperator subclasses this operator does not require an Airflow connection to exist: with no duckdb_default connection configured it runs against an in-memory database, which is the common case for a task that reads its input, transforms it and writes its output back out.

When a connection does exist, its type selects the hook. A provider can therefore ship a DuckDB connection type that adds its own behaviour (the Amazon provider’s duckdb_aws brokers AWS credentials, for example) and a Dag moves between them by changing connections.

See also

For more information on how to use this operator, take a look at the guide: Run a query

Parameters:

  • sql – the SQL statement, list of statements, or .sql template file to run.

  • conn_id (str) – DuckDB connection to use, or a connection of another DuckDB type such as duckdb_aws. The default connection need not exist, in which case an in-memory database is used. Any other id must exist, and raises if it does not.

  • database – DuckDB database to open, for example /tmp/analytics.duckdb. Overrides both the connection and hook_params["database"].

  • hook_params – extra keyword arguments for DuckDBHook, for example {"extensions": ["httpfs"], "memory_limit": "2GB"}.

ui_color = '#ffe873'[source]
hook_class: ClassVar[type[airflow.providers.duckdb.hooks.duckdb.DuckDBHook]][source]
get_db_hook()[source]

Resolve the hook from the connection, and build one directly when there is no connection.

BaseSQLOperator resolves the hook through BaseHook.get_connection(conn_id).get_hook(), which raises when the connection is absent. DuckDB needs no connection to be useful, so a missing one is handed to hook_class to interpret rather than being an error here.

When the connection does exist its type chooses the hook. That is what lets another provider add cloud-specific behaviour, such as brokering credentials, without a Dag having to swap operator: pointing this operator at a connection of that type is enough.

Was this entry helpful?