DatabricksSQLStatementsOperator¶

Use the DatabricksSQLStatementsOperator to submit a Databricks SQL Statement to Databricks using the Databricks SQL Statement Execution API.

Using the Operator¶

The DatabricksSQLStatementsOperator submits SQL statements to Databricks using the /api/2.0/sql/statements/ endpoint. It supports configurable execution parameters such as warehouse selection, catalog, schema, and parameterized queries. The operator can either synchronously poll for query completion or run in a deferrable mode for improved efficiency.

The only required parameters for using the operator are:

statement - The SQL statement to execute. The statement can optionally be parameterized, see parameters.
warehouse_id - Warehouse upon which to execute a statement.

All other parameters are optional and described in the documentation for DatabricksSQLStatementsOperator including but not limited to:

catalog
schema
parameters

Examples¶

An example usage of the DatabricksSQLStatementsOperator is as follows:

tests/system/databricks/example_databricks.py[source]

    sql_statement = DatabricksSQLStatementsOperator(
        task_id="sql_statement",
        databricks_conn_id="databricks_default",
        statement="select * from default.my_airflow_table",
        warehouse_id=WAREHOUSE_ID,
        # deferrable=True, # For using the operator in deferrable mode
    )