Databricks SQL warehouse lifecycle operators¶

Use DatabricksStartWarehouseOperator and DatabricksStopWarehouseOperator to start and stop an existing Databricks SQL warehouse through the Databricks SQL Warehouses API. Both operators require the warehouse ID and use the Databricks connection for authentication.

By default, each operator waits for the requested state: RUNNING when starting and STOPPED when stopping. Use polling_period_seconds to control the polling interval, timeout to limit the wait, or wait_for_termination=False to return after requesting the transition. Repeated task attempts are safe: an already running warehouse is not started again, and an already stopped warehouse is not stopped again. If a start is requested while a warehouse is stopping, any transition rejection from Databricks is propagated to the task.

Start a SQL warehouse¶

tests/system/databricks/example_databricks_sql_warehouse.py[source]

    start_warehouse = DatabricksStartWarehouseOperator(
        task_id="start_warehouse",
        databricks_conn_id="databricks_default",
        warehouse_id=WAREHOUSE_ID,
        wait_for_termination=True,
    )

Stop a SQL warehouse¶

The stop task uses the all_done trigger rule so the warehouse is stopped even when an upstream task fails.

tests/system/databricks/example_databricks_sql_warehouse.py[source]

    stop_warehouse = DatabricksStopWarehouseOperator(
        task_id="stop_warehouse",
        databricks_conn_id="databricks_default",
        warehouse_id=WAREHOUSE_ID,
        wait_for_termination=True,
        trigger_rule="all_done",
    )