tests.system.databricks.example_databricks¶
This is an example DAG which uses the DatabricksSubmitRunOperator. In this example, we create two tasks which execute sequentially. The first task is to run a notebook at the workspace path “/test” and the second task is to run a JAR uploaded to DBFS. Both, tasks use new clusters.
Because we have set a downstream dependency on the notebook task, the spark jar task will NOT run until the notebook task completes successfully.
The definition of a successful run is if the run has a result_state of “SUCCESS”. For more information about the state of a run refer to https://docs.databricks.com/api/latest/jobs.html#runstate