Content¶
Guides
References
Commits
apache-airflow-providers-databricks¶
Provider package¶
This is a provider package for databricks provider. All classes for this provider package
are in airflow.providers.databricks python package.
Installation¶
You can install this package on top of an existing Airflow 2 installation (see Requirements below)
for the minimum Airflow version supported) via
pip install apache-airflow-providers-databricks
Requirements¶
PIP package |
Version required |
|---|---|
|
|
|
|
|
|
|
|
|
|
Cross provider package dependencies¶
Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified provider packages in order to use them.
You can install such cross-provider dependencies when installing from PyPI. For example:
pip install apache-airflow-providers-databricks[common.sql]
Dependent package |
Extra |
|---|---|
|
Downloading official packages¶
You can download officially released packages and verify their checksums and signatures from the Official Apache Download site
The apache-airflow-providers-databricks 3.1.0 sdist package (asc, sha512)
The apache-airflow-providers-databricks 3.1.0 wheel package (asc, sha512)
Changelog¶
3.1.0¶
Features¶
Added databricks_conn_id as templated field (#24945)Add 'test_connection' method to Databricks hook (#24617)Move all SQL classes to common-sql provider (#24836)
Bug Fixes¶
Update providers to use functools compat for ''cached_property'' (#24582)
3.0.0¶
Breaking changes¶
This release of provider is only available for Airflow 2.2+ as explained in the Apache Airflow providers support policy https://github.com/apache/airflow/blob/main/README.md#support-for-providers
Features¶
Add Deferrable Databricks operators (#19736)Add git_source to DatabricksSubmitRunOperator (#23620)
Bug Fixes¶
fix: DatabricksSubmitRunOperator and DatabricksRunNowOperator cannot define .json as template_ext (#23622) (#23641)Fix UnboundLocalError when sql is empty list in DatabricksSqlHook (#23815)
2.7.0¶
Features¶
Update to the released version of DBSQL connectorDatabricksSqlOperator - switch to databricks-sql-connector 2.xFurther improvement of Databricks Jobs operators (#23199)
2.6.0¶
Features¶
More operators for Databricks Repos (#22422)Add a link to Databricks Job Run (#22541)Databricks SQL operators are now Python 3.10 compatible (#22886)
Bug Fixes¶
Databricks: Correctly handle HTTP exception (#22885)
Misc¶
Refactor 'DatabricksJobRunLink' to not create ad hoc TaskInstances (#22571)
2.5.0¶
Features¶
Operator for updating Databricks Repos (#22278)
Bug Fixes¶
Fix mistakenly added install_requires for all providers (#22382)
2.4.0¶
Features¶
Add new options to DatabricksCopyIntoOperator (#22076)Databricks hook - retry on HTTP Status 429 as well (#21852)
Misc¶
Skip some tests for Databricks from running on Python 3.10 (#22221)
2.3.0¶
Features¶
Add-showing-runtime-error-feature-to-DatabricksSubmitRunOperator (#21709)Databricks: add support for triggering jobs by name (#21663)Added template_ext = ('.json') to databricks operators #18925 (#21530)Databricks SQL operators (#21363)
Bug Fixes¶
Fixed changelog for January 2022 (delayed) provider's release (#21439)
Misc¶
Support for Python 3.10Updated Databricks docs for correct jobs 2.1 API and links (#21494)
2.2.0¶
Features¶
Add 'wait_for_termination' argument for Databricks Operators (#20536)Update connection object to ''cached_property'' in ''DatabricksHook'' (#20526)Remove 'host' as an instance attr in 'DatabricksHook' (#20540)Databricks: fix verification of Managed Identity (#20550)
2.1.0¶
Features¶
Databricks: add more methods to represent run state information (#19723)Databricks - allow Azure SP authentication on other Azure clouds (#19722)Databricks: allow to specify PAT in Password field (#19585)Databricks jobs 2.1 (#19544)Update Databricks API from 2.0 to 2.1 (#19412)Authentication with AAD tokens in Databricks provider (#19335)Update Databricks operators to match latest version of API 2.0 (#19443)Remove db call from DatabricksHook.__init__() (#20180)
Bug Fixes¶
Fixup string concatenations (#19099)Databricks hook: fix expiration time check (#20036)
2.0.0¶
Breaking changes¶
Auto-apply apply_default decorator (#15667)
Warning
Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
automatically and you will have to manually run airflow upgrade db to complete the migration.
1.0.1¶
Updated documentation and readme files.
1.0.0¶
Initial version of the provider.