Content¶
Guides
References
Commits
apache-airflow-providers-apache-spark¶
Provider package¶
This is a provider package for apache.spark provider. All classes for this provider package
are in airflow.providers.apache.spark python package.
Installation¶
You can install this package on top of an existing Airflow 2 installation (see Requirements below)
for the minimum Airflow version supported) via
pip install apache-airflow-providers-apache-spark
Changelog¶
4.0.0¶
This release of provider is only available for Airflow 2.3+ as explained in the Apache Airflow providers support policy.
Breaking changes¶
The spark-binary connection extra could be set to any binary, but with 4.0.0 version only two values
are allowed for it spark-submit and spark2-submit.
The spark-home connection extra is not allowed any more - the binary should be available on the
PATH in order to use SparkSubmitHook and SparkSubmitOperator.
Remove custom spark home and custom binaries for spark (#27646)
Misc¶
Move min airflow version to 2.3.0 for all providers (#27196)
3.0.0¶
Breaking changes¶
This release of provider is only available for Airflow 2.2+ as explained in the Apache Airflow providers support policy https://github.com/apache/airflow/blob/main/README.md#support-for-providers
Bug Fixes¶
Add typing for airflow/configuration.py (#23716)Fix backwards-compatibility introduced by fixing mypy problems (#24230)
Misc¶
AIP-47 - Migrate spark DAGs to new design #22439 (#24210)chore: Refactoring and Cleaning Apache Providers (#24219)
2.1.1¶
Bug Fixes¶
fix param rendering in docs of SparkSubmitHook (#21788)
Misc¶
Support for Python 3.10
2.1.0¶
Features¶
Add more SQL template fields renderers (#21237)Add optional features in providers. (#21074)
2.0.0¶
Breaking changes¶
Auto-apply apply_default decorator (#15667)
Warning
Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
automatically and you will have to manually run airflow upgrade db to complete the migration.
Bug fixes¶
Make SparkSqlHook use Connection (#15794)
1.0.1¶
Updated documentation and readme files.
1.0.0¶
Initial version of the provider.