`apache-airflow-providers-apache-spark`¶

Content¶

Guides

References

Python API

Resources

Commits

Detailed list of commits

Package apache-airflow-providers-apache-spark¶

Apache Spark

Release: 2.1.3

Provider package¶

This is a provider package for apache.spark provider. All classes for this provider package are in airflow.providers.apache.spark python package.

Installation¶

You can install this package on top of an existing Airflow 2.1+ installation via pip install apache-airflow-providers-apache-spark

PIP requirements¶

PIP package	Version required
`apache-airflow`	`>=2.1.0`
`pyspark`

Changelog¶

2.1.3¶

Bug Fixes¶

Fix mistakenly added install_requires for all providers (#22382)

2.1.2¶

Misc¶

Add Trove classifiers in PyPI (Framework :: Apache Airflow :: Provider)

2.1.1¶

Bug Fixes¶

fix param rendering in docs of SparkSubmitHook (#21788)

Misc¶

Support for Python 3.10

2.1.0¶

Features¶

Add more SQL template fields renderers (#21237)
Add optional features in providers. (#21074)

2.0.3¶

Bug Fixes¶

Ensure Spark driver response is valid before setting UNKNOWN status (#19978)

2.0.2¶

Bug Fixes¶

fix bug of SparkSql Operator log going to infinite loop. (#19449)

2.0.1¶

Misc¶

Optimise connection importing for Airflow 2.2.0

2.0.0¶

Breaking changes¶

Auto-apply apply_default decorator (#15667)

Warning

Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+. If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded automatically and you will have to manually run airflow upgrade db to complete the migration.

Bug fixes¶

Make SparkSqlHook use Connection (#15794)

1.0.3¶

Bug fixes¶

Fix 'logging.exception' redundancy (#14823)

1.0.2¶

Bug fixes¶

Use apache.spark provider without kubernetes (#14187)

1.0.1¶

Updated documentation and readme files.

1.0.0¶

Initial version of the provider.