Package apache-airflow-providers-apache-spark¶

Apache Spark

This is detailed commit list of changes for versions provider package: apache.spark. For high-level changelog, see package information including changelog.

4.1.3¶

Latest change: 2023-07-31

Commit	Committed	Subject
4f83e831d2	2023-07-31	`Validate conn_prefix in extra field for Spark JDBC hook (#32946)`

4.1.2¶

Latest change: 2023-07-29

Commit	Committed	Subject
d06b7af69a	2023-07-29	`Prepare docs for July 2023 3rd wave of Providers (#32875)`
e93460383f	2023-07-26	`Move all k8S classes to cncf.kubernetes provider (#32767)`
225e3041d2	2023-07-06	`Prepare docs for July 2023 wave of Providers (RC2) (#32381)`
3878fe6fab	2023-07-05	`Remove spurious headers for provider changelogs (#32373)`
cb4927a018	2023-07-05	`Prepare docs for July 2023 wave of Providers (#32298)`
8c37b74a20	2023-06-28	`D205 Support - Providers: Apache to Common (inclusive) (#32226)`
09d4718d3a	2023-06-27	`Improve provider documentation and README structure (#32125)`

4.1.1¶

Latest change: 2023-06-20

Commit	Committed	Subject
79bcc2e668	2023-06-20	`Prepare RC1 docs for June 2023 wave of Providers (#32001)`
8b146152d6	2023-06-20	`Add note about dropping Python 3.7 for providers (#32015)`
6becb70316	2023-06-16	`SparkSubmitOperator: rename spark_conn_id to conn_id (#31952)`
13890788ae	2023-06-07	`Apache provider docstring improvements (#31730)`
9276310a43	2023-06-05	`Improve docstrings in providers (#31681)`
a473facf6c	2023-06-01	`Add D400 pydocstyle check - Apache providers only (#31424)`

4.1.0¶

Latest change: 2023-05-19

Commit	Committed	Subject
45548b9451	2023-05-19	`Prepare RC2 docs for May 2023 wave of Providers (#31416)`
abea189022	2023-05-18	`Use '__version__' in providers not 'version' (#31393)`
f5aed58d9f	2023-05-18	`Fixing circular import error in providers caused by airflow version check (#31379)`
d9ff55cf6d	2023-05-16	`Prepare docs for May 2023 wave of Providers (#31252)`
eef5bc7f16	2023-05-03	`Add full automation for min Airflow version for providers (#30994)`
a7eb32a5b2	2023-04-30	`Bump minimum Airflow version in providers (#30917)`
d23a3bbed8	2023-04-04	`Add mechanism to suspend providers (#30422)`

4.0.1¶

Latest change: 2023-04-02

Commit	Committed	Subject
55dbf1ff1f	2023-04-02	`Prepare docs for April 2023 wave of Providers (#30378)`
5d1f201bb0	2023-03-22	`Only restrict spark binary passed via extra (#30213)`
d9dea5ce17	2023-03-22	`Validate host and schema for Spark JDBC Hook (#30223)`
b3259877fa	2023-03-15	`Add spark3-submit to list of allowed spark-binary values (#30068)`

4.0.0¶

Latest change: 2022-11-15

Commit	Committed	Subject
12c3c39d1a	2022-11-15	`pRepare docs for November 2022 wave of Providers (#27613)`
9358928815	2022-11-13	`Remove custom spark home and custom binarires for spark (#27646)`
78b8ea2f22	2022-10-24	`Move min airflow version to 2.3.0 for all providers (#27196)`
2a34dc9e84	2022-10-23	`Enable string normalization in python formatting - providers (#27205)`
f8db64c35c	2022-09-28	`Update docs for September Provider's release (#26731)`
06acf40a43	2022-09-13	`Apply PEP-563 (Postponed Evaluation of Annotations) to non-core airflow (#26289)`
e5ac6c7cfb	2022-08-10	`Prepare docs for new providers release (August 2022) (#25618)`
d2459a241b	2022-07-13	`Add documentation for July 2022 Provider's release (#25030)`
0de31bd73a	2022-06-29	`Move provider dependencies to inside provider folders (#24672)`
510a6bab45	2022-06-28	`Remove 'hook-class-names' from provider.yaml (#24702)`

3.0.0¶

Latest change: 2022-06-09

Commit	Committed	Subject
dcdcf3a2b8	2022-06-09	`Update release notes for RC2 release of Providers for May 2022 (#24307)`
717a7588bc	2022-06-07	`Update package description to remove double min-airflow specification (#24292)`
aeabe994b3	2022-06-07	`Prepare docs for May 2022 provider's release (#24231)`
b4a5783a2a	2022-06-06	`chore: Refactoring and Cleaning Apache Providers (#24219)`
9dc2851671	2022-06-05	`Fix backwards-compatibility introduced by fixing mypy problems (#24230)`
027b707d21	2022-06-05	`Add explanatory note for contributors about updating Changelog (#24229)`
a2bfc0e62d	2022-06-05	`AIP-47 - Migrate spark DAGs to new design #22439 (#24210)`
71e4deb1b0	2022-05-16	`Add typing for airflow/configuration.py (#23716)`

2.1.3¶

Latest change: 2022-03-22

Commit	Committed	Subject
d7dbfb7e26	2022-03-22	`Add documentation for bugfix release of Providers (#22383)`

2.1.2¶

Latest change: 2022-03-14

Commit	Committed	Subject
16adc035b1	2022-03-14	`Add documentation for Classifier release for March 2022 (#22226)`

2.1.1¶

Latest change: 2022-03-07

Commit	Committed	Subject
f5b96315fe	2022-03-07	`Add documentation for Feb Providers release (#22056)`
6322dad2ca	2022-02-25	`fix param rendering in docs of SparkSubmitHook (#21788)`

2.1.0¶

Latest change: 2022-02-08

Commit	Committed	Subject
d94fa37830	2022-02-08	`Fixed changelog for January 2022 (delayed) provider's release (#21439)`
6c3a67d4fc	2022-02-05	`Add documentation for January 2021 providers release (#21257)`
39e395f981	2022-02-04	`Add more SQL template fields renderers (#21237)`
cb73053211	2022-01-27	`Add optional features in providers. (#21074)`
602abe8394	2022-01-20	`Remove ':type' lines now sphinx-autoapi supports typehints (#20951)`

2.0.3¶

Latest change: 2021-12-31

Commit	Committed	Subject
f77417eb0d	2021-12-31	`Fix K8S changelog to be PyPI-compatible (#20614)`
97496ba2b4	2021-12-31	`Update documentation for provider December 2021 release (#20523)`
83f8e178ba	2021-12-31	`Even more typing in operators (template_fields/ext) (#20608)`
d56e7b56bb	2021-12-30	`Fix template_fields type to have MyPy friendly Sequence type (#20571)`
485ff6cc64	2021-12-29	`Fix MyPy errors in Apache Providers (#20422)`
dad2f8103b	2021-12-15	`Fix mypy providers (#20190)`
1a2a2498d6	2021-12-14	`Fix mypy spark hooks (#20290)`
a50d2ac872	2021-12-02	`Ensure Spark driver response is valid before setting UNKNOWN status (#19978)`

2.0.2¶

Latest change: 2021-11-30

Commit	Committed	Subject
853576d901	2021-11-30	`Update documentation for November 2021 provider's release (#19882)`
79b30ff59c	2021-11-24	`fix bug of SparkSql Operator log going to infinite loop. (#19449)`
ae044884d1	2021-11-03	`Cleanup of start_date and default arg use for Apache example DAGs (#18657)`
d9567eb106	2021-10-29	`Prepare documentation for October Provider's release (#19321)`
86a2a19ad2	2021-10-17	`More f-strings (#18855)`
42dc0767b8	2021-10-08	`Remove unnecessary string concatenations in AirflowException messages (#18817)`

2.0.1¶

Latest change: 2021-08-30

Commit	Committed	Subject
0a68588479	2021-08-30	`Add August 2021 Provider's documentation (#17890)`
be75dcd39c	2021-08-23	`Update description about the new ''connection-types'' provider meta-data`
76ed2a49c6	2021-08-19	`Import Hooks lazily individually in providers manager (#17682)`
87f408b1e7	2021-07-26	`Prepares docs for Rc2 release of July providers (#17116)`
91f4d80ff0	2021-07-23	`Updating Apache example DAGs to use XComArgs (#16869)`
b916b75079	2021-07-15	`Prepare documentation for July release of providers. (#17015)`
866a601b76	2021-06-28	`Removes pylint from our toolchain (#16682)`

2.0.0¶

Latest change: 2021-06-18

Commit	Committed	Subject
bbc627a3da	2021-06-18	`Prepares documentation for rc2 release of Providers (#16501)`
cbf8001d76	2021-06-16	`Synchronizes updated changelog after buggfix release (#16464)`
1fba5402bb	2021-06-15	`More documentation update for June providers release (#16405)`
5c86e3d509	2021-06-14	`Make SparkSqlHook use Connection (#15794)`
9c94b72d44	2021-06-07	`Updated documentation for June 2021 provider release (#16294)`
37681bca00	2021-05-07	`Auto-apply apply_default decorator (#15667)`

1.0.3¶

Latest change: 2021-05-01

Commit	Committed	Subject
807ad32ce5	2021-05-01	`Prepares provider release after PIP 21 compatibility (#15576)`
5b2fe0e740	2021-04-27	`Add Connection Documentation for Popular Providers (#15393)`
4b031d39e1	2021-04-27	`Make Airflow code Pylint 2.8 compatible (#15534)`
657384615f	2021-04-27	`Fix 'logging.exception' redundancy (#14823)`
9015beb316	2021-04-10	`Pass environment variables to process with yarn kill command (#15304)`
68e4c4dcb0	2021-03-20	`Remove Backport Providers (#14886)`

1.0.2¶

Latest change: 2021-02-27

Commit	Committed	Subject
589d6dec92	2021-02-27	`Prepare to release the next wave of providers: (#14487)`
f9c9e9c38f	2021-02-11	`Use apache.spark provider without kubernetes (#14187)`

1.0.1¶

Latest change: 2021-02-04

Commit	Committed	Subject
88bdcfa0df	2021-02-04	`Prepare to release a new wave of providers. (#14013)`
ac2f72c98d	2021-02-01	`Implement provider versioning tools (#13767)`
a9ac2b040b	2021-01-23	`Switch to f-strings using flynt. (#13732)`
295d66f914	2020-12-30	`Fix Grammar in PIP warning (#13380)`
6cf76d7ac0	2020-12-18	`Fix typo in pip upgrade command :( (#13148)`
5090fb0c89	2020-12-15	`Add script to generate integrations.json (#13073)`

1.0.0¶

Latest change: 2020-12-09

Commit	Committed	Subject
32971a1a2d	2020-12-09	`Updates providers versions to 1.0.0 (#12955)`
b40dffa085	2020-12-08	`Rename remaing modules to match AIP-21 (#12917)`
9b39f24780	2020-12-08	`Add support for dynamic connection form fields per provider (#12558)`
bd90136aaf	2020-11-30	`Move operator guides to provider documentation packages (#12681)`
c34ef853c8	2020-11-20	`Separate out documentation building per provider (#12444)`
0080354502	2020-11-18	`Update provider READMEs for 1.0.0b2 batch release (#12449)`
ae7cb4a1e2	2020-11-17	`Update wrong commit hash in backport provider changes (#12390)`
6889a333cf	2020-11-15	`Improvements for operators and hooks ref docs (#12366)`
7825e8f590	2020-11-13	`Docs installation improvements (#12304)`
85a18e13d9	2020-11-09	`Point at pypi project pages for cross-dependency of provider packages (#12212)`
59eb5de78c	2020-11-09	`Update provider READMEs for up-coming 1.0.0beta1 releases (#12206)`
b2a28d1590	2020-11-09	`Moves provider packages scripts to dev (#12082)`
4e8f9cc8d0	2020-11-03	`Enable Black - Python Auto Formmatter (#9550)`
8c42cf1b00	2020-11-03	`Use PyUpgrade to use Python 3.6 features (#11447)`
5a439e84eb	2020-10-26	`Prepare providers release 0.0.2a1 (#11855)`
872b1566a1	2020-10-25	`Generated backport providers readmes/setup for 2020.10.29 (#11826)`
349b0811c3	2020-10-20	`Add D200 pydocstyle check (#11688)`
16e7129719	2020-10-13	`Added support for provider packages for Airflow 2.0 (#11487)`
d305876bee	2020-10-12	`Remove redundant None provided as default to dict.get() (#11448)`
0a0e1af800	2020-10-03	`Fix Broken Markdown links in Providers README TOC (#11249)`
ca4238eb4d	2020-10-02	`Fixed month in backport packages to October (#11242)`
5220e4c384	2020-10-02	`Prepare Backport release 2020.09.07 (#11238)`
f3e87c5030	2020-09-22	`Add D202 pydocstyle check (#11032)`
fdd9b6f65b	2020-08-25	`Enable Black on Providers Packages (#10543)`
d760265452	2020-08-25	`PyDocStyle: No whitespaces allowed surrounding docstring text (#10533)`
d1bce91bb2	2020-08-25	`PyDocStyle: Enable D403: Capitalized first word of docstring (#10530)`
3696c34c28	2020-08-24	`Fix typo in the word "release" (#10528)`
ee7ca128a1	2020-08-22	`Fix broken Markdown refernces in Providers README (#10483)`
7c206a82a6	2020-08-22	`Replace assigment with Augmented assignment (#10468)`
3b3287d7ac	2020-08-05	`Enforce keyword only arguments on apache operators (#10170)`
7d24b088cd	2020-07-25	`Stop using start_date in default_args in example_dags (2) (#9985)`
33f0cd2657	2020-07-22	`apply_default keeps the function signature for mypy (#9784)`
1427e4acb4	2020-07-22	`Update Spark submit operator for Spark 3 support (#8730)`
4d74ac2111	2020-07-19	`Increase typing for Apache and http provider package (#9729)`
0873070e08	2020-07-11	`Mask other forms of password arguments in SparkSubmitOperator (#9615)`
13a827d80f	2020-07-09	`Ensure Kerberos token is valid in SparkSubmitOperator before running 'yarn kill' (#9044)`
067806d598	2020-06-29	`Add tests for spark_jdbc_script (#9491)`
d0e7db4024	2020-06-19	`Fixed release number for fresh release (#9408)`
12af6a0800	2020-06-19	`Final cleanup for 2020.6.23rc1 release preparation (#9404)`
c7e5bce57f	2020-06-19	`Prepare backport release candidate for 2020.6.23rc1 (#9370)`
40bf8f28f9	2020-06-18	`Detect automatically the lack of reference to the guide in the operator descriptions (#9290)`
f6bd817a3a	2020-06-16	`Introduce 'transfers' packages (#9320)`
0b0e4f7a4c	2020-05-26	`Preparing for RC3 relase of backports (#9026)`
00642a46d0	2020-05-26	`Fixed name of 20 remaining wrongly named operators. (#8994)`
375d1ca229	2020-05-19	`Release candidate 2 for backport packages 2020.05.20 (#8898)`
12c5e5d8ae	2020-05-17	`Prepare release candidate for backport packages (#8891)`
f3521fb0e3	2020-05-16	`Regenerate readme files for backport package release (#8886)`
92585ca4cb	2020-05-15	`Added automated release notes generation for backport operators (#8807)`
7506c73f17	2020-05-10	`Add default 'conf' parameter to Spark JDBC Hook (#8787)`
487b5cc50c	2020-05-06	`Add guide for Apache Spark operators (#8305)`
87969a350d	2020-04-09	`[AIRFLOW-6515] Change Log Levels from Info/Warn to Error (#8170)`
be1451b0e1	2020-04-02	`[AIRFLOW-7026] Improve SparkSqlHook's error message (#7749)`
4bde99f132	2020-03-23	`Make airflow/providers pylint compatible (#7802)`
7e6372a681	2020-03-23	`Add call to Super call in apache providers (#7820)`
2327aa5a26	2020-03-12	`[AIRFLOW-7025] Fix SparkSqlHook.run_query to handle its parameter properly (#7677)`
024b4bf962	2020-03-10	`[AIRFLOW-7024] Add the verbose parameter support to SparkSqlOperator (#7676)`
b59042b5ab	2020-02-28	`[AIRFLOW-6949] Respect explicit 'spark.kubernetes.namespace' conf to SparkSubmitOperator (#7575)`
97a429f9d0	2020-02-02	`[AIRFLOW-6714] Remove magic comments about UTF-8 (#7338)`
0481b9a957	2020-01-12	`[AIRFLOW-6539][AIP-21] Move Apache classes to providers.apache package (#7142)`