apache-airflow-providers-openlineage

Changelog¶

1.8.0¶

Warning

In Airflow 2.10.0, we fix the way try_number works. For Airflow >= 2.10.0, use apache-airflow-providers-openlineage >= 1.8.0. Older versions of Airflow are not affected, In case you run an incompatible version an exception will be raised asking you to upgrade provider version.

Features¶

Scheduler to handle incrementing of try_number (#39336)

Bug Fixes¶

fix: Prevent error when extractor can't be imported (#39736)
Re-configure ORM in spawned OpenLineage process in scheduler. (#39735)

Misc¶

chore: Update conf retrieval docstring and adjust pool_size (#39721)
Remove 'openlineage.common' dependencies in Google and Snowflake providers. (#39614)
Use 'ProcessPoolExecutor' over 'ThreadPoolExecutor'. (#39235)
misc: Add custom provider runtime checks (#39609)
Faster 'airflow_version' imports (#39552)
Simplify 'airflow_version' imports (#39497)
openlineage: notify that logged exception was caught (#39493)
chore: Add more OpenLineage logs to facilitate debugging (#39136)

1.7.1¶

Misc¶

openlineage, snowflake: do not run external queries for Snowflake (#39113)

1.7.0¶

Features¶

Add lineage_job_namespace and lineage_job_name OpenLineage macros (#38829)
openlineage: add 'opt-in' option (#37725)

Bug Fixes¶

fix: Remove redundant operator information from facets (#38264)
fix: disabled_for_operators now stops whole event emission (#38033)
fix: Add fallbacks when retrieving Airflow configuration to avoid errors being raised (#37994)
fix: Fix parent id macro and remove unused utils (#37877)

Misc¶

Avoid use of 'assert' outside of the tests (#37718)
Add default for 'task' on TaskInstance / fix attrs on TaskInstancePydantic (#37854)

1.6.0¶

Features¶

feat: Add OpenLineage metrics for event_size and extraction time (#37797)
feat: Add OpenLineage support for File and User Airflow's lineage entities (#37744)
[OpenLineage] Add support for JobTypeJobFacet properties. (#37255)
chore: Update comments and logging in OpenLineage ExtractorManager (#37622)

Bug Fixes¶

fix: Check if operator is disabled in DefaultExtractor.extract_on_complete (#37392)

Misc¶

docs: Update whole OpenLineage Provider docs. (#37620)

1.5.0¶

Features¶

feat: Add dag_id when generating OpenLineage run_id for task instance. (#36659)

1.4.0¶

Features¶

Add OpenLineage support for Redshift SQL. (#35794)

1.3.1¶

Bug Fixes¶

Fix typo. (#36362)

1.3.0¶

Features¶

feat: Add parent_run_id for COMPLETE and FAIL events (#36067)
Add basic metrics to stats collector. (#35368)

Bug Fixes¶

fix: Repair run_id for OpenLineage FAIL events (#36051)
Fix and reapply templates for provider documentation (#35686)

Misc¶

Remove ClassVar annotations. (#36084)

1.2.1¶

Misc¶

Make schema filter uppercase in 'create_filter_clauses' (#35428)

1.2.0¶

Features¶

Send column lineage from SQL operators. (#34843)

1.1.1¶

Misc¶

Adjust log levels in OpenLineage provider (#34801)

1.1.0¶

Features¶

Allow to disable openlineage at operator level (#33685)

Bug Fixes¶

Fix import in 'get_custom_facets'. (#34122)

Misc¶

Improve modules import in Airflow providers by some of them into a type-checking block (#33754)
Add OpenLineage support for DBT Cloud. (#33959)
Refactor unneeded jumps in providers (#33833)
Refactor: Replace lambdas with comprehensions in providers (#33771)

1.0.2¶

Bug Fixes¶

openlineage: don't run task instance listener in executor (#33366)
openlineage: do not try to redact Proxy objects from deprecated config (#33393)
openlineage: defensively check for provided datetimes in listener (#33343)

Misc¶

Add OpenLineage support for Trino. (#32910)
Simplify conditions on len() in other providers (#33569)
Replace repr() with proper formatting (#33520)

1.0.1¶

Bug Fixes¶

openlineage: disable running listener if not configured (#33120)
Don't use database as fallback when no schema parsed. (#32959)

Misc¶

openlineage, bigquery: add openlineage method support for BigQueryExecuteQueryOperator (#31293)
Move openlineage configuration to provider (#33124)

1.0.0¶

Initial version of the provider.