apache-airflow-providers-common-sql

Changelog

1.20.0

Features

  • Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook (#41916)

1.19.0

Features

  • Feature: Added fast_executemany parameter to insert_rows of DbApiHook (#43357)

  • Make conn id parameters templated in GenericTransfer and also allow passing hook parameters like in BaseSQLOperator (#42891)

  • Allow capture of replication lag in SqlSensor (#43107)

Misc

  • fix mypy failure (#43206)

1.18.0

Features

  • feat(providers/common/sql): add warning to connection setter (#42736)

Bug Fixes

  • FIX: Only pass connection to sqlalchemy engine in JdbcHook (#42705)

1.17.1

Bug Fixes

  • fix(providers/common/sql): add dummy connection setter for backward compatibility (#42490)

  • Changed type hinting for handler function (#42275)

1.17.0

Features

Note

Connection in DB Hook is now cached to avoid multiple lookups when properties from extras have to be resolved.

  • Generalize caching of connection in DbApiHook to improve performance (#40751)

Misc

  • feat: log client db messages for provider postgres (#40171)

  • remove deprecated soft_fail from providers (#41710)

1.16.0

Note

This release of provider is only available for Airflow 2.8+ as explained in the Apache Airflow providers support policy.

Bug Fixes

  • fix: rm deprecated import (#41461)

Misc

  • Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)

1.15.0

Features

  • Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669)

1.14.2

Bug Fixes

  • FIX: DbApiHook.insert_rows unnecessarily restarting connections (#40615)

Misc

  • Enable enforcing pydocstyle rule D213 in ruff. (#40448)

1.14.1

Misc

  • implement per-provider tests with lowest-direct dependency resolution (#39946)

  • Update pandas minimum requirement for Python 3.12 (#40272)

  • standardizes template fields for 'BaseSQLOperator' and adds 'database' as a templated field (#39826)

1.14.0

Features

  • Add 'parameters' as template field for SqlSensor (#39588)

Bug Fixes

  • DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is formatted correctly (#39690)

Misc

  • refactor: The executemany parameter of insert_rows should not be deprecated as for some hooks we don't want to enable a system-wide supports_executemany parameter, that way we can also keep using it in dedicated situations (#39630)

  • Faster 'airflow_version' imports (#39552)

  • Simplify 'airflow_version' imports (#39497)

  • Add typing for SqlSensor (#39773)

1.13.0

Note

This release of provider is only available for Airflow 2.7+ as explained in the Apache Airflow providers support policy.

Features

  • Add 'sqlalchemy_url' property to 'DbApiHook' class (#38871)

  • Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715)

Bug Fixes

  • Fix 'DbApiHook.insert_rows' when 'rows' is a generator (#38972)

  • Fix 'update-common-sql-api-stubs' pre-commit check (#38915)

Misc

  • Bump minimum Airflow version in providers to Airflow 2.7.0 (#39240)

  • openlineage, snowflake: do not run external queries for Snowflake (#39113)

1.12.0

Features

  • Add hook_params to template_fields for BaseSQLOperator-related Operators (#38724)

  • Make 'placeholder' of DbApiHook configurable in UI (#38528)

Misc

  • Undeprecating 'DBApiHookForTests._make_common_data_structure' (#38573)

1.11.1

Bug Fixes

  • Make 'executemany' keyword arguments only in 'DbApiHook.insert_rows' (#37840)

  • Limit 'pandas' to '<2.2' (#37748)

1.11.0

Features

  • Enhancement: Performance enhancement for insert_rows method DbApiHook with fast executemany + SAP Hana support (#37246)

Bug Fixes

  • Fix SQLThresholdCheckOperator error on falsey vals (#37150)

Misc

  • feat: Switch all class, functions, methods deprecations to decorators (#36876)

  • Add more-itertools as dependency of common-sql (#37359)

1.10.1

Misc

  • Set min pandas dependency to 1.2.5 for all providers and airflow (#36698)

1.10.0

  • Make "placeholder" of ODBC configurable in UI (#36000)

Bug Fixes

  • Return common data structure in DBApi derived classes

  • SQLCheckOperator fails if returns dict with any False values (#36273)

1.9.0

Note

This release of provider is only available for Airflow 2.6+ as explained in the Apache Airflow providers support policy.

Misc

  • Bump minimum Airflow version in providers to Airflow 2.6.0 (#36017)

  • Add Architecture Decision Record for common.sql introduction (#36015)

1.8.1

Misc

  • Add '_make_serializable' method which other SQL operators can overrides when result from cursor is not JSON-serializable (#32319)

  • Remove backcompat inheritance for DbApiHook (#35754)

1.8.0

Note

This release of provider is only available for Airflow 2.5+ as explained in the Apache Airflow providers support policy.

Bug Fixes

  • fix(providers/sql): respect soft_fail argument when exception is raised (#34199)

Misc

  • Bump min airflow version of providers (#34728)

  • Use 'airflow.exceptions.AirflowException' in providers (#34511)

1.7.2

Bug Fixes

  • Fix BigQueryValueCheckOperator deferrable mode optimisation (#34018)

Misc

  • Refactor regex in providers (#33898)

1.7.1

Misc

  • Refactor: Better percentage formatting (#33595)

  • Refactor: Simplify code in smaller providers (#33234)

1.7.0

Features

  • Add a new parameter to SQL operators to specify conn id field (#30784)

1.6.2

Misc

  • Make SQLExecute Query signature consistent with other SQL operators (#32974)

  • Get rid of Python2 numeric relics (#33050)

1.6.1

Bug Fixes

  • Fix local OpenLineage import in 'SQLExecuteQueryOperator'. (#32400)

Misc

  • Add default port to Openlineage authority method. (#32828)

  • Add more accurate typing for DbApiHook.run method (#31846)

1.6.0

Features

  • openlineage, common.sql:  provide OL SQL parser as internal OpenLineage provider API (#31398)

Misc

  • Pass SQLAlchemy engine to construct information schema query. (#32371)

1.5.2

Misc

  • Remove Python 3.7 support (#30963)

1.5.1

Misc

  • Bring back min-airflow-version for preinstalled providers (#31469)

1.5.0 (YANKED)

Warning

This release has been yanked with a reason: This version might cause unconstrained installation of old airflow version lead to Runtime Error.

Note

This release of provider is only available for Airflow 2.4+ as explained in the Apache Airflow providers support policy.

Features

  • Add conditional output processing in SQL operators (#31136)

Misc

  • Remove noisy log from SQL table check (#31037)

1.4.0

Features

  • Add option to show output of 'SQLExecuteQueryOperator' in the log (#29954)

Misc

  • Fix Python API docs formatting for Common SQL provider (#29863)

1.3.4

Bug Fixes

  • Do not process output when do_xcom_push=False  (#29599)

1.3.3

Bug Fixes

  • Handle non-compliant behaviour of Exasol cursor (#28744)

1.3.2

Bug Fixes

  • fIx isort problems introduced by recent isort release (#28434)

  • Fix template rendering for Common SQL operators (#28202)

  • Defer to hook setting for split_statements in SQLExecuteQueryOperator (#28635)

Misc

  • Clarify docstrings for updated DbApiHook (#27966)

1.3.1

This release fixes a few errors that were introduced in common.sql operator while refactoring common parts:

  • _process_output method in SQLExecuteQueryOperator has now consistent semantics and typing, it can also modify the returned (and stored in XCom) values in the operators that derive from the SQLExecuteQueryOperator).

  • descriptions of all returned results are stored as descriptions property in the DBApiHook

  • last description of the cursor whether to return single query results values are now exposed in DBApiHook via last_description property.

Lack of consistency in the operator caused 1.3.0 to be yanked - the 1.3.0 should not be used - if you have 1.3.0 installed, upgrade to 1.3.1.

Bug Fixes

  • Restore removed (but used) methods in common.sql (#27843)

  • Fix errors in Databricks SQL operator introduced when refactoring (#27854)

  • Bump common.sql provider to 1.3.1 (#27888)

  • Fixing the behaviours of SQL Hooks and Operators finally (#27912)

1.3.0 (YANKED)

Warning

This release has been yanked with a reason: Breaks Google 8.4.0 provider for SQLExecute

Note

This release of provider is only available for Airflow 2.3+ as explained in the Apache Airflow providers support policy.

Misc

  • Move min airflow version to 2.3.0 for all providers (#27196)

Features

  • Add SQLExecuteQueryOperator (#25717)

  • Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first (#26944)

  • DbApiHook consistent insert_rows logging (#26758)

Bug Fixes

  • Common sql bugfixes and improvements (#26761)

  • Use unused SQLCheckOperator.parameters in SQLCheckOperator.execute. (#27599)

1.2.0

Features

  • Make placeholder style configurable (#25939)

  • Better error message for pre-common-sql providers (#26051)

Bug Fixes

  • Fix (and test) SQLTableCheckOperator on postgresql (#25821)

  • Don't use Pandas for SQLTableCheckOperator (#25822)

  • Discard semicolon stripping in SQL hook (#25855)

1.1.0

Features

  • Improve taskflow type hints with ParamSpec (#25173)

  • Move all "old" SQL operators to common.sql providers (#25350)

  • Deprecate hql parameters and synchronize DBApiHook method APIs (#25299)

  • Unify DbApiHook.run() method with the methods which override it (#23971)

  • Common SQLCheckOperators Various Functionality Update (#25164)

Bug Fixes

  • Allow Legacy SqlSensor to use the common.sql providers (#25293)

  • Fix fetch_all_handler & db-api tests for it (#25430)

  • Align Common SQL provider logo location (#25538)

  • Fix SQL split string to include ';-less' statements (#25713)

1.0.0

Initial version of the provider. Adds SQLColumnCheckOperator and SQLTableCheckOperator. Moves DBApiHook, SQLSensor and ConnectorProtocol to the provider.

Was this entry helpful?