airflow.providers.elasticsearch.utils.sql¶

Attributes¶

log

Functions¶

`read_sql_to_polars`(client, query[, params, ...])	Execute an Elasticsearch SQL query and return results as a Polars DataFrame.
`read_sql_to_polars_by_chunks`(client, query[, params, ...])	Execute an Elasticsearch SQL query and return results as chunked Polars DataFrames.

Module Contents¶

airflow.providers.elasticsearch.utils.sql.log[source]¶

airflow.providers.elasticsearch.utils.sql.read_sql_to_polars(client, query, params=None, fetch_size=1000, max_rows=None)[source]¶

Execute an Elasticsearch SQL query and return results as a Polars DataFrame.

This uses Elasticsearch SQL cursor-based pagination instead of DB-API, as Elasticsearch does not provide a fully compliant DB-API interface.

Parameters:

client (elasticsearch.Elasticsearch) – Elasticsearch client
query (str) – SQL query string
params (collections.abc.Mapping[str, Any] | collections.abc.Iterable | None) – Optional query parameters
fetch_size (int) – Number of rows per batch
max_rows (int | None) – Optional limit on total rows fetched

airflow.providers.elasticsearch.utils.sql.read_sql_to_polars_by_chunks(client, query, params=None, fetch_size=1000, chunksize=1000)[source]¶

Execute an Elasticsearch SQL query and return results as chunked Polars DataFrames.

This uses Elasticsearch SQL cursor-based pagination instead of DB-API, as Elasticsearch does not provide a fully compliant DB-API interface.

Parameters:

client (elasticsearch.Elasticsearch) – Elasticsearch client
query (str) – SQL query string
params (collections.abc.Mapping[str, Any] | collections.abc.Iterable | None) – Optional query parameters
fetch_size (int) – Number of rows fetched per Elasticsearch request
chunksize (int) – Maximum number of rows per yielded DataFrame

Returns:

Generator of Polars DataFrames

Return type:

collections.abc.Generator[polars.DataFrame, None, None]