airflow.providers.common.sql.operators.analytics¶

Classes¶

AnalyticsOperator

Operator to run queries on various datasource's stored in object stores like S3, GCS, Azure, etc.

Module Contents¶

class airflow.providers.common.sql.operators.analytics.AnalyticsOperator(datasource_configs, queries, max_rows_check=100, engine=None, result_output_format='tabulate', **kwargs)[source]¶

Bases: airflow.providers.common.compat.sdk.BaseOperator

Operator to run queries on various datasource’s stored in object stores like S3, GCS, Azure, etc.

Parameters:

datasource_configs (list[airflow.providers.common.sql.config.DataSourceConfig]) – List of datasource configurations to register.
queries (list[str]) – List of SQL queries to execute.
max_rows_check (int) – Maximum number of rows allowed in query results. Queries exceeding this will be skipped.
engine (airflow.providers.common.sql.datafusion.engine.DataFusionEngine | None) – Optional DataFusion engine instance.
result_output_format (Literal['tabulate', 'json']) – List of output formats for results. Supported: ‘tabulate’, ‘json’. Default is ‘tabulate’.

template_fields: collections.abc.Sequence[str] = ('datasource_configs', 'queries', 'max_rows_check', 'result_output_format')[source]¶

datasource_configs[source]¶

queries[source]¶

engine = None[source]¶

max_rows_check = 100[source]¶

result_output_format = 'tabulate'[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.