airflow.providers.google.cloud.transfers.bigquery_to_sql

Base operator for BigQuery to SQL operators.

Module Contents

Classes

BigQueryToSqlBaseOperator

Fetches the data from a BigQuery table (alternatively fetch data for selected columns)
class airflow.providers.google.cloud.transfers.bigquery_to_sql.BigQueryToSqlBaseOperator(*, dataset_table, target_table_name, selected_fields=None, gcp_conn_id='google_cloud_default', database=None, replace=False, batch_size=1000, location=None, impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Fetches the data from a BigQuery table (alternatively fetch data for selected columns) and insert that data into a SQL table. This is a BaseOperator; an abstract class. Refer to children classes which are related to specific SQL databases (MySQL, MsSQL, Postgres…).

Note

If you pass fields to selected_fields which are in different order than the order of columns already in BQ table, the data will still be in the order of BQ table. For example if the BQ table has 3 columns as [A,B,C] and you pass ‘B,A’ in the selected_fields the data would still be of the form 'A,B' and passed through this form to the SQL database.

Parameters

  • dataset_table (str) – A dotted <dataset>.<table>: the big query table of origin

  • target_table_name (str | None) – target SQL table

  • selected_fields (list[str] | str | None) – List of fields to return (comma-separated). If unspecified, all fields are returned.

  • gcp_conn_id (str) – reference to a specific Google Cloud hook.

  • database (str | None) – name of database which overwrite defined one in connection

  • replace (bool) – Whether to replace instead of insert

  • batch_size (int) – The number of rows to take in each batch

  • location (str | None) – The location used for the operation.

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: Sequence[str] = ('target_table_name', 'impersonation_chain')[source]
abstract get_sql_hook()[source]

Return a concrete SQL Hook (a PostgresHook for instance).

This function persists the connection to the SQL provider.

execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Was this entry helpful?