airflow.providers.google.cloud.transfers.bigquery_to_sql
¶
Base operator for BigQuery to SQL operators.
Module Contents¶
Classes¶
Fetch data from a BigQuery table (alternatively fetch selected columns) and insert it into an SQL table. |
- class airflow.providers.google.cloud.transfers.bigquery_to_sql.BigQueryToSqlBaseOperator(*, dataset_table, target_table_name, selected_fields=None, gcp_conn_id='google_cloud_default', database=None, replace=False, batch_size=1000, location=None, impersonation_chain=None, dataset_id=None, table_id=None, **kwargs)[source]¶
Bases:
airflow.models.BaseOperator
Fetch data from a BigQuery table (alternatively fetch selected columns) and insert it into an SQL table.
This is a BaseOperator; an abstract class. Refer to children classes which are related to specific SQL databases (MySQL, MsSQL, Postgres…).
Note
If you pass fields to
selected_fields
which are in different order than the order of columns already in BQ table, the data will still be in the order of BQ table. For example if the BQ table has 3 columns as[A,B,C]
and you pass ‘B,A’ in theselected_fields
the data would still be of the form'A,B'
and passed through this form to the SQL database.- Parameters
dataset_table (str) – A dotted
<dataset>.<table>
: the big query table of origintarget_table_name (str | None) – target SQL table
selected_fields (list[str] | str | None) – List of fields to return (comma-separated). If unspecified, all fields are returned.
gcp_conn_id (str) – reference to a specific Google Cloud hook.
database (str | None) – name of database which overwrite defined one in connection
replace (bool) – Whether to replace instead of insert
batch_size (int) – The number of rows to take in each batch
location (str | None) – The location used for the operation.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).