airflow.providers.google.cloud.transfers.bigquery_to_mysql

This module contains Google BigQuery to MySQL operator.

Module Contents

Classes

BigQueryToMySqlOperator

Fetches the data from a BigQuery table (alternatively fetch data for selected columns)
class airflow.providers.google.cloud.transfers.bigquery_to_mysql.BigQueryToMySqlOperator(*, dataset_table, mysql_table, selected_fields=None, gcp_conn_id='google_cloud_default', mysql_conn_id='mysql_default', database=None, delegate_to=None, replace=False, batch_size=1000, location=None, impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Fetches the data from a BigQuery table (alternatively fetch data for selected columns) and insert that data into a MySQL table.

Note

If you pass fields to selected_fields which are in different order than the order of columns already in BQ table, the data will still be in the order of BQ table. For example if the BQ table has 3 columns as [A,B,C] and you pass 'B,A' in the selected_fields the data would still be of the form 'A,B' and passed through this form to MySQL

Example:

transfer_data = BigQueryToMySqlOperator(
     task_id='task_id',
     dataset_table='origin_bq_table',
     mysql_table='dest_table_name',
     replace=True,
 )
Parameters

  • dataset_table (str) -- A dotted <dataset>.<table>: the big query table of origin

  • selected_fields (Optional[Union[List[str], str]]) -- List of fields to return (comma-separated). If unspecified, all fields are returned.

  • gcp_conn_id (str) -- reference to a specific Google Cloud hook.

  • delegate_to (Optional[str]) -- The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • mysql_conn_id (str) -- Reference to mysql connection id.

  • database (Optional[str]) -- name of database which overwrite defined one in connection

  • replace (bool) -- Whether to replace instead of insert

  • batch_size (int) -- The number of rows to take in each batch

  • location (Optional[str]) -- The location used for the operation.

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) -- Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['dataset_id', 'table_id', 'mysql_table', 'impersonation_chain'][source]
execute(self, context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Was this entry helpful?