airflow.providers.google.cloud.transfers.mysql_to_gcs

MySQL to GCS operator.

Module Contents

class airflow.providers.google.cloud.transfers.mysql_to_gcs.MySQLToGCSOperator(*, mysql_conn_id='mysql_default', ensure_utc=False, **kwargs)[source]

Bases: airflow.providers.google.cloud.transfers.sql_to_gcs.BaseSQLToGCSOperator

Copy data from MySQL to Google Cloud Storage in JSON or CSV format.

Parameters

  • mysql_conn_id (str) -- Reference to a specific MySQL hook.

  • ensure_utc (bool) -- Ensure TIMESTAMP columns exported as UTC. If set to False, TIMESTAMP columns will be exported using the MySQL server's default timezone.

ui_color = #a0e08c[source]
type_map[source]
query(self)[source]

Queries mysql and returns a cursor to the results.

field_to_bigquery(self, field)[source]
convert_type(self, value, schema_type: str)[source]

Takes a value from MySQLdb, and converts it to a value that's safe for JSON/Google Cloud Storage/BigQuery.

  • Datetimes are converted to UTC seconds.

  • Decimals are converted to floats.

  • Dates are converted to ISO formatted string if given schema_type is DATE, or UTC seconds otherwise.

  • Binary type fields are converted to integer if given schema_type is INTEGER, or encoded with base64 otherwise. Imported BYTES data must be base64-encoded according to BigQuery documentation: https://cloud.google.com/bigquery/data-types

Parameters

  • value (Any) -- MySQLdb column value

  • schema_type (str) -- BigQuery data type

Was this entry helpful?