Google Cloud BigQuery Transfer Operator to Google Cloud Storage¶

Google Cloud BigQuery is Google Cloud’s serverless data warehouse offering. Google Cloud Storage (GCS) is a managed service for storing unstructured data. This operator can be used to export data from BigQuery tables into files in a Cloud Storage bucket.

Prerequisite Tasks¶

Operator¶

File transfer from GCS to BigQuery is performed with the BigQueryToGCSOperator operator.

Use Jinja templating with source_project_dataset_table, destination_cloud_storage_uris, export_format, labels, impersonation_chain, job_id to define values dynamically.

You may define multiple destination URIs, as well as other settings such as compression and export_format. For more information, please refer to the links above.

Importing files¶

The following Operator imports one or more files from GCS into a BigQuery table.

tests/system/providers/google/cloud/bigquery/example_bigquery_to_gcs.py[source]

bigquery_to_gcs = BigQueryToGCSOperator(
    task_id="bigquery_to_gcs",
    source_project_dataset_table=f"{DATASET_NAME}.{TABLE}",
    destination_cloud_storage_uris=[f"gs://{BUCKET_NAME}/{BUCKET_FILE}"],
)

Reference¶

For further information, look at: