Google Cloud BigQuery Transfer Operator to Postgres

Google Cloud BigQuery is Google Cloud’s serverless data warehouse offering. PostgreSQL is an open-source relational database management system. This operator can be used to copy data from a BigQuery table to PostgreSQL.

Prerequisite Tasks

To use these operators, you must do a few things:

Operator

Copying data from BigQuery table to Postgres table is performed with the BigQueryToPostgresOperator operator.

Use Jinja templating with target_table_name, impersonation_chain to define values dynamically.

You may use the parameter selected_fields to limit the fields to be copied (all fields by default), as well as the parameter replace to overwrite the destination table instead of appending to it. For more information, please refer to the links above.

Transferring data

The following Operator copies data from a BigQuery table to PostgreSQL.

tests/system/providers/google/cloud/bigquery/example_bigquery_to_postgres.py[source]

bigquery_to_postgres = BigQueryToPostgresOperator(
    task_id="bigquery_to_postgres",
    dataset_table=f"{DATASET_NAME}.{TABLE}",
    target_table_name=destination_table,
    replace=False,
)

Reference

For further information, look at:

Was this entry helpful?