airflow.providers.google.cloud.transfers.bigquery_to_postgres¶
This module contains Google BigQuery to PostgreSQL operator.
Module Contents¶
Classes¶
Fetch data from a BigQuery table (alternatively fetch selected columns) and insert into PostgreSQL table. |
- class airflow.providers.google.cloud.transfers.bigquery_to_postgres.BigQueryToPostgresOperator(*, target_table_name, postgres_conn_id='postgres_default', replace=False, selected_fields=None, replace_index=None, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.transfers.bigquery_to_sql.BigQueryToSqlBaseOperatorFetch data from a BigQuery table (alternatively fetch selected columns) and insert into PostgreSQL table.
Due to constraints of the PostgreSQL’s ON CONFLICT clause both selected_fields and replace_index parameters need to be specified when using the operator with parameter replace=True. In effect this means that in order to run this operator with replace=True your target table MUST already have a unique index column / columns, otherwise the INSERT command will fail with an error. See more at https://www.postgresql.org/docs/current/sql-insert.html.
Please note that currently most of the clauses that can be used with PostgreSQL’s INSERT command, such as ON CONSTRAINT, WHERE, DEFAULT, etc., are not supported by this operator. If you need the clauses for your queries, SQLExecuteQueryOperator will be a more suitable option.
See also
For more information on how to use this operator, take a look at the guide: Operator
- Parameters
target_table_name (str) – target Postgres table (templated)
postgres_conn_id (str) – Reference to postgres connection id.
replace (bool) – Whether to replace instead of insert
selected_fields (list[str] | str | None) – List of fields to return (comma-separated). If unspecified, all fields are returned. Must be specified if replace is True
replace_index (list[str] | str | None) – the column or list of column names to act as index for the ON CONFLICT clause. Must be specified if replace is True