airflow.providers.google.cloud.transfers.postgres_to_gcs

PostgreSQL to GCS operator.

Module Contents

Classes

PostgresToGCSOperator

Copy data from Postgres to Google Cloud Storage in JSON, CSV or Parquet format.
class airflow.providers.google.cloud.transfers.postgres_to_gcs.PostgresToGCSOperator(*, postgres_conn_id='postgres_default', use_server_side_cursor=False, cursor_itersize=2000, **kwargs)[source]

Bases: airflow.providers.google.cloud.transfers.sql_to_gcs.BaseSQLToGCSOperator

Copy data from Postgres to Google Cloud Storage in JSON, CSV or Parquet format.

See also

For more information on how to use this operator, take a look at the guide: PostgresToGCSOperator

Parameters

  • postgres_conn_id – Reference to a specific Postgres hook.

  • use_server_side_cursor – If server-side cursor should be used for querying postgres. For detailed info, check https://www.psycopg.org/docs/usage.html#server-side-cursors

  • cursor_itersize – How many records are fetched at a time in case of server-side cursor.

ui_color = '#a0e08c'[source]
type_map[source]
query()[source]

Queries Postgres and returns a cursor to the results.

field_to_bigquery(field)[source]

Convert a DBAPI field to BigQuery schema format.

convert_type(value, schema_type, stringify_dict=True)[source]

Take a value from Postgres and convert it to a value safe for JSON/Google Cloud Storage/BigQuery.

Timezone aware Datetime are converted to UTC seconds. Unaware Datetime, Date and Time are converted to ISO formatted strings. Decimals are converted to floats.

Parameters

  • value – Postgres column value.

  • schema_type – BigQuery data type.

  • stringify_dict – Specify whether to convert dict to string.

Was this entry helpful?