Module Contents

class airflow.contrib.operators.postgres_to_gcs_operator.PostgresToGoogleCloudStorageOperator(sql, bucket, filename, schema_filename=None, approx_max_file_size_bytes=1900000000, postgres_conn_id='postgres_default', google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, parameters=None, *args, **kwargs)[source]


Copy data from Postgres to Google Cloud Storage in JSON format.

template_fields = ['sql', 'bucket', 'filename', 'schema_filename', 'parameters'][source]
template_ext = ['.sql'][source]
ui_color = #a0e08c[source]
execute(self, context)[source]

Queries Postgres and returns a cursor to the results.

_write_local_data_files(self, cursor)[source]

Takes a cursor, and writes results to a local file.


A dictionary where keys are filenames to be used as object names in GCS, and values are file handles to local files that contain the data for the GCS objects.

_write_local_schema_file(self, cursor)[source]

Takes a cursor, and writes the BigQuery schema for the results to a local file system.


A dictionary where key is a filename to be used as an object name in GCS, and values are file handles to local files that contains the BigQuery schema fields in .json format.

_upload_to_gcs(self, files_to_upload)[source]

Upload all of the file splits (and optionally the schema .json file) to Google Cloud Storage.

classmethod convert_types(cls, value)[source]

Takes a value from Postgres, and converts it to a value that’s safe for JSON/Google Cloud Storage/BigQuery. Dates are converted to UTC seconds. Decimals are converted to floats. Times are converted to seconds.

classmethod type_map(cls, postgres_type)[source]

Helper function that maps from Postgres fields to BigQuery fields. Used when a schema_filename is set.