airflow.contrib.operators.postgres_to_gcs_operator

Module Contents

airflow.contrib.operators.postgres_to_gcs_operator.PY3

class airflow.contrib.operators.postgres_to_gcs_operator.PostgresToGoogleCloudStorageOperator(sql, bucket, filename, schema_filename=None, approx_max_file_size_bytes=1900000000, postgres_conn_id='postgres_default', google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, parameters=None, *args, **kwargs)

Bases:airflow.models.BaseOperator

Copy data from Postgres to Google Cloud Storage in JSON format.

template_fields = ['sql', 'bucket', 'filename', 'schema_filename', 'parameters']

template_ext = ['.sql']

ui_color = #a0e08c

execute(self, context)

_query_postgres(self)

Queries Postgres and returns a cursor to the results.

_write_local_data_files(self, cursor)

Takes a cursor, and writes results to a local file.

Returns

A dictionary where keys are filenames to be used as object names in GCS, and values are file handles to local files that contain the data for the GCS objects.

_write_local_schema_file(self, cursor)

Takes a cursor, and writes the BigQuery schema for the results to a local file system.

Returns

A dictionary where key is a filename to be used as an object name in GCS, and values are file handles to local files that contains the BigQuery schema fields in .json format.

_upload_to_gcs(self, files_to_upload)

Upload all of the file splits (and optionally the schema .json file) to Google Cloud Storage.

classmethod convert_types(cls, value)

Takes a value from Postgres, and converts it to a value that’s safe for JSON/Google Cloud Storage/BigQuery. Dates are converted to UTC seconds. Decimals are converted to floats. Times are converted to seconds.

classmethod type_map(cls, postgres_type)

Helper function that maps from Postgres fields to BigQuery fields. Used when a schema_filename is set.