airflow.contrib.operators.postgres_to_gcs_operator
¶
Module Contents¶
-
class
airflow.contrib.operators.postgres_to_gcs_operator.
PostgresToGoogleCloudStorageOperator
(sql, bucket, filename, schema_filename=None, approx_max_file_size_bytes=1900000000, postgres_conn_id='postgres_default', google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, parameters=None, *args, **kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Copy data from Postgres to Google Cloud Storage in JSON format.
-
_write_local_data_files
(self, cursor)[source]¶ Takes a cursor, and writes results to a local file.
- Returns
A dictionary where keys are filenames to be used as object names in GCS, and values are file handles to local files that contain the data for the GCS objects.
-
_write_local_schema_file
(self, cursor)[source]¶ Takes a cursor, and writes the BigQuery schema for the results to a local file system.
- Returns
A dictionary where key is a filename to be used as an object name in GCS, and values are file handles to local files that contains the BigQuery schema fields in .json format.
-
_upload_to_gcs
(self, files_to_upload)[source]¶ Upload all of the file splits (and optionally the schema .json file) to Google Cloud Storage.
-