airflow.contrib.operators.gcs_to_s3

Module Contents

class airflow.contrib.operators.gcs_to_s3.GoogleCloudStorageToS3Operator(bucket, prefix=None, delimiter=None, google_cloud_storage_conn_id='google_cloud_storage_default', delegate_to=None, dest_aws_conn_id=None, dest_s3_key=None, dest_verify=None, replace=False, *args, **kwargs)[source]

Bases:airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator

Synchronizes a Google Cloud Storage bucket with an S3 bucket.

Parameters

  • bucket (str) – The Google Cloud Storage bucket to find the objects. (templated)

  • prefix (str) – Prefix string which filters objects whose name begin with this prefix. (templated)

  • delimiter (str) – The delimiter by which you want to filter the objects. (templated) For e.g to lists the CSV files from in a directory in GCS you would use delimiter=’.csv’.

  • google_cloud_storage_conn_id (str) – The connection ID to use when connecting to Google Cloud Storage.

  • delegate_to (str) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • dest_aws_conn_id (str) – The destination S3 connection

  • dest_s3_key (str) – The base S3 key to be used to store the files. (templated)

  • dest_verify (bool or str) –

    Whether or not to verify SSL certificates for S3 connection. By default SSL certificates are verified. You can provide the following values:

    • False: do not validate SSL certificates. SSL will still be used

      (unless use_ssl is False), but SSL certificates will not be verified.

    • path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.

      You can specify this argument if you want to use a different CA cert bundle than the one used by botocore.

template_fields = ['bucket', 'prefix', 'delimiter', 'dest_s3_key'][source]
ui_color = #f0eee4[source]
execute(self, context)[source]