Amazon S3 Glacier to GCS Transfer Operator¶
Amazon Glacier is a secure, durable, and extremely low-cost Amazon S3 cloud storage class for data archiving and long-term backup.
To use these operators, you must do a few things:
Glacier To GCS¶
To transfer data from an Amazon Glacier vault to Google Cloud Storage.
transfer_archive_to_gcs = GlacierToGCSOperator( task_id="transfer_archive_to_gcs", vault_name=VAULT_NAME, bucket_name=BUCKET_NAME, object_name=OBJECT_NAME, gzip=False, # Override to match your needs # If chunk size is bigger than actual file size # then whole file will be downloaded chunk_size=1024, )
Please be aware that GlacierToGCSOperator depends on available memory. Transferring large files may exhaust memory on the worker host.