Amazon S3 Glacier to GCS Transfer Operator¶

Amazon Glacier is a secure, durable, and extremely low-cost Amazon S3 cloud storage class for data archiving and long-term backup.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation
Setup Connection.

Glacier To GCS¶

To transfer data from an Amazon Glacier vault to Google Cloud Storage. use GlacierToGCSOperator

airflow/providers/amazon/aws/example_dags/example_glacier_to_gcs.py[source]

transfer_archive_to_gcs = GlacierToGCSOperator(
    task_id="transfer_archive_to_gcs",
    vault_name=VAULT_NAME,
    bucket_name=BUCKET_NAME,
    object_name=OBJECT_NAME,
    gzip=False,
    # Override to match your needs
    # If chunk size is bigger than actual file size
    # then whole file will be downloaded
    chunk_size=1024,
)

Note

Please be aware that GlacierToGCSOperator depends on available memory. Transferring large files may exhaust memory on the worker host.