Google Cloud Storage to Amazon S3 transfer operator¶

Use the GCSToS3Operator transfer to copy the data from Google Cloud Storage to Amazon Simple Storage Service (S3).

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation of Airflow®
Setup Connection.

Operators¶

Google Cloud Storage to Amazon S3¶

To copy data from a Google Cloud Storage bucket to an Amazon S3 bucket you can use GCSToS3Operator

Example usage:

tests/system/amazon/aws/example_gcs_to_s3.py

gcs_to_s3 = GCSToS3Operator(
    task_id="gcs_to_s3",
    gcs_bucket=gcs_bucket,
    dest_s3_key=f"s3://{s3_bucket}/{s3_key}",
    replace=True,
    gcp_user_project=gcp_user_project,
)

Reference¶

Was this entry helpful?