Transfer Data from Amazon S3 to Google Cloud Storage¶

The Google Cloud Storage (GCS) is used to store large data from various applications. This is also the same with Amazon Simple Storage Service. This page shows how to transfer data from Amazon S3 to GCS.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Select or create a Cloud Platform project using the Cloud Console.

Enable billing for your project, as described in the Google Cloud documentation.

Enable the API, as described in the Cloud Console documentation.
Install API libraries via pip.
pip install 'apache-airflow[google]'
Detailed information is available for Installation.
Setup a Google Cloud Connection.

Use the S3ToGCSOperator to transfer data from Amazon S3 to Google Cloud Storage.

There is a possibility to start S3ToGCSOperator asynchronously using deferrable mode. To do so just add parameter deferrable=True into the operator call. Under the hood it will delegate data transfer to Google Cloud Storage Transfer Service. By changing parameter poll_interval=10 you can control frequency of polling a transfer job status.

Reference¶

For further information, look at: