Amazon S3 to Amazon Redshift Transfer Operator¶

Use the S3ToRedshiftOperator transfer to copy the data from an Amazon Simple Storage Service (S3) file into an Amazon Redshift table.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation
Setup Connection.

Amazon S3 To Amazon Redshift¶

This operator loads data from Amazon S3 to an existing Amazon Redshift table.

To get more information about this operator visit: S3ToRedshiftOperator

Example usage:

airflow/providers/amazon/aws/example_dags/example_s3_to_redshift.py[source]

task_transfer_s3_to_redshift = S3ToRedshiftOperator(
    s3_bucket=S3_BUCKET_NAME,
    s3_key=S3_KEY,
    schema='PUBLIC',
    table=REDSHIFT_TABLE,
    copy_options=['csv'],
    task_id='transfer_s3_to_redshift',
)

You can find more information to the COPY command used here.

Amazon S3 to Amazon Redshift Transfer Operator¶

Prerequisite Tasks¶

Amazon S3 To Amazon Redshift¶

Reference¶