Amazon Redshift Data Operators¶

Use the RedshiftDataOperator to execute statements against an Amazon Redshift cluster.

This differs from RedshiftSQLOperator in that it allows users to query and retrieve data via the AWS API and avoid the necessity of a Postgres connection.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation
Setup Connection.

Amazon Redshift Data¶

Execute a statement on an Amazon Redshift Cluster¶

This is a basic example DAG for using RedshiftDataOperator to execute statements against an Amazon Redshift cluster.

airflow/providers/amazon/aws/example_dags/example_redshift_data_execute_sql.py[source]

task_query = RedshiftDataOperator(
    task_id='redshift_query',
    cluster_identifier=REDSHIFT_CLUSTER_IDENTIFIER,
    database=REDSHIFT_DATABASE,
    db_user=REDSHIFT_DATABASE_USER,
    sql=REDSHIFT_QUERY,
    poll_interval=POLL_INTERVAL,
    await_result=True,
)

Reference¶

AWS boto3 Library Documentation for Amazon Redshift Data