airflow.providers.amazon.aws.transfers.s3_to_redshift

Module Contents

class airflow.providers.amazon.aws.transfers.s3_to_redshift.S3ToRedshiftOperator(*, schema: str, table: str, s3_bucket: str, s3_key: str, redshift_conn_id: str = 'redshift_default', aws_conn_id: str = 'aws_default', verify: Optional[Union[bool, str]] = None, column_list: Optional[List[str]] = None, copy_options: Optional[List] = None, autocommit: bool = False, truncate_table: bool = False, **kwargs)[source]

Bases: airflow.models.BaseOperator

Executes an COPY command to load files from s3 to Redshift

See also

For more information on how to use this operator, take a look at the guide: S3 To Redshift Transfer Operator

Parameters

  • schema (str) -- reference to a specific schema in redshift database

  • table (str) -- reference to a specific table in redshift database

  • s3_bucket (str) -- reference to a specific S3 bucket

  • s3_key (str) -- reference to a specific S3 key

  • redshift_conn_id (str) -- reference to a specific redshift database

  • aws_conn_id (str) -- reference to a specific S3 connection If the AWS connection contains 'aws_iam_role' in extras the operator will use AWS STS credentials with a token https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-authorization.html#copy-credentials

  • verify (bool or str) --

    Whether or not to verify SSL certificates for S3 connection. By default SSL certificates are verified. You can provide the following values:

    • False: do not validate SSL certificates. SSL will still be used

      (unless use_ssl is False), but SSL certificates will not be verified.

    • path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.

      You can specify this argument if you want to use a different CA cert bundle than the one used by botocore.

  • column_list (List[str]) -- list of column names to load

  • copy_options (list) -- reference to a list of COPY options

  • truncate_table (bool) -- whether or not to truncate the destination table before the copy

template_fields = ['s3_bucket', 's3_key', 'schema', 'table', 'column_list', 'copy_options'][source]
template_ext = [][source]
ui_color = #99e699[source]
execute(self, context)[source]

Was this entry helpful?