`airflow.providers.amazon.aws.transfers.s3_to_sql`¶

Module Contents¶

Classes¶

S3ToSqlOperator

Load Data from S3 into a SQL Database.

class airflow.providers.amazon.aws.transfers.s3_to_sql.S3ToSqlOperator(*, s3_key, s3_bucket, table, parser, column_list=None, commit_every=1000, schema=None, sql_conn_id='sql_default', aws_conn_id='aws_default', **kwargs)[source]¶

Bases: airflow.models.BaseOperator

Load Data from S3 into a SQL Database.

You need to provide a parser function that takes a filename as an input and returns an iterable of rows

See also

For more information on how to use this operator, take a look at the guide: Amazon S3 To SQL Transfer Operator

Parameters

schema (str | None) – reference to a specific schema in SQL database
table (str) – reference to a specific table in SQL database
s3_bucket (str) – reference to a specific S3 bucket
s3_key (str) – reference to a specific S3 key
sql_conn_id (str) – reference to a specific SQL database. Must be of type DBApiHook
aws_conn_id (str) – reference to a specific S3 / AWS connection
column_list (list[str] | None) – list of column names to use in the insert SQL.
commit_every (int) – The maximum number of rows to insert in one transaction. Set to 0 to insert all rows in one transaction.
parser (Callable[[str], Iterable[Iterable]]) –
parser function that takes a filepath as input and returns an iterable. e.g. to use a CSV parser that yields rows line-by-line, pass the following function:
```
def parse_csv(filepath):
    import csv

    with open(filepath, newline="") as file:
        yield from csv.reader(file)
```

template_fields: Sequence[str] = ('s3_bucket', 's3_key', 'schema', 'table', 'column_list', 'sql_conn_id')[source]¶

template_ext: Sequence[str] = ()[source]¶

ui_color = '#f4a460'[source]¶

execute(context)[source]¶

This is the main method to derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

db_hook()[source]¶

airflow.providers.amazon.aws.transfers.s3_to_sql¶

Module Contents¶

Classes¶

`airflow.providers.amazon.aws.transfers.s3_to_sql`¶