Amazon DynamoDB to S3 Transfer Operator¶

Use the DynamoDBToS3Operator transfer to copy the contents of an existing Amazon DynamoDB table to an existing Amazon Simple Storage Service (S3) bucket.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation
Setup Connection.

DynamoDB To S3 Operator¶

This operator replicates records from a DynamoDB table to a file in an S3 bucket. It scans a DynamoDB table and writes the received records to a file on the local filesystem. It flushes the file to S3 once the file size exceeds the file size limit specified by the user.

Users can also specify a filtering criteria using dynamodb_scan_kwargs to only replicate records that satisfy the criteria.

To get more information visit: DynamoDBToS3Operator

Example usage:

airflow/providers/amazon/aws/example_dags/example_dynamodb_to_s3.py[source]

backup_db = DynamoDBToS3Operator(
    task_id='backup_db',
    dynamodb_table_name=TABLE_NAME,
    s3_bucket_name=BUCKET_NAME,
    # Max output file size in bytes.  If the Table is too large, multiple files will be created.
    file_size=1000,
)

To parallelize the replication, users can create multiple DynamoDBToS3Operator tasks using the TotalSegments parameter. For instance to replicate with parallelism of 2, create two tasks:

airflow/providers/amazon/aws/example_dags/example_dynamodb_to_s3_segmented.py[source]

# Segmenting allows the transfer to be parallelized into {segment} number of parallel tasks.
backup_db_segment_1 = DynamoDBToS3Operator(
    task_id='backup-1',
    dynamodb_table_name=TABLE_NAME,
    s3_bucket_name=BUCKET_NAME,
    # Max output file size in bytes.  If the Table is too large, multiple files will be created.
    file_size=1000,
    dynamodb_scan_kwargs={
        "TotalSegments": 2,
        "Segment": 0,
    },
)

backup_db_segment_2 = DynamoDBToS3Operator(
    task_id="backup-2",
    dynamodb_table_name=TABLE_NAME,
    s3_bucket_name=BUCKET_NAME,
    # Max output file size in bytes.  If the Table is too large, multiple files will be created.
    file_size=1000,
    dynamodb_scan_kwargs={
        "TotalSegments": 2,
        "Segment": 1,
    },
)