Apache Hive to Amazon DynamoDB¶

Use the HiveToDynamoDBOperator transfer to copy the contents of an existing Apache Hive table to an existing Amazon DynamoDB table.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation of Apache Airflow®
Setup Connection.

Operators¶

Apache Hive to Amazon DynamoDB transfer operator¶

This operator replicates records from an Apache Hive table to an Amazon DynamoDB table. The user must specify an HQL query to use as filtering criteria.

To get more information visit: HiveToDynamoDBOperator

Example usage:

tests/system/providers/amazon/aws/example_hive_to_dynamodb.py[source]

backup_to_dynamodb = HiveToDynamoDBOperator(
    task_id="backup_to_dynamodb",
    hiveserver2_conn_id=hive_connection_id,
    sql=HIVE_SQL,
    table_name=dynamodb_table_name,
    table_keys=[DYNAMODB_TABLE_HASH_KEY],
)

Apache Hive to Amazon DynamoDB¶

Prerequisite Tasks¶

Operators¶

Apache Hive to Amazon DynamoDB transfer operator¶

Reference¶