Apache Hive to Amazon DynamoDB Transfer Operator¶

Use the HiveToDynamoDBOperator transfer to copy the contents of an existing Apache Hive table to an existing Amazon DynamoDB table.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation
Setup Connection.

Hive to DynamoDB Operator¶

This operator replicates records from a Hive table to a DynamoDB table. The user must specify an HQL query to use as filtering criteria.

To get more information visit: HiveToDynamoDBOperator

Example usage:

airflow/providers/amazon/aws/example_dags/example_hive_to_dynamodb.py[source]

backup_to_dynamodb = HiveToDynamoDBOperator(
    task_id='backup_to_dynamodb',
    hiveserver2_conn_id=HIVE_CONNECTION_ID,
    sql=HIVE_SQL,
    table_name=DYNAMODB_TABLE_NAME,
    table_keys=[DYNAMODB_TABLE_HASH_KEY],
)

Reference¶

For further information, look at: