tests.system.amazon.aws.example_hive_to_dynamodb

This DAG will not work unless you create an Amazon EMR cluster running Apache Hive and copy data into it following steps 1-4 (inclusive) here: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EMRforDynamoDB.Tutorial.html

Module Contents

Functions

create_dynamodb_table(table_name)

get_dynamodb_item_count(table_name)

A DynamoDB table has an ItemCount value, but it is only updated every six hours.

delete_dynamodb_table(table_name)

configure_hive_connection(connection_id, hostname)

Attributes

DAG_ID

HIVE_CONNECTION_ID_KEY

HIVE_HOSTNAME_KEY

sys_test_context_task

DYNAMODB_TABLE_HASH_KEY

HIVE_SQL

test_context

test_run

tests.system.amazon.aws.example_hive_to_dynamodb.DAG_ID = 'example_hive_to_dynamodb'[source]
tests.system.amazon.aws.example_hive_to_dynamodb.HIVE_CONNECTION_ID_KEY = 'HIVE_CONNECTION_ID'[source]
tests.system.amazon.aws.example_hive_to_dynamodb.HIVE_HOSTNAME_KEY = 'HIVE_HOSTNAME'[source]
tests.system.amazon.aws.example_hive_to_dynamodb.sys_test_context_task[source]
tests.system.amazon.aws.example_hive_to_dynamodb.DYNAMODB_TABLE_HASH_KEY = 'feature_id'[source]
tests.system.amazon.aws.example_hive_to_dynamodb.HIVE_SQL = 'SELECT feature_id, feature_name, feature_class, state_alpha FROM hive_features'[source]
tests.system.amazon.aws.example_hive_to_dynamodb.create_dynamodb_table(table_name)[source]
tests.system.amazon.aws.example_hive_to_dynamodb.get_dynamodb_item_count(table_name)[source]

A DynamoDB table has an ItemCount value, but it is only updated every six hours. To verify this DAG worked, we will scan the table and count the items manually.

tests.system.amazon.aws.example_hive_to_dynamodb.delete_dynamodb_table(table_name)[source]
tests.system.amazon.aws.example_hive_to_dynamodb.configure_hive_connection(connection_id, hostname)[source]
tests.system.amazon.aws.example_hive_to_dynamodb.test_context[source]
tests.system.amazon.aws.example_hive_to_dynamodb.test_run[source]

Was this entry helpful?