airflow.contrib.operators.hive_to_dynamodb
¶
Module Contents¶
-
class
airflow.contrib.operators.hive_to_dynamodb.
HiveToDynamoDBTransferOperator
(sql, table_name, table_keys, pre_process=None, pre_process_args=None, pre_process_kwargs=None, region_name=None, schema='default', hiveserver2_conn_id='hiveserver2_default', aws_conn_id='aws_default', *args, **kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Moves data from Hive to DynamoDB, note that for now the data is loaded into memory before being pushed to DynamoDB, so this operator should be used for smallish amount of data.
- Parameters
sql (str) – SQL query to execute against the hive database. (templated)
table_name (str) – target DynamoDB table
table_keys (list) – partition key and sort key
pre_process (function) – implement pre-processing of source data
pre_process_args (list) – list of pre_process function arguments
pre_process_kwargs (dict) – dict of pre_process function arguments
region_name (str) – aws region name (example: us-east-1)
schema (str) – hive database schema
hiveserver2_conn_id (str) – source hive connection
aws_conn_id (str) – aws connection