airflow.operators.mssql_to_hive
¶
Module Contents¶
-
class
airflow.operators.mssql_to_hive.
MsSqlToHiveTransfer
(sql, hive_table, create=True, recreate=False, partition=None, delimiter=chr(1), mssql_conn_id='mssql_default', hive_cli_conn_id='hive_cli_default', tblproperties=None, *args, **kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Moves data from Microsoft SQL Server to Hive. The operator runs your query against Microsoft SQL Server, stores the file locally before loading it into a Hive table. If the
create
orrecreate
arguments are set toTrue
, aCREATE TABLE
andDROP TABLE
statements are generated. Hive data types are inferred from the cursor’s metadata. Note that the table generated in Hive usesSTORED AS textfile
which isn’t the most efficient serialization format. If a large amount of data is loaded and/or if the table gets queried considerably, you may want to use this operator only to stage the data into a temporary table before loading it into its final destination using aHiveOperator
.- Parameters
sql (str) – SQL query to execute against the Microsoft SQL Server database. (templated)
hive_table (str) – target Hive table, use dot notation to target a specific database. (templated)
create (bool) – whether to create the table if it doesn’t exist
recreate (bool) – whether to drop and recreate the table at every execution
partition (dict) – target partition as a dict of partition columns and values. (templated)
delimiter (str) – field delimiter in the file
mssql_conn_id (str) – source Microsoft SQL Server connection
hive_conn_id (str) – destination hive connection
tblproperties (dict) – TBLPROPERTIES of the hive table being created