airflow.providers.apache.hive.sensors.metastore_partition
¶
Module Contents¶
-
class
airflow.providers.apache.hive.sensors.metastore_partition.
MetastorePartitionSensor
(*, table: str, partition_name: str, schema: str = 'default', mysql_conn_id: str = 'metastore_mysql', **kwargs)[source]¶ Bases:
airflow.sensors.sql.SqlSensor
An alternative to the HivePartitionSensor that talk directly to the MySQL db. This was created as a result of observing sub optimal queries generated by the Metastore thrift service when hitting subpartitioned tables. The Thrift service's queries were written in a way that wouldn't leverage the indexes.
- Parameters
schema (str) -- the schema
table (str) -- the table
partition_name (str) -- the partition name, as defined in the PARTITIONS table of the Metastore. Order of the fields does matter. Examples:
ds=2016-01-01
ords=2016-01-01/sub=foo
for a sub partitioned tablemysql_conn_id (str) -- a reference to the MySQL conn_id for the metastore