airflow.providers.apache.hive.sensors.metastore_partition

Module Contents

Classes

MetastorePartitionSensor

An alternative to the HivePartitionSensor that talk directly to the
class airflow.providers.apache.hive.sensors.metastore_partition.MetastorePartitionSensor(*, table, partition_name, schema='default', mysql_conn_id='metastore_mysql', **kwargs)[source]

Bases: airflow.sensors.sql.SqlSensor

An alternative to the HivePartitionSensor that talk directly to the MySQL db. This was created as a result of observing sub optimal queries generated by the Metastore thrift service when hitting subpartitioned tables. The Thrift service's queries were written in a way that wouldn't leverage the indexes.

Parameters

  • schema (str) -- the schema

  • table (str) -- the table

  • partition_name (str) -- the partition name, as defined in the PARTITIONS table of the Metastore. Order of the fields does matter. Examples: ds=2016-01-01 or ds=2016-01-01/sub=foo for a sub partitioned table

  • mysql_conn_id (str) -- a reference to the MySQL conn_id for the metastore

template_fields :Sequence[str] = ['partition_name', 'table', 'schema'][source]
ui_color = #8da7be[source]
poke_context_fields = ['partition_name', 'table', 'schema', 'mysql_conn_id'][source]
poke(self, context)[source]

Function that the sensors defined while deriving this class should override.

Was this entry helpful?