airflow.sensors.metastore_partition_sensor

Module Contents

class airflow.sensors.metastore_partition_sensor.MetastorePartitionSensor(table, partition_name, schema='default', mysql_conn_id='metastore_mysql', *args, **kwargs)

Bases:airflow.sensors.sql_sensor.SqlSensor

An alternative to the HivePartitionSensor that talk directly to the MySQL db. This was created as a result of observing sub optimal queries generated by the Metastore thrift service when hitting subpartitioned tables. The Thrift service’s queries were written in a way that wouldn’t leverage the indexes.

Parameters

• schema (str) – the schema

• table (str) – the table

• partition_name (str) – the partition name, as defined in the PARTITIONS table of the Metastore. Order of the fields does matter. Examples: ds=2016-01-01 or ds=2016-01-01/sub=foo for a sub partitioned table

• mysql_conn_id (str) – a reference to the MySQL conn_id for the metastore

template_fields = ['partition_name', 'table', 'schema']

ui_color = #8da7be

poke(self, context)