airflow.sensors.named_hive_partition_sensor

Module Contents

class airflow.sensors.named_hive_partition_sensor.NamedHivePartitionSensor(partition_names, metastore_conn_id='metastore_default', poke_interval=60 * 3, hook=None, *args, **kwargs)[source]

Bases:airflow.sensors.base_sensor_operator.BaseSensorOperator

Waits for a set of partitions to show up in Hive.

Parameters

  • partition_names (list[str]) – List of fully qualified names of the partitions to wait for. A fully qualified name is of the form schema.table/pk1=pv1/pk2=pv2, for example, default.users/ds=2016-01-01. This is passed as is to the metastore Thrift client get_partitions_by_name method. Note that you cannot use logical or comparison operators as in HivePartitionSensor.

  • metastore_conn_id (str) – reference to the metastore thrift service connection id

template_fields = ['partition_names'][source]
ui_color = #8d99ae[source]
static parse_partition_name(partition)[source]
poke_partition(self, partition)[source]
poke(self, context)[source]