airflow.sensors.hive_partition_sensor

Module Contents

class airflow.sensors.hive_partition_sensor.HivePartitionSensor(table, partition="ds='{{ ds }}'", metastore_conn_id='metastore_default', schema='default', poke_interval=60 * 3, *args, **kwargs)[source]

Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator

Waits for a partition to show up in Hive.

Note: Because partition supports general logical operators, it can be inefficient. Consider using NamedHivePartitionSensor instead if you don’t need the full flexibility of HivePartitionSensor.

Parameters

  • table (str) – The name of the table to wait for, supports the dot notation (my_database.my_table)

  • partition (str) – The partition clause to wait for. This is passed as is to the metastore Thrift client get_partitions_by_filter method, and apparently supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01"

  • metastore_conn_id (str) – reference to the metastore thrift service connection id

template_fields = ['schema', 'table', 'partition'][source]
ui_color = #C5CAE9[source]
poke(self, context)[source]

Was this entry helpful?