airflow.providers.apache.hive.sensors.hive_partition

Module Contents

Classes

HivePartitionSensor

Waits for a partition to show up in Hive.
class airflow.providers.apache.hive.sensors.hive_partition.HivePartitionSensor(*, table: str, partition: Optional[str] = "ds='{{ ds }}'", metastore_conn_id: str = 'metastore_default', schema: str = 'default', poke_interval: int = 60 * 3, **kwargs: Any)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Waits for a partition to show up in Hive.

Note: Because partition supports general logical operators, it can be inefficient. Consider using NamedHivePartitionSensor instead if you don't need the full flexibility of HivePartitionSensor.

Parameters

  • table (str) -- The name of the table to wait for, supports the dot notation (my_database.my_table)

  • partition (str) -- The partition clause to wait for. This is passed as is to the metastore Thrift client get_partitions_by_filter method, and apparently supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01"

  • metastore_conn_id (str) -- reference to the :ref: metastore thrift service connection id <howto/connection:hive_metastore>

template_fields :Sequence[str] = ['schema', 'table', 'partition'][source]
ui_color = #C5CAE9[source]
poke(self, context: airflow.utils.context.Context) → bool[source]

Function that the sensors defined while deriving this class should override.

Was this entry helpful?