airflow.providers.apache.hive.sensors.hive_partition

Module Contents

Classes

HivePartitionSensor

Waits for a partition to show up in Hive.
class airflow.providers.apache.hive.sensors.hive_partition.HivePartitionSensor(*, table, partition="ds='{{ ds }}'", metastore_conn_id='metastore_default', schema='default', poke_interval=60 * 3, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Waits for a partition to show up in Hive.

Note: Because partition supports general logical operators, it can be inefficient. Consider using NamedHivePartitionSensor instead if you don't need the full flexibility of HivePartitionSensor.

Parameters

  • table (str) -- The name of the table to wait for, supports the dot notation (my_database.my_table)

  • partition (Optional[str]) -- The partition clause to wait for. This is passed as is to the metastore Thrift client get_partitions_by_filter method, and apparently supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01"

  • metastore_conn_id (str) -- reference to the :ref: metastore thrift service connection id <howto/connection:hive_metastore>

template_fields :Sequence[str] = ['schema', 'table', 'partition'][source]
ui_color = #C5CAE9[source]
poke(self, context)[source]

Function that the sensors defined while deriving this class should override.

Was this entry helpful?