airflow.providers.apache.hive.sensors.named_hive_partition

Module Contents

Classes

NamedHivePartitionSensor

Waits for a set of partitions to show up in Hive.

class airflow.providers.apache.hive.sensors.named_hive_partition.NamedHivePartitionSensor(*, partition_names, metastore_conn_id='metastore_default', poke_interval=60 * 3, hook=None, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Waits for a set of partitions to show up in Hive.

Parameters
  • partition_names (list[str]) – List of fully qualified names of the partitions to wait for. A fully qualified name is of the form schema.table/pk1=pv1/pk2=pv2, for example, default.users/ds=2016-01-01. This is passed as is to the metastore Thrift client get_partitions_by_name method. Note that you cannot use logical or comparison operators as in HivePartitionSensor.

  • metastore_conn_id (str) – Reference to the metastore thrift service connection id.

template_fields :Sequence[str] = ['partition_names'][source]
ui_color = #8d99ae[source]
static parse_partition_name(partition)[source]

Get schema, table, and partition info.

poke_partition(partition)[source]

Check for a named partition.

poke(context)[source]

Function defined by the sensors while deriving this class should override.

Was this entry helpful?