airflow.providers.apache.hive.sensors.named_hive_partition

Module Contents

Classes

NamedHivePartitionSensor

Waits for a set of partitions to show up in Hive.
class airflow.providers.apache.hive.sensors.named_hive_partition.NamedHivePartitionSensor(*, partition_names, metastore_conn_id='metastore_default', poke_interval=60 * 3, hook=None, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Waits for a set of partitions to show up in Hive.

Parameters

  • partition_names (list[str]) – List of fully qualified names of the partitions to wait for. A fully qualified name is of the form schema.table/pk1=pv1/pk2=pv2, for example, default.users/ds=2016-01-01. This is passed as is to the metastore Thrift client get_partitions_by_name method. Note that you cannot use logical or comparison operators as in HivePartitionSensor.

  • metastore_conn_id (str) – Reference to the metastore thrift service connection id.

template_fields: Sequence[str] = ('partition_names',)[source]
ui_color = '#8d99ae'[source]
static parse_partition_name(partition)[source]

Get schema, table, and partition info.

poke_partition(partition)[source]

Check for a named partition.

poke(context)[source]

Override when deriving this class.

Was this entry helpful?