airflow.providers.apache.hive.sensors.named_hive_partition

Module Contents

Classes

NamedHivePartitionSensor

Waits for a set of partitions to show up in Hive.
class airflow.providers.apache.hive.sensors.named_hive_partition.NamedHivePartitionSensor(*, partition_names, metastore_conn_id='metastore_default', poke_interval=60 * 3, hook=None, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Waits for a set of partitions to show up in Hive.

Parameters

  • partition_names (List[str]) -- List of fully qualified names of the partitions to wait for. A fully qualified name is of the form schema.table/pk1=pv1/pk2=pv2, for example, default.users/ds=2016-01-01. This is passed as is to the metastore Thrift client get_partitions_by_name method. Note that you cannot use logical or comparison operators as in HivePartitionSensor.

  • metastore_conn_id (str) -- Reference to the metastore thrift service connection id.

template_fields :Sequence[str] = ['partition_names'][source]
ui_color = #8d99ae[source]
poke_context_fields = ['partition_names', 'metastore_conn_id'][source]
static parse_partition_name(partition)[source]

Get schema, table, and partition info.

poke_partition(self, partition)[source]

Check for a named partition.

poke(self, context)[source]

Function that the sensors defined while deriving this class should override.

is_smart_sensor_compatible(self)[source]

Return if this operator can use smart service. Default False.

Was this entry helpful?