airflow.providers.apache.hive.sensors.named_hive_partition

Module Contents

class airflow.providers.apache.hive.sensors.named_hive_partition.NamedHivePartitionSensor(*, partition_names: List[str], metastore_conn_id: str = 'metastore_default', poke_interval: int = 60 * 3, hook: Any = None, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Waits for a set of partitions to show up in Hive.

Parameters

  • partition_names (list[str]) -- List of fully qualified names of the partitions to wait for. A fully qualified name is of the form schema.table/pk1=pv1/pk2=pv2, for example, default.users/ds=2016-01-01. This is passed as is to the metastore Thrift client get_partitions_by_name method. Note that you cannot use logical or comparison operators as in HivePartitionSensor.

  • metastore_conn_id (str) -- Reference to the metastore thrift service connection id.

template_fields = ['partition_names'][source]
ui_color = #8d99ae[source]
poke_context_fields = ['partition_names', 'metastore_conn_id'][source]
static parse_partition_name(partition: str)[source]

Get schema, table, and partition info.

poke_partition(self, partition: str)[source]

Check for a named partition.

poke(self, context: Dict[str, Any])[source]
is_smart_sensor_compatible(self)[source]

Was this entry helpful?