airflow.contrib.sensors.qubole_sensor

Module Contents

class airflow.contrib.sensors.qubole_sensor.QuboleSensor(data, qubole_conn_id='qubole_default', *args, **kwargs)[source]

Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator

Base class for all Qubole Sensors

template_fields = ['data', 'qubole_conn_id'][source]
template_ext = ['.txt'][source]
poke(self, context)[source]
class airflow.contrib.sensors.qubole_sensor.QuboleFileSensor(*args, **kwargs)[source]

Bases: airflow.contrib.sensors.qubole_sensor.QuboleSensor

Wait for a file or folder to be present in cloud storage and check for its presence via QDS APIs

Parameters

  • qubole_conn_id (str) – Connection id which consists of qds auth_token

  • data (a JSON object) – a JSON object containing payload, whose presence needs to be checked Check this example for sample payload structure.

Note

Both data and qubole_conn_id fields support templating. You can also use .txt files for template-driven use cases.

class airflow.contrib.sensors.qubole_sensor.QubolePartitionSensor(*args, **kwargs)[source]

Bases: airflow.contrib.sensors.qubole_sensor.QuboleSensor

Wait for a Hive partition to show up in QHS (Qubole Hive Service) and check for its presence via QDS APIs

Parameters

  • qubole_conn_id (str) – Connection id which consists of qds auth_token

  • data (a JSON object) –

    a JSON object containing payload, whose presence needs to be checked. Check this example for sample payload structure.

Note

Both data and qubole_conn_id fields support templating. You can also use .txt files for template-driven use cases.