airflow.providers.qubole.sensors.qubole

Module Contents

Classes

QuboleSensor

Base class for all Qubole Sensors

QuboleFileSensor

Wait for a file or folder to be present in cloud storage

QubolePartitionSensor

Wait for a Hive partition to show up in QHS (Qubole Hive Service)
class airflow.providers.qubole.sensors.qubole.QuboleSensor(*, data, qubole_conn_id='qubole_default', **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Base class for all Qubole Sensors

template_fields :Sequence[str] = ['data', 'qubole_conn_id'][source]
template_ext :Sequence[str] = ['.txt'][source]
poke(self, context)[source]

Function that the sensors defined while deriving this class should override.

class airflow.providers.qubole.sensors.qubole.QuboleFileSensor(**kwargs)[source]

Bases: QuboleSensor

Wait for a file or folder to be present in cloud storage and check for its presence via QDS APIs

See also

For more information on how to use this sensor, take a look at the guide: File sensor

Parameters

  • qubole_conn_id -- Connection id which consists of qds auth_token

  • data -- a JSON object containing payload, whose presence needs to be checked Check this example for sample payload structure.

Note

Both data and qubole_conn_id fields support templating. You can also use .txt files for template-driven use cases.

class airflow.providers.qubole.sensors.qubole.QubolePartitionSensor(**kwargs)[source]

Bases: QuboleSensor

Wait for a Hive partition to show up in QHS (Qubole Hive Service) and check for its presence via QDS APIs

See also

For more information on how to use this sensor, take a look at the guide: Partition sensor

Parameters

  • qubole_conn_id -- Connection id which consists of qds auth_token

  • data --

    a JSON object containing payload, whose presence needs to be checked. Check this example for sample payload structure.

Note

Both data and qubole_conn_id fields support templating. You can also use .txt files for template-driven use cases.

Was this entry helpful?