airflow.providers.qubole.sensors.qubole

Module Contents

Classes

QuboleSensor

Base class for all Qubole Sensors.

QuboleFileSensor

Wait for a file or folder to be present in cloud storage.

QubolePartitionSensor

Wait for a Hive partition to show up in QHS (Qubole Hive Service).

class airflow.providers.qubole.sensors.qubole.QuboleSensor(*, data, qubole_conn_id='qubole_default', **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Base class for all Qubole Sensors.

template_fields: Sequence[str] = ('data', 'qubole_conn_id')[source]
template_ext: Sequence[str] = ('.txt',)[source]
poke(context)[source]

Override when deriving this class.

class airflow.providers.qubole.sensors.qubole.QuboleFileSensor(**kwargs)[source]

Bases: QuboleSensor

Wait for a file or folder to be present in cloud storage.

Check for file or folder presence via QDS APIs.

See also

For more information on how to use this sensor, take a look at the guide: File sensor

Parameters
  • qubole_conn_id – Connection id which consists of qds auth_token

  • data – a JSON object containing payload, whose presence needs to be checked Check this example for sample payload structure.

Note

Both data and qubole_conn_id fields support templating. You can also use .txt files for template-driven use cases.

class airflow.providers.qubole.sensors.qubole.QubolePartitionSensor(**kwargs)[source]

Bases: QuboleSensor

Wait for a Hive partition to show up in QHS (Qubole Hive Service).

Check for Hive partition presence via QDS APIs.

See also

For more information on how to use this sensor, take a look at the guide: Partition sensor

Parameters
  • qubole_conn_id – Connection id which consists of qds auth_token

  • data

    a JSON object containing payload, whose presence needs to be checked. Check this example for sample payload structure.

Note

Both data and qubole_conn_id fields support templating. You can also use .txt files for template-driven use cases.

Was this entry helpful?