airflow.providers.qubole.sensors.qubole
¶
Module Contents¶
Classes¶
Base class for all Qubole Sensors. |
|
Wait for a file or folder to be present in cloud storage. |
|
Wait for a Hive partition to show up in QHS (Qubole Hive Service). |
- class airflow.providers.qubole.sensors.qubole.QuboleSensor(*, data, qubole_conn_id='qubole_default', **kwargs)[source]¶
Bases:
airflow.sensors.base.BaseSensorOperator
Base class for all Qubole Sensors.
- class airflow.providers.qubole.sensors.qubole.QuboleFileSensor(**kwargs)[source]¶
Bases:
QuboleSensor
Wait for a file or folder to be present in cloud storage.
Check for file or folder presence via QDS APIs.
See also
For more information on how to use this sensor, take a look at the guide: File sensor
- Parameters
qubole_conn_id – Connection id which consists of qds auth_token
data – a JSON object containing payload, whose presence needs to be checked Check this example for sample payload structure.
Note
Both
data
andqubole_conn_id
fields support templating. You can also use.txt
files for template-driven use cases.
- class airflow.providers.qubole.sensors.qubole.QubolePartitionSensor(**kwargs)[source]¶
Bases:
QuboleSensor
Wait for a Hive partition to show up in QHS (Qubole Hive Service).
Check for Hive partition presence via QDS APIs.
See also
For more information on how to use this sensor, take a look at the guide: Partition sensor
- Parameters
qubole_conn_id – Connection id which consists of qds auth_token
data –
a JSON object containing payload, whose presence needs to be checked. Check this example for sample payload structure.
Note
Both
data
andqubole_conn_id
fields support templating. You can also use.txt
files for template-driven use cases.