airflow.contrib.hooks.qubole_hook
¶
Qubole hook
Module Contents¶
-
airflow.contrib.hooks.qubole_hook.
build_command_args
()[source]¶ -
Build Command argument from command and options
-
class
airflow.contrib.hooks.qubole_hook.
QuboleHook
(*args, **kwargs)[source]¶ Bases:
airflow.hooks.base_hook.BaseHook
Hook for Qubole communication
-
kill
(self, ti)[source]¶ Kill (cancel) a Qubole command
- Parameters
ti – Task Instance of the dag, used to determine the Quboles command id
- Returns
response from Qubole
-
get_results
(self, ti=None, fp=None, inline=True, delim=None, fetch=True)[source]¶ Get results (or just s3 locations) of a command from Qubole and save into a file
- Parameters
ti – Task Instance of the dag, used to determine the Quboles command id
fp – Optional file pointer, will create one and return if None passed
inline – True to download actual results, False to get s3 locations only
delim – Replaces the CTL-A chars with the given delim, defaults to ‘,’
fetch – when inline is True, get results directly from s3 (if large)
- Returns
file location containing actual results or s3 locations of results
-
get_log
(self, ti)[source]¶ Get Logs of a command from Qubole
- Parameters
ti – Task Instance of the dag, used to determine the Quboles command id
- Returns
command log as text
-
get_jobs_id
(self, ti)[source]¶ Get jobs associated with a Qubole commands
- Parameters
ti – Task Instance of the dag, used to determine the Quboles command id
- Returns
Job information associated with command
-