airflow.contrib.hooks.qubole_hook

Qubole hook

Module Contents

airflow.contrib.hooks.qubole_hook.COMMAND_CLASSES[source]
airflow.contrib.hooks.qubole_hook.POSITIONAL_ARGS[source]
airflow.contrib.hooks.qubole_hook.flatten_list(list_of_lists)[source]
Flatten the list
airflow.contrib.hooks.qubole_hook.filter_options(options)[source]
Remove options from the list
airflow.contrib.hooks.qubole_hook.get_options_list(command_class)[source]
Get options list
airflow.contrib.hooks.qubole_hook.build_command_args()[source]
Build Command argument from command and options
class airflow.contrib.hooks.qubole_hook.QuboleHook(*args, **kwargs)[source]

Bases: airflow.hooks.base_hook.BaseHook

Hook for Qubole communication

static handle_failure_retry(context)[source]

Handle retries in case of failures

execute(self, context)[source]

Execute call

kill(self, ti)[source]

Kill (cancel) a Qubole command :param ti: Task Instance of the dag, used to determine the Quboles command id :return: response from Qubole

get_results(self, ti=None, fp=None, inline=True, delim=None, fetch=True)[source]

Get results (or just s3 locations) of a command from Qubole and save into a file :param ti: Task Instance of the dag, used to determine the Quboles command id :param fp: Optional file pointer, will create one and return if None passed :param inline: True to download actual results, False to get s3 locations only :param delim: Replaces the CTL-A chars with the given delim, defaults to ‘,’ :param fetch: when inline is True, get results directly from s3 (if large) :return: file location containing actual results or s3 locations of results

get_log(self, ti)[source]

Get Logs of a command from Qubole :param ti: Task Instance of the dag, used to determine the Quboles command id :return: command log as text

get_jobs_id(self, ti)[source]

Get jobs associated with a Qubole commands :param ti: Task Instance of the dag, used to determine the Quboles command id :return: Job information associated with command

Get link to qubole command result page.

Parameters

  • operator – operator

  • dttm – datetime

Returns

url link

create_cmd_args(self, context)[source]

Creates command arguments

static _add_tags(tags, value)[source]

Was this entry helpful?