airflow.providers.qubole.hooks.qubole

Qubole hook

Module Contents

airflow.providers.qubole.hooks.qubole.log[source]
airflow.providers.qubole.hooks.qubole.COMMAND_CLASSES[source]
airflow.providers.qubole.hooks.qubole.POSITIONAL_ARGS[source]
airflow.providers.qubole.hooks.qubole.flatten_list(list_of_lists) → list[source]
Flatten the list
airflow.providers.qubole.hooks.qubole.filter_options(options: list) → list[source]
Remove options from the list
airflow.providers.qubole.hooks.qubole.get_options_list(command_class) → list[source]
Get options list
airflow.providers.qubole.hooks.qubole.build_command_args() → Tuple[Dict[str, list], list][source]
Build Command argument from command and options
class airflow.providers.qubole.hooks.qubole.QuboleHook(*args, **kwargs)[source]

Bases: airflow.hooks.base.BaseHook

Hook for Qubole communication

conn_name_attr = qubole_conn_id[source]
default_conn_name = qubole_default[source]
conn_type = qubole[source]
hook_name = Qubole[source]
static get_ui_field_behaviour()[source]

Returns custom field behaviour

static handle_failure_retry(context)[source]

Handle retries in case of failures

execute(self, context)[source]

Execute call

kill(self, ti)[source]

Kill (cancel) a Qubole command

Parameters

ti -- Task Instance of the dag, used to determine the Quboles command id

Returns

response from Qubole

get_results(self, ti=None, fp=None, inline: bool = True, delim=None, fetch: bool = True, include_headers: bool = False)[source]

Get results (or just s3 locations) of a command from Qubole and save into a file

Parameters

  • ti -- Task Instance of the dag, used to determine the Quboles command id

  • fp -- Optional file pointer, will create one and return if None passed

  • inline -- True to download actual results, False to get s3 locations only

  • delim -- Replaces the CTL-A chars with the given delim, defaults to ','

  • fetch -- when inline is True, get results directly from s3 (if large)

Returns

file location containing actual results or s3 locations of results

get_log(self, ti)[source]

Get Logs of a command from Qubole

Parameters

ti -- Task Instance of the dag, used to determine the Quboles command id

Returns

command log as text

get_jobs_id(self, ti)[source]

Get jobs associated with a Qubole commands

Parameters

ti -- Task Instance of the dag, used to determine the Quboles command id

Returns

Job information associated with command

create_cmd_args(self, context)[source]

Creates command arguments

Was this entry helpful?