airflow.providers.papermill.hooks.kernel

Module Contents

Classes

KernelConnection

Class to represent kernel connection object.

KernelHook

The KernelHook can be used to interact with remote jupyter kernel.

RemoteKernelManager

Jupyter kernel manager that connects to a remote kernel.

RemoteKernelEngine

Papermill engine to use RemoteKernelManager to connect to remote kernel and execute notebook.

Functions

register_remote_kernel_engine()

Register RemoteKernelEngine papermill engine.

Attributes

JUPYTER_KERNEL_SHELL_PORT

JUPYTER_KERNEL_IOPUB_PORT

JUPYTER_KERNEL_STDIN_PORT

JUPYTER_KERNEL_CONTROL_PORT

JUPYTER_KERNEL_HB_PORT

REMOTE_KERNEL_ENGINE

airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_SHELL_PORT = 60316[source]
airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_IOPUB_PORT = 60317[source]
airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_STDIN_PORT = 60318[source]
airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_CONTROL_PORT = 60319[source]
airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_HB_PORT = 60320[source]
airflow.providers.papermill.hooks.kernel.REMOTE_KERNEL_ENGINE = 'remote_kernel_engine'[source]
class airflow.providers.papermill.hooks.kernel.KernelConnection[source]

Class to represent kernel connection object.

ip: str[source]
shell_port: int[source]
iopub_port: int[source]
stdin_port: int[source]
control_port: int[source]
hb_port: int[source]
session_key: str[source]
class airflow.providers.papermill.hooks.kernel.KernelHook(kernel_conn_id=default_conn_name, *args, **kwargs)[source]

Bases: airflow.hooks.base.BaseHook

The KernelHook can be used to interact with remote jupyter kernel.

Takes kernel host/ip from connection and refers to jupyter kernel ports and session_key

from extra field.

Parameters

kernel_conn_id (str) – connection that has kernel host/ip

conn_name_attr = 'kernel_conn_id'[source]
default_conn_name = 'jupyter_kernel_default'[source]
conn_type = 'jupyter_kernel'[source]
hook_name = 'Jupyter Kernel'[source]
get_conn()[source]

Return connection for the hook.

airflow.providers.papermill.hooks.kernel.register_remote_kernel_engine()[source]

Register RemoteKernelEngine papermill engine.

class airflow.providers.papermill.hooks.kernel.RemoteKernelManager(*args, **kwargs)[source]

Bases: jupyter_client.AsyncKernelManager

Jupyter kernel manager that connects to a remote kernel.

property has_kernel: bool[source]

Has a kernel process been started that we are actively managing.

session_key[source]
shutdown_kernel(now=False, restart=False)[source]
client(**kwargs)[source]

Create a client configured to connect to our kernel.

class airflow.providers.papermill.hooks.kernel.RemoteKernelEngine[source]

Bases: papermill.engines.NBClientEngine

Papermill engine to use RemoteKernelManager to connect to remote kernel and execute notebook.

classmethod execute_managed_notebook(nb_man, kernel_name, log_output=False, stdout_file=None, stderr_file=None, start_timeout=60, execution_timeout=None, **kwargs)[source]

Perform the actual execution of the parameterized notebook locally.

Was this entry helpful?