airflow.providers.papermill.hooks.kernel¶

Attributes¶

`JUPYTER_KERNEL_SHELL_PORT`
`JUPYTER_KERNEL_IOPUB_PORT`
`JUPYTER_KERNEL_STDIN_PORT`
`JUPYTER_KERNEL_CONTROL_PORT`
`JUPYTER_KERNEL_HB_PORT`
`REMOTE_KERNEL_ENGINE`

Classes¶

`KernelConnection`	Class to represent kernel connection object.
`KernelHook`	The KernelHook can be used to interact with remote jupyter kernel.
`RemoteKernelManager`	Jupyter kernel manager that connects to a remote kernel.
`RemoteKernelEngine`	Papermill engine to use `RemoteKernelManager` to connect to remote kernel and execute notebook.

Functions¶

register_remote_kernel_engine()

Register RemoteKernelEngine papermill engine.

Module Contents¶

airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_SHELL_PORT = 60316[source]¶

airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_IOPUB_PORT = 60317[source]¶

airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_STDIN_PORT = 60318[source]¶

airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_CONTROL_PORT = 60319[source]¶

airflow.providers.papermill.hooks.kernel.JUPYTER_KERNEL_HB_PORT = 60320[source]¶

airflow.providers.papermill.hooks.kernel.REMOTE_KERNEL_ENGINE = 'remote_kernel_engine'[source]¶

class airflow.providers.papermill.hooks.kernel.KernelConnection[source]¶

Class to represent kernel connection object.

ip: str[source]¶

shell_port: int[source]¶

iopub_port: int[source]¶

stdin_port: int[source]¶

control_port: int[source]¶

hb_port: int[source]¶

session_key: str[source]¶

class airflow.providers.papermill.hooks.kernel.KernelHook(kernel_conn_id=default_conn_name, *args, **kwargs)[source]¶

Bases: airflow.hooks.base.BaseHook

The KernelHook can be used to interact with remote jupyter kernel.

Takes kernel host/ip from connection and refers to jupyter kernel ports and session_key: from extra field.

Parameters:: kernel_conn_id (str) – connection that has kernel host/ip

conn_name_attr = 'kernel_conn_id'[source]¶

default_conn_name = 'jupyter_kernel_default'[source]¶

conn_type = 'jupyter_kernel'[source]¶

hook_name = 'Jupyter Kernel'[source]¶

kernel_conn[source]¶

get_conn()[source]¶

Return connection for the hook.

airflow.providers.papermill.hooks.kernel.register_remote_kernel_engine()[source]¶: Register RemoteKernelEngine papermill engine.

class airflow.providers.papermill.hooks.kernel.RemoteKernelManager(*args, **kwargs)[source]¶

Bases: jupyter_client.AsyncKernelManager

Jupyter kernel manager that connects to a remote kernel.

session_key[source]¶

property has_kernel: bool[source]¶

Has a kernel process been started that we are actively managing.

shutdown_kernel(now=False, restart=False)[source]¶

client(**kwargs)[source]¶

Create a client configured to connect to our kernel.

class airflow.providers.papermill.hooks.kernel.RemoteKernelEngine[source]¶

Bases: papermill.engines.NBClientEngine

Papermill engine to use RemoteKernelManager to connect to remote kernel and execute notebook.

classmethod execute_managed_notebook(nb_man, kernel_name, log_output=False, stdout_file=None, stderr_file=None, start_timeout=60, execution_timeout=None, **kwargs)[source]¶: Perform the actual execution of the parameterized notebook locally.