airflow.providers.cncf.kubernetes.hooks.kubernetes¶

Attributes¶

`LOADING_KUBE_CONFIG_FILE_RESOURCE`
`JOB_FINAL_STATUS_CONDITION_TYPES`
`JOB_STATUS_CONDITION_TYPES`

Classes¶

`KubernetesHook`	Creates Kubernetes API connection.
`AsyncKubernetesHook`	Hook to use Kubernetes SDK asynchronously.

Module Contents¶

airflow.providers.cncf.kubernetes.hooks.kubernetes.LOADING_KUBE_CONFIG_FILE_RESOURCE = 'Loading Kubernetes configuration file kube_config from {}...'[source]¶

airflow.providers.cncf.kubernetes.hooks.kubernetes.JOB_FINAL_STATUS_CONDITION_TYPES[source]¶

airflow.providers.cncf.kubernetes.hooks.kubernetes.JOB_STATUS_CONDITION_TYPES[source]¶

class airflow.providers.cncf.kubernetes.hooks.kubernetes.KubernetesHook(conn_id=default_conn_name, client_configuration=None, cluster_context=None, config_file=None, in_cluster=None, disable_verify_ssl=None, disable_tcp_keepalive=None)[source]¶

Bases: airflow.hooks.base.BaseHook, airflow.providers.cncf.kubernetes.utils.pod_manager.PodOperatorHookProtocol

Creates Kubernetes API connection.

use in cluster configuration by using extra field in_cluster in connection
use custom config by providing path to the file using extra field kube_config_path in connection
use custom configuration by providing content of kubeconfig file via
extra field kube_config in connection
use default config by providing no extras

This hook check for configuration option in the above order. Once an option is present it will use this configuration.

See also

For more information about Kubernetes connection: Kubernetes cluster Connection

Parameters:

conn_id (str | None) – The kubernetes connection to Kubernetes cluster.
client_configuration (kubernetes.client.Configuration | None) – Optional dictionary of client configuration params. Passed on to kubernetes client.
cluster_context (str | None) – Optionally specify a context to use (e.g. if you have multiple in your kubeconfig.
config_file (str | None) – Path to kubeconfig file.
in_cluster (bool | None) – Set to True if running from within a kubernetes cluster.
disable_verify_ssl (bool | None) – Set to True if SSL verification should be disabled.
disable_tcp_keepalive (bool | None) – Set to True if you want to disable keepalive logic.

conn_name_attr = 'kubernetes_conn_id'[source]¶

default_conn_name = 'kubernetes_default'[source]¶

conn_type = 'kubernetes'[source]¶

hook_name = 'Kubernetes Cluster Connection'[source]¶

DEFAULT_NAMESPACE = 'default'[source]¶

classmethod get_connection_form_widgets()[source]¶

Return connection widgets to add to connection form.

classmethod get_ui_field_behaviour()[source]¶

Return custom field behaviour.

conn_id = 'kubernetes_default'[source]¶

client_configuration = None[source]¶

cluster_context = None[source]¶

config_file = None[source]¶

in_cluster = None[source]¶

disable_verify_ssl = None[source]¶

disable_tcp_keepalive = None[source]¶

classmethod get_connection(conn_id)[source]¶

Return requested connection.

If missing and conn_id is “kubernetes_default”, will return empty connection so that hook will default to cluster-derived credentials.

property conn_extras[source]¶

get_conn()[source]¶

Return kubernetes api session for use with requests.

property is_in_cluster: bool[source]¶

Expose whether the hook is configured with load_incluster_config or not.

property api_client: kubernetes.client.ApiClient[source]¶

Cached Kubernetes API client.

property core_v1_client: kubernetes.client.CoreV1Api[source]¶

Get authenticated client object.

property apps_v1_client: kubernetes.client.AppsV1Api[source]¶

property custom_object_client: kubernetes.client.CustomObjectsApi[source]¶

property batch_v1_client: kubernetes.client.BatchV1Api[source]¶

create_custom_object(group, version, plural, body, namespace=None)[source]¶

Create custom resource definition object in Kubernetes.

Parameters:

group (str) – api group
version (str) – api version
plural (str) – api plural
body (str | dict) – crd object definition
namespace (str | None) – kubernetes namespace

get_custom_object(group, version, plural, name, namespace=None)[source]¶

Get custom resource definition object from Kubernetes.

Parameters:

group (str) – api group
version (str) – api version
plural (str) – api plural
name (str) – crd object name
namespace (str | None) – kubernetes namespace

delete_custom_object(group, version, plural, name, namespace=None, **kwargs)[source]¶

Delete custom resource definition object from Kubernetes.

Parameters:

group (str) – api group
version (str) – api version
plural (str) – api plural
name (str) – crd object name
namespace (str | None) – kubernetes namespace

get_namespace()[source]¶

Return the namespace that defined in the connection.

get_xcom_sidecar_container_image()[source]¶: Return the xcom sidecar image that defined in the connection.

get_xcom_sidecar_container_resources()[source]¶: Return the xcom sidecar resources that defined in the connection.

get_pod_log_stream(pod_name, container='', namespace=None)[source]¶

Retrieve a log stream for a container in a kubernetes pod.

Parameters:

pod_name (str) – pod name
container (str | None) – container name
namespace (str | None) – kubernetes namespace

get_pod_logs(pod_name, container='', namespace=None)[source]¶

Retrieve a container’s log from the specified pod.

Parameters:

pod_name (str) – pod name
container (str | None) – container name
namespace (str | None) – kubernetes namespace

get_pod(name, namespace)[source]¶

Read pod object from kubernetes API.

get_namespaced_pod_list(label_selector='', namespace=None, watch=False, **kwargs)[source]¶

Retrieve a list of Kind pod which belong default kubernetes namespace.

Parameters:

label_selector (str | None) – A selector to restrict the list of returned objects by their labels
namespace (str | None) – kubernetes namespace
watch (bool) – Watch for changes to the described resources and return them as a stream

get_deployment_status(name, namespace='default', **kwargs)[source]¶

Get status of existing Deployment.

Parameters:

name (str) – Name of Deployment to retrieve
namespace (str) – Deployment namespace

create_job(job, **kwargs)[source]¶

Run Job.

Parameters:: job (kubernetes.client.models.V1Job) – A kubernetes Job object

get_job(job_name, namespace)[source]¶

Get Job of specified name and namespace.

Parameters:

job_name (str) – Name of Job to fetch.
namespace (str) – Namespace of the Job.

Returns:

Job object

Return type:

kubernetes.client.models.V1Job

get_job_status(job_name, namespace)[source]¶

Get job with status of specified name and namespace.

Parameters:

job_name (str) – Name of Job to fetch.
namespace (str) – Namespace of the Job.

Returns:

Job object

Return type:

kubernetes.client.models.V1Job

wait_until_job_complete(job_name, namespace, job_poll_interval=10)[source]¶

Block job of specified name and namespace until it is complete or failed.

Parameters:

job_name (str) – Name of Job to fetch.
namespace (str) – Namespace of the Job.
job_poll_interval (float) – Interval in seconds between polling the job status

Returns:

Job object

Return type:

kubernetes.client.models.V1Job

list_jobs_all_namespaces()[source]¶

Get list of Jobs from all namespaces.

Returns:: V1JobList object
Return type:: kubernetes.client.V1JobList

list_jobs_from_namespace(namespace)[source]¶

Get list of Jobs from dedicated namespace.

Parameters:: namespace (str) – Namespace of the Job.
Returns:: V1JobList object
Return type:: kubernetes.client.V1JobList

is_job_complete(job)[source]¶

Check whether the given job is complete (with success or fail).

Returns:: Boolean indicating that the given job is complete.
Return type:: bool

static is_job_failed(job)[source]¶

Check whether the given job is failed.

Returns:: Error message if the job is failed, and False otherwise.
Return type:: str | bool

static is_job_successful(job)[source]¶

Check whether the given job is completed successfully..

Returns:: Error message if the job is failed, and False otherwise.
Return type:: str | bool

patch_namespaced_job(job_name, namespace, body)[source]¶

Update the specified Job.

Parameters:

job_name (str) – name of the Job
namespace (str) – the namespace to run within kubernetes
body (object) – json object with parameters for update

apply_from_yaml_file(api_client=None, yaml_file=None, yaml_objects=None, verbose=False, namespace='default')[source]¶

Perform an action from a yaml file.

Parameters:

api_client (Any) – A Kubernetes client application.
yaml_file (str | None) – Contains the path to yaml file.
yaml_objects (list[dict] | None) – List of YAML objects; used instead of reading the yaml_file.
verbose (bool) – If True, print confirmation from create action. Default is False.
namespace (str) – Contains the namespace to create all resources inside. The namespace must preexist otherwise the resource creation will fail.

check_kueue_deployment_running(name, namespace, timeout=300.0, polling_period_seconds=2.0)[source]¶

static get_yaml_content_from_file(kueue_yaml_url)[source]¶

Download content of YAML file and separate it into several dictionaries.

class airflow.providers.cncf.kubernetes.hooks.kubernetes.AsyncKubernetesHook(config_dict=None, *args, **kwargs)[source]¶

Bases: KubernetesHook

Hook to use Kubernetes SDK asynchronously.

config_dict = None[source]¶

async get_conn_extras()[source]¶

async get_conn()[source]¶

Return kubernetes api session for use with requests.

async get_pod(name, namespace)[source]¶

Get pod’s object.

Parameters:

name (str) – Name of the pod.
namespace (str) – Name of the pod’s namespace.

async delete_pod(name, namespace)[source]¶

Delete pod’s object.

Parameters:

name (str) – Name of the pod.
namespace (str) – Name of the pod’s namespace.

async read_logs(name, namespace)[source]¶

Read logs inside the pod while starting containers inside.

All the logs will be outputted with its timestamp to track the logs after the execution of the pod is completed. The method is used for async output of the logs only in the pod failed it execution or the task was cancelled by the user.

Parameters:

name (str) – Name of the pod.
namespace (str) – Name of the pod’s namespace.

async get_job_status(name, namespace)[source]¶

Get job’s status object.

Parameters:

name (str) – Name of the pod.
namespace (str) – Name of the pod’s namespace.

async wait_until_job_complete(name, namespace, poll_interval=10)[source]¶

Block job of specified name and namespace until it is complete or failed.

Parameters:

name (str) – Name of Job to fetch.
namespace (str) – Namespace of the Job.
poll_interval (float) – Interval in seconds between polling the job status

Returns:

Job object

Return type:

kubernetes.client.models.V1Job

async wait_until_container_complete(name, namespace, container_name, poll_interval=10)[source]¶

Wait for the given container in the given pod to be completed.

Parameters:

name (str) – Name of Pod to fetch.
namespace (str) – Namespace of the Pod.
container_name (str) – name of the container within the pod to monitor
poll_interval (float) – Interval in seconds between polling the container status

async wait_until_container_started(name, namespace, container_name, poll_interval=10)[source]¶

Wait for the given container in the given pod to be started.

Parameters:

name (str) – Name of Pod to fetch.
namespace (str) – Namespace of the Pod.
container_name (str) – name of the container within the pod to monitor
poll_interval (float) – Interval in seconds between polling the container status