airflow.providers.cncf.kubernetes.operators.custom_object_launcher¶

Launches Custom object.

Classes¶

`SparkJobSpec`	Spark job spec.
`KubernetesSpec`	Spark kubernetes spec.
`SparkResources`	spark resources.
`CustomObjectStatus`	Status of the PODs.
`CustomObjectLauncher`	Launches PODS.

Functions¶

should_retry_start_spark_job(exception)

Check if an Exception indicates a transient error and warrants retrying.

Module Contents¶

airflow.providers.cncf.kubernetes.operators.custom_object_launcher.should_retry_start_spark_job(exception)[source]¶

Check if an Exception indicates a transient error and warrants retrying.

class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.SparkJobSpec(**entries)[source]¶

Spark job spec.

validate()[source]¶

update_resources()[source]¶

class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.KubernetesSpec(**entries)[source]¶

Spark kubernetes spec.

set_attribute()[source]¶

class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.SparkResources(driver=None, executor=None)[source]¶

spark resources.

default[source]¶

driver[source]¶

executor[source]¶

property resources[source]¶: Return job resources.

property driver_resources[source]¶: Return resources to use.

property executor_resources[source]¶: Return resources to use.

convert_resources()[source]¶

class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.CustomObjectStatus[source]¶

Status of the PODs.

SUBMITTED = 'SUBMITTED'[source]¶

RUNNING = 'RUNNING'[source]¶

FAILED = 'FAILED'[source]¶

SUCCEEDED = 'SUCCEEDED'[source]¶

class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.CustomObjectLauncher(name, namespace, kube_client, custom_obj_api, template_body=None)[source]¶

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Launches PODS.

name[source]¶

namespace[source]¶

template_body = None[source]¶

body: dict[source]¶

kind[source]¶

plural = 'Uninferables'[source]¶

custom_obj_api[source]¶

spark_obj_spec: dict[source]¶

pod_spec: kubernetes.client.models.V1Pod | None = None[source]¶

property pod_manager: airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager[source]¶

get_body()[source]¶

start_spark_job(image=None, code_path=None, startup_timeout=600)[source]¶

Launch the pod synchronously and waits for completion.

Parameters:

image – image name
code_path – path to the .py file for python and jar file for scala
startup_timeout (int) – Timeout for startup of the pod (if pod is pending for too long, fails task)

Returns:

spark_job_not_running(spark_obj_spec)[source]¶: Test if spark_obj_spec has not started.

check_pod_start_failure()[source]¶

delete_spark_job(spark_job_name=None)[source]¶: Delete spark job.