airflow.providers.cncf.kubernetes.operators.custom_object_launcher

Launches Custom object.

Module Contents

Classes

SparkJobSpec

Spark job spec.

KubernetesSpec

Spark kubernetes spec.

SparkResources

spark resources.

CustomObjectStatus

Status of the PODs.

CustomObjectLauncher

Launches PODS.

Functions

should_retry_start_spark_job(exception)

Check if an Exception indicates a transient error and warrants retrying.
airflow.providers.cncf.kubernetes.operators.custom_object_launcher.should_retry_start_spark_job(exception)[source]

Check if an Exception indicates a transient error and warrants retrying.

class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.SparkJobSpec(**entries)[source]

Spark job spec.

validate()[source]
update_resources()[source]
class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.KubernetesSpec(**entries)[source]

Spark kubernetes spec.

set_attribute()[source]
class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.SparkResources(driver=None, executor=None)[source]

spark resources.

property resources[source]

Return job resources.

property driver_resources[source]

Return resources to use.

property executor_resources[source]

Return resources to use.

convert_resources()[source]
class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.CustomObjectStatus[source]

Status of the PODs.

SUBMITTED = 'SUBMITTED'[source]
RUNNING = 'RUNNING'[source]
FAILED = 'FAILED'[source]
SUCCEEDED = 'SUCCEEDED'[source]
class airflow.providers.cncf.kubernetes.operators.custom_object_launcher.CustomObjectLauncher(name, namespace, kube_client, custom_obj_api, template_body=None)[source]

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Launches PODS.

pod_manager()[source]
get_body()[source]
start_spark_job(image=None, code_path=None, startup_timeout=600)[source]

Launch the pod synchronously and waits for completion.

Parameters

  • image – image name

  • code_path – path to the .py file for python and jar file for scala

  • startup_timeout (int) – Timeout for startup of the pod (if pod is pending for too long, fails task)

Returns

spark_job_not_running(spark_obj_spec)[source]

Test if spark_obj_spec has not started.

check_pod_start_failure()[source]
delete_spark_job(spark_job_name=None)[source]

Delete spark job.

Was this entry helpful?