airflow.providers.cncf.kubernetes.operators.spark_kubernetes

Module Contents

class airflow.providers.cncf.kubernetes.operators.spark_kubernetes.SparkKubernetesOperator(*, application_file: str, namespace: Optional[str] = None, kubernetes_conn_id: str = 'kubernetes_default', **kwargs)[source]

Bases: airflow.models.BaseOperator

Creates sparkApplication object in kubernetes cluster:

See also

For more detail about Spark Application Object have a look at the reference: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.1.0-2.4.5/docs/api-docs.md#sparkapplication

Parameters

  • application_file (str) -- filepath to kubernetes custom_resource_definition of sparkApplication

  • namespace (str) -- kubernetes namespace to put sparkApplication

  • kubernetes_conn_id (str) -- the connection to Kubernetes cluster

template_fields = ['application_file', 'namespace'][source]
template_ext = ['yaml', 'yml', 'json'][source]
ui_color = #f4a460[source]
execute(self, context)[source]

Was this entry helpful?