airflow.providers.cncf.kubernetes.operators.spark_kubernetes
¶
Module Contents¶
Classes¶
Creates sparkApplication object in kubernetes cluster: |
- class airflow.providers.cncf.kubernetes.operators.spark_kubernetes.SparkKubernetesOperator(*, application_file, namespace=None, kubernetes_conn_id='kubernetes_default', api_group='sparkoperator.k8s.io', api_version='v1beta2', **kwargs)[source]¶
Bases:
airflow.models.BaseOperator
Creates sparkApplication object in kubernetes cluster:
See also
For more detail about Spark Application Object have a look at the reference: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.1.0-2.4.5/docs/api-docs.md#sparkapplication
- Parameters
application_file (str) -- Defines Kubernetes 'custom_resource_definition' of 'sparkApplication' as either a path to a '.yaml' file, '.json' file, YAML string or JSON string.
namespace (Optional[str]) -- kubernetes namespace to put sparkApplication
kubernetes_conn_id (str) -- The kubernetes connection id for the to Kubernetes cluster.
api_group (str) -- kubernetes api group of sparkApplication
api_version (str) -- kubernetes api version of sparkApplication