airflow.providers.cncf.kubernetes.sensors.spark_kubernetes
¶
Module Contents¶
Classes¶
Checks sparkApplication object in kubernetes cluster: |
- class airflow.providers.cncf.kubernetes.sensors.spark_kubernetes.SparkKubernetesSensor(*, application_name, attach_log=False, namespace=None, container_name='spark-kubernetes-driver', kubernetes_conn_id='kubernetes_default', api_group='sparkoperator.k8s.io', api_version='v1beta2', **kwargs)[source]¶
Bases:
airflow.sensors.base.BaseSensorOperator
Checks sparkApplication object in kubernetes cluster:
See also
For more detail about Spark Application Object have a look at the reference: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.1.0-2.4.5/docs/api-docs.md#sparkapplication
- Parameters
application_name (str) – spark Application resource name
namespace (str | None) – the kubernetes namespace where the sparkApplication reside in
container_name (str) – the kubernetes container name where the sparkApplication reside in
kubernetes_conn_id (str) – The kubernetes connection to Kubernetes cluster.
attach_log (bool) – determines whether logs for driver pod should be appended to the sensor log
api_group (str) – kubernetes api group of sparkApplication
api_version (str) – kubernetes api version of sparkApplication