Module Contents

class airflow.providers.cncf.kubernetes.sensors.spark_kubernetes.SparkKubernetesSensor(*, application_name: str, attach_log: bool = False, namespace: Optional[str] = None, kubernetes_conn_id: str = 'kubernetes_default', api_group: str = '', api_version: str = 'v1beta2', **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Checks sparkApplication object in kubernetes cluster:

See also

For more detail about Spark Application Object have a look at the reference:

  • application_name (str) -- spark Application resource name

  • namespace (str) -- the kubernetes namespace where the sparkApplication reside in

  • kubernetes_conn_id (str) -- The kubernetes connection to Kubernetes cluster.

  • attach_log (bool) -- determines whether logs for driver pod should be appended to the sensor log

  • api_group (str) -- kubernetes api group of sparkApplication

  • api_version (str) -- kubernetes api version of sparkApplication

template_fields = ['application_name', 'namespace'][source]
_log_driver(self, application_state: str, response: dict)[source]
poke(self, context: Dict)[source]

Was this entry helpful?