airflow.providers.apache.spark.operators.spark_pyspark¶

Attributes¶

SPARK_CONTEXT_KEYS

Classes¶

PySparkOperator

Submit the run of a pyspark job to an external spark-connect service or directly run the pyspark job in a standalone mode.

Module Contents¶

airflow.providers.apache.spark.operators.spark_pyspark.SPARK_CONTEXT_KEYS = ['spark', 'sc'][source]¶

class airflow.providers.apache.spark.operators.spark_pyspark.PySparkOperator(python_callable, conn_id=None, config_kwargs=None, **kwargs)[source]¶

Bases: airflow.providers.common.compat.standard.operators.PythonOperator

Submit the run of a pyspark job to an external spark-connect service or directly run the pyspark job in a standalone mode.

template_fields: collections.abc.Sequence[str][source]¶

conn_id = None[source]¶

config_kwargs[source]¶

execute_callable()[source]¶