airflow.providers.apache.spark.hooks.spark_jdbc_script

Module Contents

airflow.providers.apache.spark.hooks.spark_jdbc_script.SPARK_WRITE_TO_JDBC :str = spark_to_jdbc[source]
airflow.providers.apache.spark.hooks.spark_jdbc_script.SPARK_READ_FROM_JDBC :str = jdbc_to_spark[source]
airflow.providers.apache.spark.hooks.spark_jdbc_script.set_common_options(spark_source: Any, url: str = 'localhost:5432', jdbc_table: str = 'default.default', user: str = 'root', password: str = 'root', driver: str = 'driver') → Any[source]
Get Spark source from JDBC connection
Parameters
  • spark_source – Spark source, here is Spark reader or writer

  • url – JDBC resource url

  • jdbc_table – JDBC resource table name

  • user – JDBC resource user name

  • password – JDBC resource password

  • driver – JDBC resource driver

airflow.providers.apache.spark.hooks.spark_jdbc_script.spark_write_to_jdbc(spark_session: SparkSession, url: str, user: str, password: str, metastore_table: str, jdbc_table: str, driver: Any, truncate: bool, save_mode: str, batch_size: int, num_partitions: int, create_table_column_types: str)None[source]
Transfer data from Spark to JDBC source
airflow.providers.apache.spark.hooks.spark_jdbc_script.spark_read_from_jdbc(spark_session: SparkSession, url: str, user: str, password: str, metastore_table: str, jdbc_table: str, driver: Any, save_mode: str, save_format: str, fetch_size: int, num_partitions: int, partition_column: str, lower_bound: str, upper_bound: str)None[source]
Transfer data from JDBC source to Spark
airflow.providers.apache.spark.hooks.spark_jdbc_script._parse_arguments(args: Optional[List[str]] = None) → Any[source]
airflow.providers.apache.spark.hooks.spark_jdbc_script._create_spark_session(arguments: Any) → SparkSession[source]
airflow.providers.apache.spark.hooks.spark_jdbc_script._run_spark(arguments: Any)None[source]

Was this entry helpful?