airflow.providers.apache.spark.hooks.spark_jdbc_script

Module Contents

Functions

set_common_options(spark_source: Any, url: str = 'localhost:5432', jdbc_table: str = 'default.default', user: str = 'root', password: str = 'root', driver: str = 'driver') → Any

Get Spark source from JDBC connection

spark_write_to_jdbc(spark_session: pyspark.sql.SparkSession, url: str, user: str, password: str, metastore_table: str, jdbc_table: str, driver: Any, truncate: bool, save_mode: str, batch_size: int, num_partitions: int, create_table_column_types: str) → None

Transfer data from Spark to JDBC source

spark_read_from_jdbc(spark_session: pyspark.sql.SparkSession, url: str, user: str, password: str, metastore_table: str, jdbc_table: str, driver: Any, save_mode: str, save_format: str, fetch_size: int, num_partitions: int, partition_column: str, lower_bound: str, upper_bound: str) → None

Transfer data from JDBC source to Spark

Attributes

SPARK_WRITE_TO_JDBC

SPARK_READ_FROM_JDBC

airflow.providers.apache.spark.hooks.spark_jdbc_script.SPARK_WRITE_TO_JDBC :str = spark_to_jdbc[source]
airflow.providers.apache.spark.hooks.spark_jdbc_script.SPARK_READ_FROM_JDBC :str = jdbc_to_spark[source]
airflow.providers.apache.spark.hooks.spark_jdbc_script.set_common_options(spark_source: Any, url: str = 'localhost:5432', jdbc_table: str = 'default.default', user: str = 'root', password: str = 'root', driver: str = 'driver') → Any[source]

Get Spark source from JDBC connection

Parameters

  • spark_source -- Spark source, here is Spark reader or writer

  • url -- JDBC resource url

  • jdbc_table -- JDBC resource table name

  • user -- JDBC resource user name

  • password -- JDBC resource password

  • driver -- JDBC resource driver

airflow.providers.apache.spark.hooks.spark_jdbc_script.spark_write_to_jdbc(spark_session: pyspark.sql.SparkSession, url: str, user: str, password: str, metastore_table: str, jdbc_table: str, driver: Any, truncate: bool, save_mode: str, batch_size: int, num_partitions: int, create_table_column_types: str) → None[source]

Transfer data from Spark to JDBC source

airflow.providers.apache.spark.hooks.spark_jdbc_script.spark_read_from_jdbc(spark_session: pyspark.sql.SparkSession, url: str, user: str, password: str, metastore_table: str, jdbc_table: str, driver: Any, save_mode: str, save_format: str, fetch_size: int, num_partitions: int, partition_column: str, lower_bound: str, upper_bound: str) → None[source]

Transfer data from JDBC source to Spark

Was this entry helpful?