airflow.providers.apache.spark.hooks.spark_jdbc_script¶
Module Contents¶
-
airflow.providers.apache.spark.hooks.spark_jdbc_script.SPARK_WRITE_TO_JDBC:str = spark_to_jdbc[source]¶
-
airflow.providers.apache.spark.hooks.spark_jdbc_script.SPARK_READ_FROM_JDBC:str = jdbc_to_spark[source]¶
-
airflow.providers.apache.spark.hooks.spark_jdbc_script.set_common_options(spark_source: Any, url: str = 'localhost:5432', jdbc_table: str = 'default.default', user: str = 'root', password: str = 'root', driver: str = 'driver') → Any[source]¶ -
Get Spark source from JDBC connection - Parameters
spark_source – Spark source, here is Spark reader or writer
url – JDBC resource url
jdbc_table – JDBC resource table name
user – JDBC resource user name
password – JDBC resource password
driver – JDBC resource driver
-
airflow.providers.apache.spark.hooks.spark_jdbc_script.spark_write_to_jdbc(spark_session: SparkSession, url: str, user: str, password: str, metastore_table: str, jdbc_table: str, driver: Any, truncate: bool, save_mode: str, batch_size: int, num_partitions: int, create_table_column_types: str) → None[source]¶ -
Transfer data from Spark to JDBC source
-
airflow.providers.apache.spark.hooks.spark_jdbc_script.spark_read_from_jdbc(spark_session: SparkSession, url: str, user: str, password: str, metastore_table: str, jdbc_table: str, driver: Any, save_mode: str, save_format: str, fetch_size: int, num_partitions: int, partition_column: str, lower_bound: str, upper_bound: str) → None[source]¶ -
Transfer data from JDBC source to Spark
-
airflow.providers.apache.spark.hooks.spark_jdbc_script._parse_arguments(args: Optional[List[str]] = None) → Any[source]¶