Spark读取mysql(jdbc)数据分区两种API

最新推荐文章于 2022-12-09 10:41:58 发布

遥遥晚风点点

最新推荐文章于 2022-12-09 10:41:58 发布

阅读量701

点赞数

分类专栏： Spark 文章标签： spark 大数据

本文链接：https://blog.csdn.net/Mr_ye931/article/details/127038237

版权

Spark 专栏收录该内容

14 篇文章 0 订阅

订阅专栏

1，sparkSession.read.jdbc

    val modelDataSql = "select pay_time, ...   "
    
    //根据时间分区
    val partitionClause = Array(
                            "EXTRACT(month FROM pay_time) = 1 "
                          , "EXTRACT(month FROM pay_time) = 2 "
                          , "EXTRACT(month FROM pay_time) = 3 "
                          , "EXTRACT(month FROM pay_time) = 4 "
                          , "EXTRACT(month FROM pay_time) = 5 "
                          , "EXTRACT(month FROM pay_time) = 6 "
                          , "EXTRACT(month FROM pay_time) = 7 "
                          , "EXTRACT(month FROM pay_time) = 8 "
                          , "EXTRACT(month FROM pay_time) = 9 "
                          , "EXTRACT(month FROM pay_time) = 10 "
                          , "EXTRACT(month FROM pay_time) = 11 "
                          , "EXTRACT(month FROM pay_time) = 12 "
            )

    sparkSession
        .read
        .jdbc(
                      "jdbc.xxxx.url"
                    , s" ( $modelDataSql ) T"
                    , partitionClause
                    , jdbcProperties
)

2,写在option中

    var dataDf: DataFrame = spark.read.format("jdbc")
      .option("url", jdbcArgs.get("url").get)
      .option("driver", jdbcArgs.get("driver").get)
      .option("user", jdbcArgs.get("user").get)
      .option("password", jdbcArgs.get("password").get)
      .option("fetchSize", jdbcArgs.get("fetchSize").get)
      .option("dbtable", "(" + jdbcArgs.get("sql").get + ") temp")
      .option("partitionColumn","xxxxx")
      .option("lowerBound","xxxxx")
      .option("upperBound","xxxxx")
      .option("numPartitions",xxx)
      .load()