spark编程mysql数据源_Spark - 直接操作数据源 MySQL

如果我们的Mysql服务器性能不咋滴，但是硬盘很够，如何才能做各种复杂的聚合操作？答案就是使用spark的计算能力的，我们可以将mysql数据源接入到spark中。

读取

val mysqlDF = spark

.read

.format("jdbc")

.option("driver","com.mysql.jdbc.Driver")

.option("url","jdbc:mysql://localhost:3306/ttable")

.option("user","root")

.option("password","root")

.option("dbtable","(select * from ttt where userId >1 AND userId < 10) as log")//条件查询出想要的表

//.option("dbtable","ttable.ttt")//整张表

.option("fetchsize","100")

.option("useSSL","false")

.load()

分区读取

spark

.read

.format("jdbc")

.option("url", url)

.option("dbtable", "ttt")

.option("user", user)

.option("password", password)

.option("numPartitions", 10)

.option("partitionColumn", "userId")

.option("lowerBound", 1)

.option("upperBound", 10000)

.load()

实际会生成如下查询语句,(所有分区会一直查询，直到整张表数据查询完为止)

SELECT * FROM ttt WHERE userId >= 1 and userId < 1000

SELECT * FROM ttt WHERE userId >= 1000 and userId < 2000

SELECT * FROM ttt WHERE userId >= 2000 and userId < 3000

...

写入

mysqlDF.createTempView("log")

spark

.sql("select * from log")

.toDF()

.write

.mode(SaveMode.Overwrite)

.format("jdbc")

.option("driver","com.mysql.jdbc.Driver")

.option("url","jdbc:mysql://localhost:3306/ttable")

.option("dbtable","a")

.option("user","root")

.option("password","root")

.option("fetchsize","100")

.option("useSSL","false")

.save()