mysql中有一个数据库为luciusdb,其中有一张表people,有8条数据。
1、如果采用spark-shell来访问数据库,可以像下面那样在启动spark-shell时在命令之前加上“SPARK_CLASSPATH=驱动包”来 导入数据库驱动:
SPARK_CLASSPATH=/opt/spark-1.3.0/lib/mysql-connector-java-5.1.35.jar bin/spark-shell
2、如果采用IDEA, 也需要导入包:
代码如下:
Java Code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import
java.sql.DriverManager
import
org.apache.spark.rdd.JdbcRDD
import
org.apache.spark.{SparkContext}
object HiSpark {
def
main (args: Array[
String
]) {
val sc =
new
SparkContext(
"local"
,
"mysql"
)
val rdd =
new
JdbcRDD(
sc,
() => {
Class.forName(
"com.mysql.jdbc.Driver"
).newInstance()
DriverManager.getConnection(
"jdbc:mysql://192.168.70.135:3306/luciusdb"
,
"root"
,
"123456"
)
},
"SELECT * FROM people where id>=? and id <=?"
,
1
,
4
,
2
,
r =>r.getString(
3
) +
" "
+ r.getString(
1
) +
" "
+r.getString(
2
)
)
rdd.foreach(println)
}
}
查到的数据如下:
相关参数说明:
参考网址:
http://www.iteblog.com/archives/1113
http://spark.apache.org/docs/1.3.0/sql-programming-guide.html