mysql jdbcrdd_Spark与Mysql(JdbcRDD)整合开发(zh)

在Spark中提供了一个JdbcRDD类，该RDD就是读取JDBC中的数据并转换成RDD，之后我们就可以对该RDD进行各种的操作。我们先看看该类的构造函数：

1

JdbcRDD[T: ClassTag](

2

sc: SparkContext,

3

getConnection: () => Connection,

4

sql: String,

5

lowerBound: Long,

6

upperBound: Long,

7

numPartitions: Int,

8

mapRow: (ResultSet) => T = JdbcRDD.resultSetToObjectArray _)

这个类带了很多参数，关于这个函数的各个参数的含义，我觉得直接看英文就可以很好的理解，如下：

@param getConnection a function that returns an open Connection.

The RDD takes care of closing the connection.

@param sql the text of the query.

The query must contain two ? placeholders for parameters used to partition the results.

E.g. “select title, author from books where ? < = id and id <= ?”

@param lowerBound the minimum value of the first placeholder

@param upperBound the maximum value of the second placeholder

The lower and upper bounds are inclusive.

@param numPartitions the number of partitions.

Given a lowerBound of 1, an upperBound of 20, and a numPartitions of 2,

the query would be executed twice, once with (1, 10) and once with (11, 20)

@param mapRow a function from a ResultSet to a single row of the desired result type(s).

This should only call getInt, getString, etc; the RDD takes care of calling next.

The default maps a ResultSet to an array of Object.

上面英文看不懂？？那好吧，我给你翻译一下。

1、getConnection 返回一个已经打开的结构化数据库连接，JdbcRDD会自动维护关闭。

2、sql 是查询语句，此查询语句必须包含两处占位符?来作为分割数据库ResulSet的参数，例如：”select title, author from books where ? < = id