SparkStreaming中常用的操作之一就是将数据导出,例如,将数据结果发送到mysql数据库中。
这时候很容易会犯一些错误,例如:
dstream.foreachRDD { rdd =>
val connection = createNewConnection() // executed at the driver
rdd.foreach { record =>
connection.send(record) // executed at the worker
}
}
这种写法将会产生序列化错误,因为我们知道在foreach中闭包函数将会从driver节点发送到各个worker节点上执行,这时候由于connection并非序列化类产生的对象,因此该对象将不能被序列化,从而引发序列化错误的异常。
那么,这时候有些人可能会写出如下代码:
dstream.foreachRDD { rdd =>
rdd.foreach { record =>
val connection = createNewConnection()
connection.send(record)
connection.close()
}
}
这样写虽然不报错,但是会出现为每条记录创建一个connection,众所周知connection的创建十分耗费资源,因此这种写法虽然能够正常运行但是在效率上却大打折扣。
在这之上对spark RDD机制有了解的读者可能会对上述代码继续优化构造为如下代码:
dstream.foreachRDD { rdd =>
rdd.foreachPartition { partitionOfRecords =>
val connection = createNewConnection()
partitionOfRecords.foreach(record => connection.send(record))
connection.close()
}
}
但上述代码,仍有不足。有过不错编程经验的读者可能会联想到datapool 或者 threadpool,从而实现connection的重用。正如spark官网中的解决方案,同样是创建一个pool,在这个pool里面存放connection,如下所示:
dstream.foreachRDD { rdd =>
rdd.foreachPartition { partitionOfRecords =>
// ConnectionPool is a static, lazily initialized pool of connections
val connection = ConnectionPool.getConnection()
partitionOfRecords.foreach(record => connection.send(record))
ConnectionPool.returnConnection(connection) // return to the pool for future reuse
}
}
至此,优化完成。对于ConnectionPool的实现,下面给出简单例子
Connection类
public class Connection {
public String conn(String s) {
s = "donggege";
System.out.println("conn .....");
return s;
}
}
ConnectionPool 类
public class ConnectionPool{
private static LinkedList<Connection> connectionQueue;
public synchronized static Connection getConnection() {
try {
if (connectionQueue == null) {
connectionQueue = new LinkedList<Connection>();
for (int i = 0; i < 5; i++) {
Connection conn = new Connection();
connectionQueue.push(conn);
}
}
} catch (Exception e) {
e.printStackTrace();
}
return connectionQueue.poll();
}
public static void returnConnection(Connection conn) {
connectionQueue.push(conn);
}
}
SparkStreaming调用
dstream.foreachRDD { rdd =>
rdd.foreachPartition { partitionOfRecords =>
// ConnectionPool is a static, lazily initialized pool of connections
val connection = ConnectionPool.getConnection()
partitionOfRecords.foreach(record => connection.<span style="font-family: Consolas, 'Courier New', Courier, mono, serif; line-height: 18px; background-color: rgb(248, 248, 248);">conn</span>(record))
ConnectionPool.returnConnection(connection) // return to the pool for future reuse
}
}