Spark中的Broadcast处理
首先先来看一看broadcast的使用代码:
val values = List[Int](1,2,3)
val broadcastValues = sparkContext.broadcast(values)
rdd.mapPartitions(iter => {
broadcastValues.getValue.foreach(println)
})
在上面的代码中,首先生成了一个集合变量,把这个变量通过sparkContext的broadcast函数进行广播,
最后在rdd的每一个partition的迭代时,使用这个广播变量.
接下来看看广播变量的生成与数据的读取实现部分:
def broadcast[T: ClassTag](value: T): Broadcast[T] = {
assertNotStopped()
if (classOf[RDD[_]].isAssignableFrom(classTag[T].runtimeClass)) {
这里要注意,使用broadcast时,不能直接对RDD进行broadcast的操作.
// This is a warning instead of an exception in order to avoid breaking
// user programs that
// might have created RDD broadcast variables but not used them:
logWarning("Can not directly broadcast RDDs; instead, call collect() and "
+ "broadcast the result (see SPARK-5063)")
}
通过broadcastManager中的newBroadcast函数来进行广播.
val bc = env.broadcastManager.newBroadcast[T](value, isLocal)
val callSite = getCallSite
logInfo("Created broadcast " + bc.id + " from " + callSite.shortForm)
cleaner.foreach(_.registerBroadcastForCleanup(bc))
bc
}
在BroadcastManager中生成广播变量的函数,这个函数直接使用的broadcastFactory的对应函数.
broadcastFactory的实例通过配置spark.broadcast.factory,
默认是TorrentBroadcastFactory