Broadcast 广播
Broadcast 可以将数据广播到每个节点上,这样节点需要使用该数据是无需发送请求,大大提升了效率。
Broadcast 提供了接口,具体的方法在子类中实现,如子类TorrentBroadcast。
源码清单和我的理解注释
ListenerBus源码中第一行,创建了一个线程安全的ArrayList——CopyOnWriteArrayList,之后添加、者删除事件等操作都在这个线程安全的ArrayList中执行
/**
* 标记广播变量是否有效的标志
*/
@volatile private var _isValid = true
/** 先判断广播变量是否有效再获取值 */
def value: T = {
assertValid()
getValue()
}
删除执行器excutor中的广播副本
/**
* Asynchronously delete cached copies of this broadcast on the executors.
* If the broadcast is used after this is called, it will need to be re-sent to each executor.
*/
def unpersist() {
unpersist(blocking = false)
}
/**
* Delete cached copies of this broadcast on the executors. If the broadcast is used after
* this is called, it will need to be re-sent to each executor.
* @param blocking Whether to block until unpersisting has completed
*/
def unpersist(blocking: Boolean) {
assertValid()
doUnpersist(blocking)
}
摧毁广播变量
/**
* Destroy all data and metadata related to this broadcast variable. Use this with caution;
* once a broadcast variable has been destroyed, it cannot be used again.
* This method blocks until destroy has completed
* 摧毁与此广播变量相关的所有数据和元数据,一旦摧毁便无法再使用它
* 这个函数将被阻塞,直到摧毁工作完成
*/
def destroy() {
destroy(blocking = true)
}
/**
* Destroy all data and metadata related to this broadcast variable. Use this with caution;
* once a broadcast variable has been destroyed, it cannot be used again.
* @param blocking Whether to block until destroy has completed
* 摧毁与此广播变量相关的所有数据和元数据,一旦摧毁便无法再使用它
* 参数blocking可以选择是否在摧毁完成前阻止摧毁工作
*/
private[spark] def destroy(blocking: Boolean) {
assertValid()
_isValid = false
_destroySite = Utils.getCallSite().shortForm
logInfo("Destroying %s (from %s)".format(toString, _destroySite))
doDestroy(blocking)
}