Dispatcher是Spark通信框架中的消息分发器,会在NettyRpcEnv初始化的时候创建,NettyRpcEnv的初始化过程我们以后再谈,现在我们先看一下Dispatcher初始化时都干了哪些事。
private val endpoints: ConcurrentMap[String, EndpointData] =
new ConcurrentHashMap[String, EndpointData]
private val endpointRefs: ConcurrentMap[RpcEndpoint, RpcEndpointRef] =
new ConcurrentHashMap[RpcEndpoint, RpcEndpointRef]
// Track the receivers whose inboxes may contain messages.
private val receivers = new LinkedBlockingQueue[EndpointData]
/**
* True if the dispatcher has been stopped. Once stopped, all messages posted will be bounced
* immediately.
*/
@GuardedBy("this")
private var stopped = false
1. 创建一个endpoints:ConcurrentMap 用来存放EndPoint,key是EndPoint的名称,value是EndPointData
2. 创建一个endpointRefs:ConcurrentMap 用来存放EndPoint的引用,key是RpcEndpoint自身,value是RpcEndpointRef
3. 创建一个receivers:LinkedBlockingQueue[EndpointData] 来存放Dispatcher接收到的EndpointData
4. 声明标量 stopped 标记Dispatcher是否已停止,默认false
/** Thread pool used for dispatching messages. */
private val threadpool: ThreadPoolExecutor = {
val availableCores =
if (numUsableCores > 0) numUsableCores else Runtime.getRuntime.availableProcessors()
val numThreads = nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
math.max(2, availableCores))
val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
for (i <- 0 until numThreads) {
pool.execute(new MessageLoop)
}
pool
}
5. 定义了一个线程池 threadpool:ThreadPoolExecutor,并调用 pool.execute(new MessageLoop),初始化了一个MessageLoop,我们来看下MessageLoop
/** Message loop used for dispatching messages. */
private class MessageLoop extends Runnable {
override def run(): Unit = {
try {
while (true) {
try {
val data = receivers.take() // 从receivers中拿出一个EndpointData
if (data == PoisonPill) { // 如果是毒药丸就放回去(毒药丸是空数据,代表队列中没有EndpointData)
// Put PoisonPill back so that other MessageLoops can see it.
receivers.offer(PoisonPill)
return
}
data.inbox.process(Dispatcher.this) // 调用inbox的process方法
} catch {
case NonFatal(e) => logError(e.getMessage, e)
}
}
} catch {
case ie: InterruptedException => // exit
}
}
}
/** A poison endpoint that indicates MessageLoop should exit its message loop. */
private val PoisonPill = new EndpointData(null, null, null)
5.1 从receivers中拿出一个EndpointData 判断是否是毒药丸,如果是则放回去,循环结束
5.2 如果不是毒药丸,则调用inbox.process(Dispatcher)处理EndpointData
下面我们看下Inbox初始化时都干了啥:
/**
* An inbox that stores messages for an [[RpcEndpoint]] and posts messages to it thread-safely.
*/
private[netty] class Inbox(
val endpointRef: NettyRpcEndpointRef,
val endpoint: RpcEndpoint)
extends Logging {
inbox => // Give this an alias so we can use it more clearly in closures.
@GuardedBy("this")
protected val messages = new java.util.LinkedList[InboxMessage]()
/** True if the inbox (and its associated endpoint) is stopped. */
@GuardedBy("this")
private var stopped = false
/** Allow multiple threads to process messages at the same time. */
@GuardedBy("this")
private var enableConcurrent = false
/** The number of threads processing messages for this inbox. */
@GuardedBy("this")
private var numActiveThreads = 0
// OnStart should be the first message to process
inbox.synchronized {
messages.add(OnStart)
}
5.2.1 声明一个messages:java.util.LinkedList[InboxMessage]
5.2.2 声明变量stopped 标记inbox是否已经停止,默认false
5.2.3 声明变量enableConcurrent标记inbox是否允许并发,默认false
5.2.4 声明变量numActiveThreads表示当前活动线程数,默认0
5.2.5 向messages中添加一个OnStart对象
下面我们看下inbox.process(Dispatcher)方法:
/**
* Process stored messages.
*/
def process(dispatcher: Dispatcher): Unit = {
var message: InboxMessage = null
inbox.synchronized {
if (!enableConcurrent && numActiveThreads != 0) {
return
}
message = messages.poll()
if (message != null) {
numActiveThreads += 1
} else {
return
}
}
5.2.6 声明变量var message: InboxMessage = null
5.2.7 判断是否是“不允许多线程处理消息”并且“当前的活动线程数不为零”,满足条件则方法结束
5.2.8 如果允许多线程处理且至少有一个活动线程,则从messages取出一条message
5.2.9 判断取出的message是否非null,是则活动线程数加1,否则方法结束。
while (true) {
safelyCall(endpoint) {
message match {
case RpcMessage(_sender, content, context) =>
try {
endpoint.receiveAndReply(context).applyOrElse[Any, Unit](content, { msg =>
throw new SparkException(s"Unsupported message $message from ${_sender}")
})
} catch {
case NonFatal(e) =>
context.sendFailure(e)
// Throw the exception -- this exception will be caught by the safelyCall function.
// The endpoint's onError function will be called.
throw e
}
case OneWayMessage(_sender, content) =>
endpoint.receive.applyOrElse[Any, Unit](content, { msg =>
throw new SparkException(s"Unsupported message $message from ${_sender}")
})
case OnStart =>
endpoint.onStart()
if (!endpoint.isInstanceOf[ThreadSafeRpcEndpoint]) {
inbox.synchronized {
if (!stopped) {
enableConcurrent = true
}
}
}
case OnStop =>
val activeThreads = inbox.synchronized { inbox.numActiveThreads }
assert(activeThreads == 1,
s"There should be only a single active thread but found $activeThreads threads.")
dispatcher.removeRpcEndpointRef(endpoint)
endpoint.onStop()
assert(isEmpty, "OnStop should be the last message")
case RemoteProcessConnected(remoteAddress) =>
endpoint.onConnected(remoteAddress)
case RemoteProcessDisconnected(remoteAddress) =>
endpoint.onDisconnected(remoteAddress)
case RemoteProcessConnectionError(cause, remoteAddress) =>
endpoint.onNetworkError(cause, remoteAddress)
}
}
inbox.synchronized {
// "enableConcurrent" will be set to false after `onStop` is called, so we should check it
// every time.
if (!enableConcurrent && numActiveThreads != 1) {
// If we are not the only one worker, exit
numActiveThreads -= 1
return
}
message = messages.poll()
if (message == null) {
numActiveThreads -= 1
return
}
}
}
}
5.2.5 根据message类型进行模式匹配,根据不同数据类型进行相应的处理。
5.2.6 判断是否是“不允许多线程处理消息”并且“当前的活动线程数不为1”,满足条件则当前活动线程数减1,循环结束
5.2.7 如果不满足上述条件,则从messages中取出一条message
5.2.8 如果message为null,则当前活动线程数减1,循环结束
致此,Dispatcher的初始化完成,我们来总结一下Dispatcher初始化中干了哪些事:
定义了两个ConcurrentMap(endpoints,endpointRefs) 分别来存放终端名称和终端数据的映射、终端和终端引用的映射,一个receivers:LinkedBlockingQueue[EndpointData] 来存放Dispatcher接收到的EndpointData,一个线程池 threadpool:ThreadPoolExecutor,并执行pool.execute(new MessageLoop), pool.execute(new MessageLoop)方法不断的从receivers中拿数据,如果不是空数据,则调用inbox.process方法,inbox.process方法则根据messages中的数据,进行相应的处理操作。