注:spark版本2.1.1,启动模式:Standalone ,需要启动Master和Worker守护进程
一、脚本分析
start-all.sh中会直接启动start-master.sh
start-master.sh
二、源码解析
org.apache.spark.deploy.master.Master
1、Master主类进入main方法,main方法主要是创建RPC环境
//主方法
def main(argStrings: Array[String]) {
Thread.setDefaultUncaughtExceptionHandler(new SparkUncaughtExceptionHandler(
exitOnUncaughtException = false))
Utils.initDaemon(log)
val conf = new SparkConf
val args = new MasterArguments(argStrings, conf)
// 创建RPC 环境和Endpoint (RPC 远程过程调用)
// 在Spark中 Driver, Master ,Worker角色都有各自的Endpoint,相当于各自的通信邮箱。
val (rpcEnv, _, _) = startRpcEnvAndEndpoint(args.host, args.port, args.webUiPort, conf)
//这里会阻塞等待
rpcEnv.awaitTermination()
}
2、此处先了解一下endpoint
RpcEndpoint
表示一个个需要通信的个体(如master,worker,driver),主要根据接收的消息来进行对应的处理。一个RpcEndpoint经历的过程依次是:构建->onStart→receive→onStop。其中onStart在接收任务消息前调用,receive和receiveAndReply分别用来接收另一个RpcEndpoint(也可以是本身)send和ask过来的消息。
RpcEndpointRef
RpcEndpointRef是对远程RpcEndpoint的一个引用。当我们需要向一个具体的RpcEndpoint发送消息时,一般我们需要获取到该RpcEndpoint的引用,然后通过该应用发送消息。
RpcAddress
表示远程的RpcEndpointRef的地址,Host + Port。
RpcEnv
RpcEnv为RpcEndpoint提供处理消息的环境。RpcEnv负责RpcEndpoint整个生命周期的管理,包括:注册endpoint,endpoint之间消息的路由,以及停止endpoint。
3、main函数中的startRpcEnvAndEndpoint()方法创建RpcEnv--------RpcEnv.create()
RpcEnv.create()中会通过NettyRpcEnvFactory类创建一个NettyRpcEnv,RpcEnv只是一个抽象类,NettyRpcEnv是其中一个实现,查看一下工厂类的create方法
private[rpc] class NettyRpcEnvFactory extends RpcEnvFactory with Logging {
def create(config: RpcEnvConfig): RpcEnv = {
val sparkConf = config.conf
// Use JavaSerializerInstance in multiple threads is safe. However, if we plan to support
// KryoSerializer in future, we have to use ThreadLocal to store SerializerInstance
val javaSerializerInstance =
new JavaSerializer(sparkConf).newInstance().asInstanceOf[JavaSerializerInstance]
/**
* 创建nettyRPC通信环境。
* 当new NettyRpcEnv时会做一些初始化:
* Dispatcher:这个对象中有存放消息的队列和消息的转发
* TransportContext:可以创建了NettyRpcHandler
*/
val nettyEnv =
new NettyRpcEnv(sparkConf, javaSerializerInstance, config.advertiseAddress,
config.securityManager, config.numUsableCores)
if (!config.clientMode) {
//启动nettyRPCEnv
val startNettyRpcEnv: Int => (NettyRpcEnv, Int) = { actualPort =>
nettyEnv.startServer(config.bindAddress, actualPort)
(nettyEnv, nettyEnv.address.port)
}
try {
//以上 startNettyRpcEnv 匿名函数在此处会最终被调用,当匿名函数被调用时,重点方法是483行 nettyEnv.startServer 方法
Utils.startServiceOnPort(config.port, startNettyRpcEnv, sparkConf, config.name)._1
} catch {
case NonFatal(e) =>
nettyEnv.shutdown()
throw e
}
}
nettyEnv
}
}
}
4、NettyRpcEnv被实例化(new)后,此处初始化了一个Dispatcher实例对象
/**
* dispatcher 这个对象中有消息队列和消息的循环获取转发
*/
private val dispatcher: Dispatcher = new Dispatcher(this, numUsableCores)
4.1、而在dispatcher中,定义了一个结构体EndpointData
/**
* Dispatcher 当new 时:
* 里面会初始化 MessageLoop 【负责一直读取处理消息】,继承了 Runnable ,会一直运行run 方法,run方法中 会一直从 receivers 中获取消息处理
* 同时还会有个receivers 消息队列,负责存放消息
*/
private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) extends Logging {
//看做一个重要的数据结构,其中inbox是存放数据的结构体,下面会详细查看
private class EndpointData(
val name: String,
val endpoint: RpcEndpoint,
val ref: NettyRpcEndpointRef) {
val inbox = new Inbox(ref, endpoint)
}
//初始化两个线程安全的HashMap,分别用来记录 name到 EndpointData的关联关系,和 RpcEndpoint与 RpcEndpointRef的关联关系
private val endpoints: ConcurrentMap[String, EndpointData] = new ConcurrentHashMap[String, EndpointData]
private val endpointRefs: ConcurrentMap[RpcEndpoint, RpcEndpointRef] = new ConcurrentHashMap[RpcEndpoint, RpcEndpointRef]
// Track the receivers whose inboxes may contain messages.
//一个队列,存放等待处理的 EndpointData
private val receivers = new LinkedBlockingQueue[EndpointData]
4.2、EndpointData属性inbox做什么的
/**
* An inbox that stores messages for an [[RpcEndpoint]] and posts messages to it thread-safely.
*/
private[netty] class Inbox(
val endpointRef: NettyRpcEndpointRef,
val endpoint: RpcEndpoint)
extends Logging {
inbox => // Give this an alias so we can use it more clearly in closures.
@GuardedBy("this")
protected val messages = new java.util.LinkedList[InboxMessage]()
/** True if the inbox (and its associated endpoint) is stopped. */
@GuardedBy("this")
private var stopped = false
/** Allow multiple threads to process messages at the same time. */
@GuardedBy("this")
private var enableConcurrent = false
/** The number of threads processing messages for this inbox. */
@GuardedBy("this")
private var numActiveThreads = 0
// OnStart should be the first message to process
//当注册endpoint时都会调用这个异步方法,messags中放入一个OnStart样例类 消息对象
inbox.synchronized {
messages.add(OnStart)
}
4.3、其中,inbox最重要的一个方式是process
/**
* Process stored messages.
* 这里是当new Dispatcher 时,已经运行了这个process 方法。
*/
def process(dispatcher: Dispatcher): Unit = {
var message: InboxMessage = null
inbox.synchronized {
if (!enableConcurrent && numActiveThreads != 0) {
return
}
message = messages.poll()
if (message != null) {
numActiveThreads += 1
} else {
return
}
}
while (true) {
safelyCall(endpoint) {
message match {
case RpcMessage(_sender, content, context) =>
try {
endpoint.receiveAndReply(context).applyOrElse[Any, Unit](content, { msg =>
throw new SparkException(s"Unsupported message $message from ${_sender}")
})
} catch {
case NonFatal(e) =>
context.sendFailure(e)
// Throw the exception -- this exception will be caught by the safelyCall function.
// The endpoint's onError function will be called.
throw e
}
case OneWayMessage(_sender, content) =>
endpoint.receive.applyOrElse[Any, Unit](content, { msg =>
throw new SparkException(s"Unsupported message $message from ${_sender}")
})
case OnStart =>
//调用Endpoint 的onStart方法
endpoint.onStart()
if (!endpoint.isInstanceOf[ThreadSafeRpcEndpoint]) {
inbox.synchronized {
if (!stopped) {
enableConcurrent = true
}
}
}
case OnStop =>
val activeThreads = inbox.synchronized { inbox.numActiveThreads }
assert(activeThreads == 1,
s"There should be only a single active thread but found $activeThreads threads.")
dispatcher.removeRpcEndpointRef(endpoint)
endpoint.onStop()
assert(isEmpty, "OnStop should be the last message")
case RemoteProcessConnected(remoteAddress) =>
endpoint.onConnected(remoteAddress)
case RemoteProcessDisconnected(remoteAddress) =>
endpoint.onDisconnected(remoteAddress)
case RemoteProcessConnectionError(cause, remoteAddress) =>
endpoint.onNetworkError(cause, remoteAddress)
}
}
inbox.synchronized {
// "enableConcurrent" will be set to false after `onStop` is called, so we should check it
// every time.
if (!enableConcurrent && numActiveThreads != 1) {
// If we are not the only one worker, exit
numActiveThreads -= 1
return
}
message = messages.poll()
if (message == null) {
numActiveThreads -= 1
return
}
}
}
}
4.4Dispatcher同时初始化一个线程池Threadpool
/** Thread pool used for dispatching messages. */
private val threadpool: ThreadPoolExecutor = {
// 计算线程池大小,根据配置文件和当前可用 CPU进程数
val availableCores = if (numUsableCores > 0) numUsableCores else Runtime.getRuntime.availableProcessors()
val numThreads = nettyEnv.conf.get(RPC_NETTY_DISPATCHER_NUM_THREADS).getOrElse(math.max(2, availableCores))
// newDaemonFixedThreadPool线程池,均去执行MessageLoop实例的run方法
val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
for (i <- 0 until numThreads) {
pool.execute(new MessageLoop)
}
pool
}
/** Message loop used for dispatching messages. */
private class MessageLoop extends Runnable {
override def run(): Unit = {
try {
//分发线程从 Dispatcher实例化后就一直循环运行
while (true) {
try {
//取出一条 EndpointData,这些 EndpointData是在注册的时候加入到receivers的,后面会看到相关源码
val data = receivers.take()
if (data == PoisonPill) { // PoisonPill作为结束线程的标志
// Put PoisonPill back so that other MessageLoops can see it.
receivers.offer(PoisonPill)
return
}
// 调用EndpointData结构体重的 inbox.process方法
data.inbox.process(Dispatcher.this)
} catch {
case NonFatal(e) => logError(e.getMessage, e)
}
}
} catch {
case ie: InterruptedException => // exit
}
}
}
/** A poison endpoint that indicates MessageLoop should exit its message loop. */
private val PoisonPill = new EndpointData(null, null, null)
到此、master主要代码跟踪完毕,接下来总结一下具体流程
三、总结
这里只粘贴关键代码
1、master的main方法调用startRpcEnvAndEndpoint初始化一个RpcEnv,也就是rpc通信环境
val rpcEnv = RpcEnv.create(SYSTEM_NAME, host, port, conf, securityMgr)
.....
//创建NettyRpc 环境
new NettyRpcEnvFactory().create(config)
2、NettyRpcEnv被实例化时会实例化一个dispatcher对象和TransportContext对象
//dispatcher 这个对象中有消息队列和消息的循环获取转发
private val dispatcher: Dispatcher = new Dispatcher(this, numUsableCores)
//TransportContext 中会创建 NettyPpcHandler
//TransportContext 这个对象中参数类型 RpcHandler 就是这里的 NettyRpcHandler
private val transportContext = new TransportContext(transportConf, new NettyRpcHandler(dispatcher, this, streamManager))
3、RPC通信环境由NettyRpcEnv实现,NettyRpcEnv被实例化后调用startServer方法启动服务
//启动Rpc 服务
def startServer(bindAddress: String, port: Int): Unit = {.....
4、startServer()方法中,TransportContext的createServer方法会绑定地址和端口,启动NettyRpc服务
server = transportContext.createServer(bindAddress, port, bootstraps)
5、Dispatcher调用registerRpcEndpoint()注册Endpoint后返回一个引用endpointRef
dispatcher.registerRpcEndpoint(
RpcEndpointVerifier.NAME, new RpcEndpointVerifier(this, dispatcher))
6、在registerRpcEndpoint方法中,endpointRef和endpoint被封装为EndpointData放入一个线程安全的ConcurrentMap;同时类型为LinkedBlockingQueue[EndpointData]的receivers集合放入当前EndpointData对象
if (endpoints.putIfAbsent(name, new EndpointData(name, endpoint, endpointRef)) != null) {
......
//获取刚刚封装的EndPointData
val data: EndpointData = endpoints.get(name)
......
//receivers 这个消息队列中放着应该去哪个Endpoint 中获取Message 处理
//这里其实就是进入 Dispatcher 当前这个类中的 MessageLoop 方法。这个方法当new Dispatcher后会一直//运行。将消息放入待处理的消息队列中,消息首先找到对应的Endpoint ,再会获取当前Endpoint的Inbox 中//message,使用process 方法处理
receivers.offer(data) // for the OnStart message
7、EndpointData对象被实例化时Inbox也被实例化,inbox中的process方法在dispatcher对象被实例化后就被执行
private class EndpointData(
val name: String,
val endpoint: RpcEndpoint,
val ref: NettyRpcEndpointRef) {
//将endpoint封装到Inbox中
val inbox = new Inbox(ref, endpoint)}
8、inbox中的process方法中会调用Endpoint的onStart方法
//当注册endpoint时都会调用这个异步方法,messags中放入一个OnStart样例类 消息对象
inbox.synchronized {
messages.add(OnStart)
}
//这里是当new Dispatcher 时,已经运行了这个process 方法。
def process(dispatcher: Dispatcher): Unit = {
var message: InboxMessage = null
......
while (true) {
safelyCall(endpoint) {
message match {
....
case OnStart =>
//调用Endpoint 的onStart方法
endpoint.onStart()
......
这一阶段只是完整的初始化了一个RPCEnv,接下来才会创建master对象
9、rpcEnv.setupEndpoint(name,new Master)向RpcEnv中注册Master,也就是rpcEnv实现类NettyRpcEnv执行setupEndpoint
RpcEnv.setupEndpoint()
val masterEndpoint: RpcEndpointRef = rpcEnv.setupEndpoint(ENDPOINT_NAME,new Master(rpcEnv, rpcEnv.address, webUiPort, securityMgr, conf))
NettyRpcEnv
override def setupEndpoint(name: String, endpoint: RpcEndpoint): RpcEndpointRef = {
dispatcher.registerRpcEndpoint(name, endpoint)
}
10、masterEndpoint.askSync,
askSync调用的是实现类NettyRpcEndpointRef的ask()方法,ask会将消息发送到Endpoint的inbox中
Master.main()
val portsResponse = masterEndpoint.askSync[BoundPortsResponse](BoundPortsRequest)(rpcEnv, portsResponse.webUIPort, portsResponse.restPort)
RpcEndpointRef
def askSync[T: ClassTag](message: Any, timeout: RpcTimeout): T = {
val future = ask[T](message, timeout)
timeout.awaitResult(future)
}
NettyRpcEndpointRef
private[netty] def ask[T: ClassTag](message: RequestMessage, timeout: RpcTimeout):
......
if (remoteAddr == address) {
val p = Promise[Any]()
p.future.onComplete {
case Success(response) => onSuccess(response)
case Failure(e) => onFailure(e)
}(ThreadUtils.sameThread)
dispatcher.postLocalMessage(message, p)
} else {
val rpcMessage = RpcOutboxMessage(message.serialize(this),
.......
11、master类的receive()收到消息匹配类型然后回复
override def receiveAndReply: PartialFunction[Any, Unit] = {
.......
case BoundPortsRequest =>
context.reply(BoundPortsResponse(address.port, webUi.boundPort, restServerBoundPort))
.........