我们继续了解SparkContext中的_heartbeatReceiver: RpcEndpointRef,其初始化代码如下:
// We need to register "HeartbeatReceiver" before "createTaskScheduler" because Executor will
// retrieve "HeartbeatReceiver" in the constructor. (SPARK-6640)
_heartbeatReceiver = env.rpcEnv.setupEndpoint(
HeartbeatReceiver.ENDPOINT_NAME, new HeartbeatReceiver(this))
在上两节中已经了解了RpcEnv了,_heartbeatReceiver是端点引用类型RpcEndpointRef,类HeartbeatReceiver是RpcEndpoint类型。
- HeartbeatReceiver
类HeartbeatReceiver具有伴生对象,伴生对象仅有成员val ENDPOINT_NAME = “HeartbeatReceiver”。
HeartbeatReceiver所在文件为HeartbeatReceiver.scala,该文件开始部分包含一些HeartbeatReceiver接受/返回的消息类的定义: Heartbeat表示executors发给driver的心跳信息; TaskSchedulerIsSet是SparkContext通知HeartbeatReceiver; taskScheduler已经创建好; ExpireDeadHosts用于通知HeartbeatReceiver将已经没有心跳的Host销毁掉; ExecutorRegistered, ExecutorRemoved表示Executor的注册和移除; HeartbeatResponse是HeartbeatReceiver处理Heartbeat消息后返回的消息。
这些类的代码如下:
package org.apache.spark
......
/**
* A heartbeat from executors to the driver. This is a shared message used by several internal
* components to convey liveness or execution information for in-progress tasks. It will also
* expire the hosts that have not heartbeated for more than spark.network.timeout.
* spark.executor.heartbeatInterval should be significantly less than spark.network.timeout.
*/
private[spark] case class Heartbeat(
executorId: String,
accumUpdates: Array[(Long, Seq[AccumulatorV2[_, _]])], // taskId -> accumulator updates
blockManagerId: BlockManagerId)
/**
* An event that SparkContext uses to notify HeartbeatReceiver that SparkContext.taskScheduler is
* created.
*/
private[spark] case object TaskSchedulerIsSet
private[spark] case object ExpireDeadHosts
private case class ExecutorRegistered(executorId: String)
private case class ExecutorRemoved(executorId: String)
private[spark] case class HeartbeatResponse(reregisterBlockManager: Boolean)
先贴下HeartbeatReceiver的源码吧
/**
* Lives in the driver to receive heartbeats from executors..
*/
private[spark] class HeartbeatReceiver(sc: SparkContext, clock: Clock)
extends SparkListener with ThreadSafeRpcEndpoint with Logging {
def this(sc: SparkContext) {
this(