Spark源码学习笔记7-HeartbeatReceiver

最新推荐文章于 2023-04-13 11:17:34 发布

lzy2014

最新推荐文章于 2023-04-13 11:17:34 发布

阅读量3.1k

点赞数 1

分类专栏： spark 文章标签： spark 源码

本文链接：https://blog.csdn.net/lzy2014/article/details/72084211

版权

我们继续了解SparkContext中的_heartbeatReceiver: RpcEndpointRef，其初始化代码如下：

// We need to register "HeartbeatReceiver" before "createTaskScheduler" because Executor will
    // retrieve "HeartbeatReceiver" in the constructor. (SPARK-6640)
    _heartbeatReceiver = env.rpcEnv.setupEndpoint(
      HeartbeatReceiver.ENDPOINT_NAME, new HeartbeatReceiver(this))

在上两节中已经了解了RpcEnv了，_heartbeatReceiver是端点引用类型RpcEndpointRef，类HeartbeatReceiver是RpcEndpoint类型。

HeartbeatReceiver

类HeartbeatReceiver具有伴生对象，伴生对象仅有成员val ENDPOINT_NAME = “HeartbeatReceiver”。

HeartbeatReceiver所在文件为HeartbeatReceiver.scala，该文件开始部分包含一些HeartbeatReceiver接受/返回的消息类的定义： Heartbeat表示executors发给driver的心跳信息； TaskSchedulerIsSet是SparkContext通知HeartbeatReceiver； taskScheduler已经创建好； ExpireDeadHosts用于通知HeartbeatReceiver将已经没有心跳的Host销毁掉； ExecutorRegistered, ExecutorRemoved表示Executor的注册和移除； HeartbeatResponse是HeartbeatReceiver处理Heartbeat消息后返回的消息。
这些类的代码如下：

package org.apache.spark
......
/**
 * A heartbeat from executors to the driver. This is a shared message used by several internal
 * components to convey liveness or execution information for in-progress tasks. It will also
 * expire the hosts that have not heartbeated for more than spark.network.timeout.
 * spark.executor.heartbeatInterval should be significantly less than spark.network.timeout.
 */
private[spark] case class Heartbeat(
    executorId: String,
    accumUpdates: Array[(Long, Seq[AccumulatorV2[_, _]])], // taskId -> accumulator updates
    blockManagerId: BlockManagerId)

/**
 * An event that SparkContext uses to notify HeartbeatReceiver that SparkContext.taskScheduler is
 * created.
 */
private[spark] case object TaskSchedulerIsSet

private[spark] case object ExpireDeadHosts

private case class ExecutorRegistered(executorId: String)

private case class ExecutorRemoved(executorId: String)

private[spark] case class HeartbeatResponse(reregisterBlockManager: Boolean)

先贴下HeartbeatReceiver的源码吧

/**
 * Lives in the driver to receive heartbeats from executors..
 */
private[spark] class HeartbeatReceiver(sc: SparkContext, clock: Clock)
  extends SparkListener with ThreadSafeRpcEndpoint with Logging {

  def this(sc: SparkContext) {
    this(