BlockManager初始化和注册，BlockManager核心方法详解

本篇博文的重要内容如下：
1. BlockManager的注册和初始化
2. BlockManager里面的重要函数详细介绍
先对一些基本概念名词介绍：

Block是Spark数据处理的时候最小单位

一：BlockManager初始化
1. BlockManager的实例对象调用initializes的时候才能正常工作。
启动initializes方法又两个作用：BlockTransferService(网络通信)，ShuffleClient

/**
 * Initializes the BlockManager with the given appId. This is not performed in the constructor as
 * the appId may not be known at BlockManager instantiation time (in particular for the driver,
 * where it is only learned after registration with the TaskScheduler).
 *
 * This method initializes the BlockTransferService and ShuffleClient, registers with the
 * BlockManagerMaster, starts the BlockManagerWorker endpoint, and registers with a local shuffle
 * service if configured.
 */

在executor启动的时候通过BlockManager.initialize来实例化Executor的BlockManager。

if (!isLocal) {
  env.metricsSystem.registerSource(executorSource)
  env.blockManager.initialize(conf.getAppId)
}

BlockManager在启动的时候都会向BlockManagerMaster注册。

master.registerBlockManager(blockManagerId, maxMemory, slaveEndpoint)

并且创建BlockManagerSlaveEndpoint这个消息循环体来接受Driver中的BlockManagerMaster发过来的指令，例如删除Block等；

private val slaveEndpoint = rpcEnv.setupEndpoint(
  "BlockManagerEndpoint" + BlockManager.ID_GENERATOR.next,
  new BlockManagerSlaveEndpoint(rpcEnv, this, mapOutputTracker))

下面就具体看一下BlockManagerSlaveEndpoint，从注释里面可以看到， BlockManagerSlaveEndpoint接收BlockManagerMaster发过来的信息。

/**
 * An RpcEndpoint to take commands from the master to execute options. For example,
 * this is used to remove blocks from the slave's BlockManager.
 */

BlockManager注册
1． 通过RegisterBlockManager注册BlockManager

/** Register the BlockManager's id with the driver. */
def registerBlockManager(
    blockManagerId: BlockManagerId, maxMemSize: Long, slaveEndpoint: RpcEndpointRef): Unit = {
  logInfo("Trying to register BlockManager")
  tell(RegisterBlockManager(blockManagerId, maxMemSize, slaveEndpoint))
  logInfo("Registered BlockManager")
}

2.  Tell就将此方法发送给Driver端。

/** Send a one-way message to the master endpoint, to which we expect it to reply with true. */
private def tell(message: Any) {
  if (!driverEndpoint.askWithRetry[Boolean](message)) {
    throw new SparkException("BlockManagerMasterEndpoint returned false, expected true.")
  }
}

3.  当BlockManagerSlaveEndpoint实例化后，Executor上的BlockManager需要向Driver上的BlockManagerMasterEndpoint注册。