SparkCore — BlockManager读取数据

BlockManager的读取数据操作

  BlockManager读取数据分为从本地读取和从远程节点上拉取，我们这里分别分析一下这两种方式。首先分析从本地拉取数据的方法doGetLocal()，在进行分析这个方法之前，我们先看看BlockManager的初始化方法initialize()：

BlockManager初始化initialize

def initialize(appId: String): Unit = {

    // 初始化用于远程通信的blockTransferService
    blockTransferService.init(this)
    shuffleClient.init(appId)

    // 为当前这个BlockManager创建一个唯一的BlockManagerId
    // 参数包括，executorId，每个BlockManager关联一个Executor，以及blockTransferService的hostName和port
    // 从blockManagerId的初始化看出，一个BlockManager是通过一个节点上的executor唯一标识的
    blockManagerId = BlockManagerId(
      executorId, blockTransferService.hostName, blockTransferService.port)

    // 如果使用了堆外内存，那么相应的创建堆外内存的BlockManagerId
    shuffleServerId = if (externalShuffleServiceEnabled) {
      logInfo(s"external shuffle service port = $externalShuffleServicePort")
      BlockManagerId(executorId, blockTransferService.hostName, externalShuffleServicePort)
    } else {
      blockManagerId
    }

    // 使用BlockManagerMasterEndpoint的引用，进行BlockManager的注册
    // 发送消息到BlockManagerMasterEndpoint。
    master.registerBlockManager(blockManagerId, maxMemory, slaveEndpoint)

    // Register Executors' configuration with the local shuffle service, if one should exist.
    // 对堆外内存进行注册
    if (externalShuffleServiceEnabled && !blockManagerId.isDriver) {
      registerWithExternalShuffleServer()
    }
  }

  首先创建用于远程通信的组件blockTransferService，接着创建一个唯一的BlockManagerId，包括参数executorId，每个BlockManager关联一个Executor，以及blockTransferService的hostName和port，从blockManagerId的初始化看出，一个BlockManager是通过一个节点上的executor唯一标识的；假如使用到了堆外内存，那么也要创建一个BlockManagerId；接着就向BlockManagerMaster进行注册。

BlockManager读取本地数据doGetLocal()

private def doGetLocal(blockId: BlockId, asBlockResult: Boolean): Option[Any] = {
    // 首先尝试获取blockId对应的BlockInfo锁
    val info = blockInfo.get(blockId).orNull
    if (info != null) {
      // 对所有的BlockInfo，都会进行并发同步访问，BlockInfo，相当于是一个对Block，
      // 用于作为多线程并发访问同步的监视
      info.synchronized {
        if (blockInfo.get(blockId).isEmpty) {
          logWarning(s"Block $blockId had been removed")
          return None
        }
        // 其余线程操作这个block，会阻塞在这边。
        if (!info.waitForReady()) {
          logWarning(s"Block $blockId was marked as failure.")
          return None
        }

        // 获取持久化级别
        val level = info.level
        logDebug(s"Level for block $blockId is $level")
        
        // 如果持久化级别使用了内存，尝试从memoryStore中获取数据
        if (level.useMemory) {
          logDebug(s"Getting block $blockId from memory")
          val result = if (asBlockResult) {
            memoryStore.getValues(blockId).map(new BlockResult(_, DataReadMethod.Memory, info.size))
          } else {
            memoryStore.getBytes(blockId)
          }
          result match {
            case Some(values) =>
              return result
            case None =>
              logDebug(s"Block $blockId not found in memory")
          }
        }
        // 使用了堆外内存
        ........
       
        // 如果使用了Disk持久化级别
        if (level.useDisk) {
          logDebug(s"Getting block $blockId from disk")
          // 从磁盘中读取数据
          val bytes: ByteBuffer = diskStore.getBytes(blockId) match {
            case Some(b) => b
            case None =>
              throw new BlockException(
                blockId, s"Block $blockId not found on disk, though it should be")
          }
          assert(0 == bytes.position())

          if (!level.useMemory) {
            if (asBlockResult) {
              // 如果仅仅使用了Disk没有使用memory，那么就将数据反序列化，在返回
              return Some(new BlockResult(dataDeserialize(blockId, bytes), DataReadMethod.Disk,
                info.size))
            } else {
              return Some(bytes)
            }
          } else {
            if (!level.deserialized || !asBlockResult) {
              // 如果即使用了Disk级别又使用了memory级别，那么从disk中读取出来之后，会尝试将其放入
              // memorystore中，也就缓存到内存中
              memoryStore.putBytes(blockId, bytes.limit, () => {
                val copyForMemory = ByteBuffer.allocate(bytes.limit)
                copyForMemory.put(bytes)
              })
              bytes.rewind()
            }
            if (!asBlockResult) {
              return Some(bytes)
            } else {
              val values = dataDeserialize(blockId, bytes)
              if (level.deserialized) {
                // Cache the values before returning them
                val putResult = memoryStore.putIterator(
                  blockId, values, level, returnValues = true, allowPersistToDisk = false)
                putResult.data match {
                  case Left(it) =>
                    return Some(new BlockResult(it, DataReadMethod.Disk, info.size))
                  case _ =>
                    // This only happens if we dropped the values back to disk (which is never)
                    throw new SparkException("Memory store did not return an iterator!")
                }
              } else {
                return Some(new BlockResult(values, DataReadMethod.Disk, info.size))
              }
            }
          }
        }
      }
    } else {
      logDebug(s"Block $blockId not registered locally")
    }
    None
  }

  首先获取BlockId对应的锁，为了防止多个executor同时读取。然后获得Block的持久化级别，根据持久化级别进行读取；如果持久化级别使用到了内存，那么就尝试从MemoryStore中获取数据，如果使用堆外内存那么就从堆外内存读取；假设使用了是Disk持久化，那么从磁盘读取，这里会区分是只有Disk还是既有Disk又有Memory，分别进行读取，这里就是依据不同的持久化，从不同的地方获取数据。

BlockManager读取远程数据doGetRemote()

private def doGetRemote(blockId: BlockId, asBlockResult: Boolean): Option[Any] = {
    require(blockId != null, "BlockId is null")
    // 首先从BlockManagerMaster上获取每个BlockManager的信息，然后随机打散
    val locations = Random.shuffle(master.getLocations(blockId))
    // 记录获取失败的次数
    var numFetchFailures = 0
    // 遍历每一个BlockManager
    for (loc <- locations) {
      logDebug(s"Getting remote block $blockId from $loc")
      // 使用blockTransferService进行异步的远程网络获取，将block数据传输回来
      // 连接的时候，使用的BlockManager的唯一标识，host、port、executorId
      val data = try {
        blockTransferService.fetchBlockSync(
          loc.host, loc.port, loc.executorId, blockId.toString).nioByteBuffer()
      } catch {
        // 拉取过程中的出错处理 
        case NonFatal(e) =>
          numFetchFailures += 1
          if (numFetchFailures == locations.size) {
            // An exception is thrown while fetching this block from all locations
            throw new BlockFetchException(s"Failed to fetch block from" +
              s" ${locations.size} locations. Most recent failure cause:", e)
          } else {
            // This location failed, so we retry fetch from a different one by returning null here
            logWarning(s"Failed to fetch remote block $blockId " +
              s"from $loc (failed attempt $numFetchFailures)", e)
            null
          }
      }
      // 封装成BlockResult形式
      if (data != null) {
        if (asBlockResult) {
          return Some(new BlockResult(
            dataDeserialize(blockId, data),
            DataReadMethod.Network,
            data.limit()))
        } else {
          return Some(data)
        }
      }
      logDebug(s"The value of block $blockId is null")
    }
    logDebug(s"Block $blockId not found")
    None
  }

  从BlockManagerMaster上获取每个BlockManager的信息，然后随机打散，接着遍历每个位置信息，使用blockTransferService到远程节点上异步拉取数据，接着将获取到的信息进行封装并返回。
  总结一下，BlockManager获取数据的途径有两个一个是本地，一个是远程拉取，如果是本地方式的话，那么依据持久化级别来获取数据；如果是远程的话，先从BlockManagerMaster上获取数据位置信息，然后与远程节点建立连接拉取数据。