BlockManager是对外提供的统一访问block的接口，在Master和Slave上都有一个实例，他提供读写数据的方法，并且根据不同StorageLevel调用不同的BlockStore来读写数据。
在应用程序启动的时候，SparkContext会创建Driver端的SpakEnv，在该SparkEnv中实例化BlockManager和BlockManagerMaster，在其内部创建消息通信的终端BlockManagerMasterEndpoint.
在Executor启动的时候，也会创建其SparkEnv，在该SparkEnv中实例化BlockManager和BlockTransferService. 在BlockManager初始化的过程一方面会加入BlockManagerSlaveEndpoint的消息终端，并把该BlockManagerSlaveEndpoint的该终端引用注册到Driver中，这样Driver和Executor就可以相互持有通信终端的引用
// 创建BlockManagerMaster
val blockManagerMaster= new BlockManagerMaster(registerOrLookupEndpoint(
BlockManagerMaster.DRIVER_ENDPOINT_NAME,
// 创建BlockManagerMasterEndpoint
new BlockManagerMasterEndpoint(rpcEnv,isLocal, conf,listenerBus)),
conf, isDriver)
// NB: blockManager is not valid until initialize() is called later.
// 创建BlockManager
val blockManager= new BlockManager(executorId,rpcEnv, blockManagerMaster,
serializer, conf,mapOutputTracker, shuffleManager, blockTransferService, securityManager,
numUsableCores)
一 核心属性
String executorId: executorId或者是driverId
RpcEnv rpcEnv: rpc通信环境
BlockManagerMaster master: 主要负责整个应用程序在运行期间block元数据的管理和维护，和指令的发送
Serializer defaultSerializer： 默认的序列化机制
Long maxMemory：分配的最大可用内存
Int numUsableCores: 可以使用cpu核数
MapOutputTracker mapOutputTracker：map端shuffle过程的输出状态
ShuffleManager shuffleManager： Shuffle管理器
BlockTransferService blockTransferService： 用于远程间传输数据，用于获取上床block
DiskBlockManager diskBlockManager:主要用于管理磁盘上block以及对应文件和目录
TimeStampedHashMap[BlockId, BlockInfo] blockInfo: 构建一个维护<blockId,blockInfo>的映射
Boolean externalBlockStoreInitialized: 是否使用外部存储
MemoryStore memoryStore：内存存储对象
DiskStore diskStore：磁盘存储对象
ExternalBlockStore externalBlockStore：外部寸处对象
BlockManagerId blockManagerId：当前BlockManager对应的id
ShuffleClient shuffleClient:读取其他executor上的shuffle 文件的客户端，可能是外部服务也有可能是标准的数据块传输服务,如果启用了如果启用外部shuffle 服务，则创建ExternalShuffleClient否则创建 BlockTransferService
Boolean compressBroadcast:是否对广播数据进行压缩
Boolean compressShuffle:是否压缩map输出文件，一般建议打开，但是如果cpu资源消耗太大，则不建议设置为true
Boolean compressRdds:是否要压缩序列化的RDD分区
Boolean compressShuffleSpill:是否对map端溢写的临时文件进行压缩
BlockManagerSlaveEndpoint slaveEndpoint: 持有的BlockManagerSlaveEndpoint通信终端
二 重要方法
2.1initialize 初始化方法，用指定的application id 实例化BlockManager
# 初始化BlockTransferService，构建rpc server等，BlockTransferService主要是用于跨节点传输数据
# 初始化ShuffleClient，读取其他executor上的shuffle文件的客户端
# 构建blockManagerId
# 构建shuffleServerId
# 向BlockManagerMaster注册BlockManager
# 如果外部Shuffle服务启动并且为Executor节点，则注册外部Shuffle服务
def initialize(appId: String): Unit = {
// 初始化BlockTransferService，构建rpc server等，BlockTransferService主要是用于跨节点传输数据
blockTransferService.init(this)
// 初始化ShuffleClient，读取其他executor上的shuffle文件的客户端
shuffleClient.init(appId)
// 构建blockManagerId
blockManagerId = BlockManagerId(
executorId, blockTransferService.hostName, blockTransferService.port)
// 构建shuffleServerId
shuffleServerId = if (externalShuffleServiceEnabled) {
BlockManagerId(executorId, blockTransferService.hostName, externalShuffleServicePort)
} else {
blockManagerId
}
// 向BlockManagerMaster注册BlockManager，传入了slaveEndpoint,用于和BlockManagerMaster通信
master.registerBlockManager(blockManagerId, maxMemory, slaveEndpoint)
// 如果外部Shuffle服务启动并且为Executor节点，则注册外部Shuffle服务
if (externalShuffleServiceEnabled && !blockManagerId.isDriver) {
registerWithExternalShuffleServer()
}
}
2.2reportAllBlocks 报告BlockManager所有的数据块
private def reportAllBlocks(): Unit = {
logInfo(s"Reporting ${blockInfo.size} blocks to the master.")
for ((blockId, info) <- blockInfo) {
// 获取每一个block当前的状态
val status = getCurrentBlockStatus(blockId, info)
if (!tryToReportBlockStatus(blockId, info, status)) {
logError(s"Failed to report $blockId to master; giving up.")
return
}
}
}
private def getCurrentBlockStatus(blockId: BlockId, info: BlockInfo): BlockStatus = {
info.synchronized {
info.level match {
case null =>
BlockStatus(StorageLevel.NONE, 0L, 0L, 0L)
case level =>
val inMem = level.useMemory && memoryStore.contains(blockId)
val inExternalBlockStore = level.useOffHeap && externalBlockStore.contains(blockId)
val onDisk = level.useDisk && diskStore.contains(blockId)
val deserialized = if (inMem) level.deserialized else false
val replication = if (inMem || inExternalBlockStore || onDisk) level.replication else 1
val storageLevel =
StorageLevel(onDisk, inMem, inExternalBlockStore, deserialized, replication)
val memSize = if (inMem) memoryStore.getSize(blockId) else 0L
val externalBlockStoreSize =
if (inExternalBlockStore) externalBlockStore.getSize(blockId) else 0L
val diskSize = if (onDisk) diskStore.getSize(blockId) else 0L
BlockStatus(storageLevel, memSize, diskSize, externalBlockStoreSize)
}
}
}
2.3reregister 注册BlockManager，并且报告BlockManager所有数据块的状态
def reregister(): Unit = {
// TODO: We might need to rate limit re-registering.
logInfo("BlockManager re-registering with master")
master.registerBlockManager(blockManagerId, maxMemory, slaveEndpoint)
reportAllBlocks()
}
2.4getBlockData 获取本地数据块数据
# 如果是shuffle的数据块，则通过ShuffleBlockResolver获取数据块，否则调用doGetLocal从本地获取
# 结果封装成buffer，然后创建NioManagedBuffer，然后返回
override def getBlockData(blockId: BlockId): ManagedBuffer = {
// 首先判断是不是shuffle的数据块
if (blockId.isShuffle) {
// 如果是shuffle的数据块，则通过ShuffleBlockResolver获取数据块
shuffleManager.shuffleBlockResolver.getBlockData(blockId.asInstanceOf[ShuffleBlockId])
} else {
// 获取本地block数据
val blockBytesOpt = doGetLocal(blockId, asBlockResult = false)
.asInstanceOf[Option[ByteBuffer]]
if (blockBytesOpt.isDefined) {
// 获取buffer，然后创建NioManagedBuffer
val buffer = blockBytesOpt.get
new NioManagedBuffer(buffer)
} else {
throw new BlockNotFoundException(blockId.toString)
}
}
}
# doGetLocal:根据指定的blockId获取本地block数据,如果是存在内存上的直接从内存获取；如果是存储在磁盘上的则从磁盘获取，如果是MEMORY_AND_DISK，先放入内存，再返回数据，下次就可以从内存获取，否则直接返回等
private def doGetLocal(blockId: BlockId, asBlockResult: Boolean): Option[Any] = {
// 根据blockId获取blockInfo
val info = blockInfo.get(blockId).orNull
// 如果blockInfo不为空
if (info != null) {
info.synchronized {
// 再次检查blockInfo是否为空
if (blockInfo.get(blockId).isEmpty) {
logWarning(s"Block $blockId had been removed")
return None
}
// 检查该block是否其他线程正在写，等待该block变为ready状态
if (!info.waitForReady()) {
// If we get here, the block write failed.
logWarning(s"Block $blockId was marked as failure.")
return None
}
// 获取该Block的存储级别
val level = info.level
logDebug(s"Level for block $blockId is $level")
// 如果存储级别是内存，则查找MemoryStore返回数据
if (level.useMemory) {
logDebug(s"Getting block $blockId from memory")
val result = if (asBlockResult) {
memoryStore.getValues(blockId).map(new BlockResult(_, DataReadMethod.Memory, info.size))
} else {
memoryStore.getBytes(blockId)
}
result match {
case Some(values) =>
return result
case None =>
logDebug(s"Block $blockId not found in memory")
}
}
// 如果使用堆外存储，则查找ExternalBlockStore返回数据
if (level.useOffHeap) {
logDebug(s"Getting block $blockId from ExternalBlockStore")
if (externalBlockStore.contains(blockId)) {
val result = if (asBlockResult) {
externalBlockStore.getValues(blockId)
.map(new BlockResult(_, DataReadMethod.Memory, info.size))
} else {
externalBlockStore.getBytes(blockId)
}
result match {
case Some(values) =>
return result
case None =>
logDebug(s"Block $blockId not found in ExternalBlockStore")
}
}
}
// 如果使用磁盘存储，则查找DiskStore返回数据
if (level.useDisk) {
logDebug(s"Getting block $blockId from disk")
val bytes: ByteBuffer = diskStore.getBytes(blockId) match {
case Some(b) => b
case None =>
throw new BlockException(
blockId, s"Block $blockId not found on disk, though it should be")
}
assert(0 == bytes.position())
// 如果不能使用内存存储，则直接返回结果
if (!level.useMemory) {
// If the block shouldn't be stored in memory, we can just return it
if (asBlockResult) {
return Some(new BlockResult(dataDeserialize(blockId, bytes), DataReadMethod.Disk,
info.size))
} else {
return Some(bytes)
}
} else {
// 若可以存入内存，将查询出来的数据放入内存，这样下次再查找该block数据直接从内存获取，以提高速度
if (!level.deserialized || !asBlockResult) {
memoryStore.putBytes(blockId, bytes.limit, () => {
val copyForMemory = ByteBuffer.allocate(bytes.limit)
// 如果内存放不下该block数据，将会发生OOM，因此放入ByteBuffer并且懒加载
copyForMemory.put(bytes)
})
bytes.rewind()
}
if (!asBlockResult) {
return Some(bytes)
} else {// 如果想返回BlockResult对象
// 将字节数据数据反序列化
val values = dataDeserialize(blockId, bytes)
// 如果存储级别是反序列化
if (level.deserialized) {
// 再返回之前放入内存缓存
val putResult = memoryStore.putIterator(
blockId, values, level, returnValues = true, allowPersistToDisk = false)
putResult.data match {
case Left(it) =>
return Some(new BlockResult(it, DataReadMethod.Disk, info.size))
case _ =>
// This only happens if we dropped the values back to disk (which is never)
throw new SparkException("Memory store did not return an iterator!")
}
} else {
return Some(new BlockResult(values, DataReadMethod.Disk, info.size))
}
}
}
}
}
} else {
logDebug(s"Block $blockId not registered locally")
}
None
}
2.5getLocal 从本地BlockManager获取数据
def getLocal(blockId: BlockId): Option[BlockResult] = {
logDebug(s"Getting local block $blockId")
doGetLocal(blockId, asBlockResult = true).asInstanceOf[Option[BlockResult]]
}
2.6getLocalgetLocalBytes从本地BlockManager获取数据并且序列化结果
def getLocalBytes(blockId: BlockId): Option[ByteBuffer] = {
logDebug(s"Getting local block $blockId as bytes")
if (blockId.isShuffle) {
val shuffleBlockResolver = shuffleManager.shuffleBlockResolver
// TODO: This should gracefully handle case where local block is not available. Currently
// downstream code will throw an exception.
Option(
shuffleBlockResolver.getBlockData(blockId.asInstanceOf[ShuffleBlockId]).nioByteBuffer())
} else {
doGetLocal(blockId, asBlockResult = false).asInstanceOf[Option[ByteBuffer]]
}
}
2.7doGetRemote 从远端获取数据
private def doGetRemote(blockId: BlockId, asBlockResult: Boolean): Option[Any] = {
require(blockId != null, "BlockId is null")
// 从blockId对应的BlockManagerId，然后打乱顺序
val locations = Random.shuffle(master.getLocations(blockId))
// 遍历每一个BlockManagerId,调用blockTransferService的fetchBlockSync去拉取数据，返回
for (loc <- locations) {
logDebug(s"Getting remote block $blockId from $loc")
val data = blockTransferService.fetchBlockSync(
loc.host, loc.port, loc.executorId, blockId.toString).nioByteBuffer()
if (data != null) {
if (asBlockResult) {
return Some(new BlockResult(
dataDeserialize(blockId, data),
DataReadMethod.Network,
data.limit()))
} else {
return Some(data)
}
}
logDebug(s"The value of block $blockId is null")
}
logDebug(s"Block $blockId not found")
None
}
2.8doPut根据StorageLevel存放数据
private def doPut(blockId: BlockId, data: BlockValues,
level: StorageLevel, tellMaster: Boolean = true,
effectiveStorageLevel: Option[StorageLevel] = None)
: Seq[(BlockId, BlockStatus)] = {
// 检查BlockId，StorageLevel是否为空
require(blockId != null, "BlockId is null")
require(level != null && level.isValid, "StorageLevel is null or invalid")
effectiveStorageLevel.foreach { level =>
require(level != null && level.isValid, "Effective StorageLevel is null or invalid")
}
// 新建数组，存放blockId和block状态
val updatedBlocks = new ArrayBuffer[(BlockId, BlockStatus)]
// 获取BlockInfo
val putBlockInfo = {
// 创建BlockInfo对象
val tinfo = new BlockInfo(level, tellMaster)
// 将该blockId和创建BlockInfo对象放入内存，并且返回更新key之对应的value之前的value，即
// 如果存在该key对应的value，则返回，如果没有则直接存入，返回为空，即不能进行更新
val oldBlockOpt = blockInfo.putIfAbsent(blockId, tinfo)
// 如果已经有旧的BlockInfo，则判断是否正准备写，如果准备些则直接返回更新的(BlockId, BlockStatus)数组
if (oldBlockOpt.isDefined) {
if (oldBlockOpt.get.waitForReady()) {
logWarning(s"Block $blockId already exists on this machine; not re-adding it")
return updatedBlocks
}
oldBlockOpt.get
} else {
tinfo // 否则直接返回blockInfo
}
}
val startTimeMs = System.currentTimeMillis
var valuesAfterPut: Iterator[Any] = null
var bytesAfterPut: ByteBuffer = null
// 数据块大小
var size = 0L
// 存储的级别
val putLevel = effectiveStorageLevel.getOrElse(level)
// 启动一个线程在本地存储之前，异步初始化好要进行的备份的数据，这有助于提高发送数据的速度
val replicationFuture = data match {
case b: ByteBufferValues if putLevel.replication > 1 =>
// 复制出一个新的buffer
val bufferView = b.buffer.duplicate()
Future {
// 拷贝数据到其他节点
replicate(blockId, bufferView, putLevel)
}(futureExecutionContext)
case _ => null
}
// 防止其他线程put这个block,所以需要使用同步操作指导marked置为true
putBlockInfo.synchronized {
logTrace("Put for block %s took %s to get into synchronized block"
.format(blockId, Utils.getUsedTimeMs(startTimeMs)))
var marked = false
try {
// returnValues:是否返回put操作的值
// blockStore：存储方式
val (returnValues, blockStore: BlockStore) = {
// 内存存储返回true和MemoryStore
if (putLevel.useMemory) {
(true, memoryStore)
} else if (putLevel.useOffHeap) {
//对外存储返回false和ExternalBlockStore
(false, externalBlockStore)
} else if (putLevel.useDisk) {
// 磁盘存储，如果复制因子大于1返回true和DiskStore，否则返回false和DiskStore
(putLevel.replication > 1, diskStore)
} else {
assert(putLevel == StorageLevel.NONE)
throw new BlockException(
blockId, s"Attempted to put block $blockId without specifying storage level!")
}
}
// 匹配put操作存入的值类型，调用blockStore方法
val result = data match {
// 可迭代值类型，调用BlockStore的putIterator方法
case IteratorValues(iterator) =>
blockStore.putIterator(blockId, iterator, putLevel, returnValues)
// 数组类型，调用BlockStore的putArray方法
case ArrayValues(array) =>
blockStore.putArray(blockId, array, putLevel, returnValues)
// ByteBufferValues类型,调用BlockStore的putBytes方法
case ByteBufferValues(bytes) =>
bytes.rewind()
blockStore.putBytes(blockId, bytes, putLevel)
}
// 获取结果大小
size = result.size
result.data match {
case Left (newIterator) if putLevel.useMemory => valuesAfterPut = newIterator
case Right (newBytes) => bytesAfterPut = newBytes
case _ =>
}
// 如果使用内存存储，遍历result结果中的droppedBlocks,将溢出到磁盘的block添加到updateBlock集合中
if (putLevel.useMemory) {
result.droppedBlocks.foreach { updatedBlocks += _ }
}
// 获取当前block状态
val putBlockStatus = getCurrentBlockStatus(blockId, putBlockInfo)
if (putBlockStatus.storageLevel != StorageLevel.NONE) {
// 将marked标记置为true
marked = true
// 调用blockInfo的markReady标记该block写操作完成
putBlockInfo.markReady(size)
// 向Master汇报block状态
if (tellMaster) {
reportBlockStatus(blockId, putBlockInfo, putBlockStatus)
}
// 更新updatedBlocks
updatedBlocks += ((blockId, putBlockStatus))
}
} finally {
// If we failed in putting the block to memory/disk, notify other possible readers
// that it has failed, and then remove it from the block info map.
if (!marked) {
// Note that the remove must happen before markFailure otherwise another thread
// could've inserted a new BlockInfo before we remove it.
blockInfo.remove(blockId)
putBlockInfo.markFailure()
logWarning(s"Putting block $blockId failed")
}
}
}
logDebug("Put block %s locally took %s".format(blockId, Utils.getUsedTimeMs(startTimeMs)))
// 如果复制因子大于1，开始异步复制数据
if (putLevel.replication > 1) {
data match {
case ByteBufferValues(bytes) =>
if (replicationFuture != null) {
Await.ready(replicationFuture, Duration.Inf)
}
case _ =>
val remoteStartTime = System.currentTimeMillis
// Serialize the block if not already done
if (bytesAfterPut == null) {
if (valuesAfterPut == null) {
throw new SparkException(
"Underlying put returned neither an Iterator nor bytes! This shouldn't happen.")
}
bytesAfterPut = dataSerialize(blockId, valuesAfterPut)
}
replicate(blockId, bytesAfterPut, putLevel)
logDebug("Put block %s remotely took %s"
.format(blockId, Utils.getUsedTimeMs(remoteStartTime)))
}
}
// 如果是基于内存映射的，则开始清理ByteBuffer
BlockManager.dispose(bytesAfterPut)
if (putLevel.replication > 1) {
logDebug("Putting block %s with replication took %s"
.format(blockId, Utils.getUsedTimeMs(startTimeMs)))
} else {
logDebug("Putting block %s without replication took %s"
.format(blockId, Utils.getUsedTimeMs(startTimeMs)))
}
updatedBlocks
}
2.9getPeers 获取集群中非当前BlockManagerId和非Driver端的BlockManagerId的所有BlockManagerId
private def getPeers(forceFetch: Boolean): Seq[BlockManagerId] = {
peerFetchLock.synchronized {
val cachedPeersTtl = conf.getInt("spark.storage.cachedPeersTtl", 60 * 1000) // milliseconds
val timeout = System.currentTimeMillis - lastPeerFetchTime > cachedPeersTtl
if (cachedPeers == null || forceFetch || timeout) {
cachedPeers = master.getPeers(blockManagerId).sortBy(_.hashCode)
lastPeerFetchTime = System.currentTimeMillis
logDebug("Fetched peers from master: " + cachedPeers.mkString("[", ",", "]"))
}
cachedPeers
}
}
2.10replicate 复制数据块到其他节点
private def replicate(blockId: BlockId, data: ByteBuffer, level: StorageLevel): Unit = {
// 所允许最大复制失败次数
val maxReplicationFailures = conf.getInt("spark.storage.maxReplicationFailures", 1)
// 需要复制到其他的BlockManagerId数量
val numPeersToReplicateTo = level.replication - 1
// 需要被复制的BlockManagerId数组
val peersForReplication = new ArrayBuffer[BlockManagerId]
// 需要复制到其他的BlockManagerId数组
val peersReplicatedTo = new ArrayBuffer[BlockManagerId]
// 复制失败的BlockManagerId数组
val peersFailedToReplicateTo = new ArrayBuffer[BlockManagerId]
// 创建存储级别
val tLevel = StorageLevel(
level.useDisk, level.useMemory, level.useOffHeap, level.deserialized, 1)
val startTime = System.currentTimeMillis
val random = new Random(blockId.hashCode)
var replicationFailed = false
var failures = 0
var done = false
// 获取Executor非当前BlockManagerId集合
peersForReplication ++= getPeers(forceFetch = false)
// 获取一个随机的BlockManagerId
def getRandomPeer(): Option[BlockManagerId] = {
// 如果复制失败
if (replicationFailed) {
// 清理数组
peersForReplication.clear()
// 把获取Executor非当前BlockManagerId集合放入该集合
peersForReplication ++= getPeers(forceFetch = true)
// 移除掉需要复制到那个BlockManagerId
peersForReplication --= peersReplicatedTo
// 移除掉失败复制到那个BlockManagerId
peersForReplication --= peersFailedToReplicateTo
}
// 如果需要被复制的BlockManagerId集合部位空，则则随机取出一个BlockManagerId
if (!peersForReplication.isEmpty) {
Some(peersForReplication(random.nextInt(peersForReplication.size)))
} else {
None
}
}
// 如果复制还没完成，则不断地循环
while (!done) {
// 获取随机的一个BlockManagerId
getRandomPeer() match {
case Some(peer) =>
try {
val onePeerStartTime = System.currentTimeMillis
data.rewind()
logTrace(s"Trying to replicate $blockId of ${data.limit()} bytes to $peer")
// 调用BlockTransferService的uploadBlockSync方法，同步上床block
blockTransferService.uploadBlockSync(
peer.host, peer.port, peer.executorId, blockId, new NioManagedBuffer(data), tLevel)
logTrace(s"Replicated $blockId of ${data.limit()} bytes to $peer in %s ms"
.format(System.currentTimeMillis - onePeerStartTime))
// 更新复制到的BlockManagerId的集合
peersReplicatedTo += peer
// 被复制到的BlockManagerId的集合移除掉这个BlockManagerId，避免下一次复制到一个BlockManager
peersForReplication -= peer
replicationFailed = false
// 如果已经复制的数量等于需要需要复制的数量，则表示复制完成
if (peersReplicatedTo.size == numPeersToReplicateTo) {
done = true
}
} catch {
case e: Exception =>
logWarning(s"Failed to replicate $blockId to $peer, failure #$failures", e)
failures += 1
replicationFailed = true
peersFailedToReplicateTo += peer
if (failures > maxReplicationFailures) { // too many failures in replcating to peers
done = true
}
}
case None => // no peer left to replicate to
done = true
}
}
val timeTakeMs = (System.currentTimeMillis - startTime)
logDebug(s"Replicating $blockId of ${data.limit()} bytes to " +
s"${peersReplicatedTo.size} peer(s) took $timeTakeMs ms")
if (peersReplicatedTo.size < numPeersToReplicateTo) {
logWarning(s"Block $blockId replicated to only " +
s"${peersReplicatedTo.size} peer(s) instead of $numPeersToReplicateTo peers")
}
}
2.11dropFromMemory 从内存中放弃某一个block,可能内存已经满了，放在磁盘比较合适等
def dropFromMemory(
blockId: BlockId,
data: () => Either[Array[Any], ByteBuffer]): Option[BlockStatus] = {
logInfo(s"Dropping block $blockId from memory")
// 获取BlockInfo
val info = blockInfo.get(blockId).orNull
// If the block has not already been dropped
if (info != null) {
info.synchronized {
if (!info.waitForReady()) {
logWarning(s"Block $blockId was marked as failure. Nothing to drop")
return None
} else if (blockInfo.get(blockId).isEmpty) {
logWarning(s"Block $blockId was already dropped.")
return None
}
var blockIsUpdated = false
val level = info.level
// 存储级别是磁盘且磁盘不包括这个block,则存入磁盘
if (level.useDisk && !diskStore.contains(blockId)) {
logInfo(s"Writing block $blockId to disk")
data() match {
case Left(elements) =>
diskStore.putArray(blockId, elements, level, returnValues = false)
case Right(bytes) =>
diskStore.putBytes(blockId, bytes, level)
}
blockIsUpdated = true
}
// 如果内存包含这个blockId,则获取这个blocl大小
val droppedMemorySize =
if (memoryStore.contains(blockId)) memoryStore.getSize(blockId) else 0L
// MemoryStore移除这个block
val blockIsRemoved = memoryStore.remove(blockId)
if (blockIsRemoved) {
blockIsUpdated = true
} else {
logWarning(s"Block $blockId could not be dropped from memory as it does not exist")
}
// 重新获取当前block状态
val status = getCurrentBlockStatus(blockId, info)
if (info.tellMaster) {
// 向master报告数据块状态
reportBlockStatus(blockId, info, status, droppedMemorySize)
}
// 如果存储级别不是磁盘，则<blockId.,blockInfo>移除这个blockId
if (!level.useDisk) {
// The block is completely gone from this node; forget it so we can put() it again later.
blockInfo.remove(blockId)
}
if (blockIsUpdated) {
return Some(status)
}
}
}
None
}
2.12removeRdd 删除所有数据当前RDD的数据块
def removeRdd(rddId: Int): Int = {
// TODO: Avoid a linear scan by creating another mapping of RDD.id to blocks.
logInfo(s"Removing RDD $rddId")
val blocksToRemove = blockInfo.keys.flatMap(_.asRDDId).filter(_.rddId == rddId)
blocksToRemove.foreach { blockId => removeBlock(blockId, tellMaster = false) }
blocksToRemove.size
}
2.13removeBlock 从内存和磁盘中移除block
def removeBlock(blockId: BlockId, tellMaster: Boolean = true): Unit = {
logDebug(s"Removing block $blockId")
// 根据blockId获取blockInfo
val info = blockInfo.get(blockId).orNull
if (info != null) {
info.synchronized {
// 内存移除这个block
val removedFromMemory = memoryStore.remove(blockId)
// 磁盘移除这个block
val removedFromDisk = diskStore.remove(blockId)
// ExternalBlockStore移除这个block
val removedFromExternalBlockStore =
if (externalBlockStoreInitialized) externalBlockStore.remove(blockId) else false
if (!removedFromMemory && !removedFromDisk && !removedFromExternalBlockStore) {
logWarning(s"Block $blockId could not be removed as it was not found in either " +
"the disk, memory, or external block store")
}
// <blockId,blockInfo>开始移除这个blockId
blockInfo.remove(blockId)
// 如果需要，则向master报告数据块状态
if (tellMaster && info.tellMaster) {
val status = getCurrentBlockStatus(blockId, info)
reportBlockStatus(blockId, info, status)
}
}
} else {
// The block has already been removed; do nothing.
logWarning(s"Asked to remove block $blockId, which does not exist")
}
}
198
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
