BlockStore

private[spark] abstract class BlockStore(val blockManager: BlockManager) extends Logging {

def putBytes(blockId: BlockId, bytes: ByteBuffer, level: StorageLevel): PutResult

/**
* Put in a block and, possibly, also return its content as either bytes or another Iterator.
* This is used to efficiently write the values to multiple locations (e.g. for replication).
*
* @return a PutResult that contains the size of the data, as well as the values put if
* returnValues is true (if not, the result's data field can be null)
*/
defputIterator(
blockId: BlockId,
values: Iterator[Any],
level: StorageLevel,
returnValues: Boolean): PutResult

def putArray(
blockId: BlockId,
values: Array[Any],
level: StorageLevel,
returnValues: Boolean): PutResult

/**
* Return the size of a block in bytes.
*/
def getSize(blockId: BlockId): Long

def getBytes(blockId: BlockId): Option[ByteBuffer]

def getValues(blockId: BlockId): Option[Iterator[Any]]

/**
* Remove a block, if it exists.
* @param blockId the block to remove.
* @return True if the block was found and removed, False otherwise.
*/
def remove(blockId: BlockId): Boolean

def contains(blockId: BlockId): Boolean

def clear() { }
}


BlockStore 这个接口 实现了 两个功能 Disk

Spark 源码分析 -- BlockStore

BlockStore

抽象接口类, 关键get和put都有两个版本
序列化, putBytes, getBytes
非序列化, putValues, getValues

其中putValues的返回值为PutResult, 其中的data可能是Iterator或ByteBuffer

对应DiskStore其实很单纯, 就是打开相应的文件读或写.

上面奇葩的注释


private def getBytes(file: File, offset: Long, length: Long): Option[ByteBuffer] = {
val channel = new RandomAccessFile(file, "r").getChannel
Utils.tryWithSafeFinally {
// For small files, directly read rather than memory map

我理解这个意思 内存需要 整块对

读小文件 可以从硬盘读一小块

if (length < minMemoryMapBytes) {
val buf = ByteBuffer.allocate(length.toInt)
channel.position(offset)
while (buf.remaining() != 0) {
if (channel.read(buf) == -1) {
throw new IOException("Reached EOF before filling buffer\n" +
s"offset=$offset\nfile=${file.getAbsolutePath}\nbuf.remaining=${buf.remaining}")
}
}
buf.flip()
Some(buf)
} else {
Some(channel.map(MapMode.READ_ONLY, offset, length))
}
} {
channel.close()
}
}