文章目录
概述
介绍StaticMemoryManager的创建以及内存的申请、释放等。
前言
这部分内容相对较为简单,只需要知道这是Spark 1.6之前的唯一内存管理器,现在为了兼容性,依旧保留,但是默认使用的是统一内存管理器。
关于原理,尤其是内存分配,在Spark内存管理之堆内/堆外内存原理详解中,我们已经详细介绍过了,而源码其实与Spark 内存管理之UnifiedMemoryManager中内存分配部分差不多,只是少了动态分配而已,这里不会细讲,只是附上源码。
1. 创建StaticMemoryManager
查看说明:
查看类定义:
相对于UnifiedMemoryManager 构造时调用工厂方法apply( )
,StaticMemoryManager类中有一个this()方法,用于构造此对象(两个方法在后面):
而由于StaticMemoryManager不支持堆外存储内存,因此会进行一下操作,将堆外存储内存转化为堆外执行内存:
最终内存分配情况如下
this()方法中用到的两个方法StaticMemoryManager.getMaxExecutionMemory()
、StaticMemoryManager.getMaxStorageMemory()
:
1.1 StaticMemoryManager.getMaxStorageMemory()
1.2 StaticMemoryManager.getMaxExecutionMemory()
2. 实现内存操作方法
2.1 acquireStorageMemory()
由于StaticMemoryManager不支持堆外存储内存,因此只能从堆内存储内存来获取内存。
2.2 acquireExecutionMemory()
判断申请的执行内存种类,交给对应的 堆内/堆外 执行内存池 去申请。
2.3 acquireUnrollMemory()
由于静态方式下,StorageMemory是固定的,因此我们要限制UnrollMemory内存的大小。
总结
介绍StaticMemoryManager的创建以及内存的申请、释放等。
参考
附录
/**
*一个[[MemoryManager]]将堆空间静态划分为不相交的区域。
*
*执行区和存储区的大小分别通过`spark.shuffle.memoryFraction`和`spark.storage.memoryFraction`确定。
* 两个区域完全分开,因此任何一个使用都不能从另一个借用内存。
*/
private[spark] class StaticMemoryManager(
conf: SparkConf,
maxOnHeapExecutionMemory: Long,
override val maxOnHeapStorageMemory: Long,
numCores: Int)
extends MemoryManager(
conf,
numCores,
maxOnHeapStorageMemory,
maxOnHeapExecutionMemory) {
def this(conf: SparkConf, numCores: Int) {
this(
conf,
StaticMemoryManager.getMaxExecutionMemory(conf),
StaticMemoryManager.getMaxStorageMemory(conf),
numCores)
}
// StaticMemoryManager不支持堆外存储内存:
offHeapExecutionMemoryPool.incrementPoolSize(offHeapStorageMemoryPool.poolSize)
offHeapStorageMemoryPool.decrementPoolSize(offHeapStorageMemoryPool.poolSize)
override def maxOffHeapStorageMemory: Long = 0L
override def acquireStorageMemory(
blockId: BlockId,
numBytes: Long,
memoryMode: MemoryMode): Boolean = synchronized {
require(memoryMode != MemoryMode.OFF_HEAP,
"StaticMemoryManager does not support off-heap storage memory")
if (numBytes > maxOnHeapStorageMemory) {
// Fail fast if the block simply won't fit
logInfo(s"Will not store $blockId as the required space ($numBytes bytes) exceeds our " +
s"memory limit ($maxOnHeapStorageMemory bytes)")
false
} else {
onHeapStorageMemoryPool.acquireMemory(blockId, numBytes)
}
}
// unRoll时值得驱逐的最大字节数
private val maxUnrollMemory: Long = {
(maxOnHeapStorageMemory * conf.getDouble("spark.storage.unrollFraction", 0.2)).toLong
}
override def acquireUnrollMemory(
blockId: BlockId,
numBytes: Long,
memoryMode: MemoryMode): Boolean = synchronized {
require(memoryMode != MemoryMode.OFF_HEAP,
"StaticMemoryManager does not support off-heap unroll memory")
val currentUnrollMemory = onHeapStorageMemoryPool.memoryStore.currentUnrollMemory
val freeMemory = onHeapStorageMemoryPool.memoryFree
// 展开时,我们将使用所有现有的可用内存,并在必要时使用一些额外的空间来免除缓存的块。
// 我们必须限制要通过展开而退出的内存量,否则展开一个大块可能会耗尽整个缓存。
val maxNumBytesToFree = math.max(0, maxUnrollMemory - currentUnrollMemory - freeMemory)
// Keep it within the range 0 <= X <= maxNumBytesToFree
val numBytesToFree = math.max(0, math.min(maxNumBytesToFree, numBytes - freeMemory))
onHeapStorageMemoryPool.acquireMemory(blockId, numBytes, numBytesToFree)
}
private[memory]
override def acquireExecutionMemory(
numBytes: Long,
taskAttemptId: Long,
memoryMode: MemoryMode): Long = synchronized {
memoryMode match {
case MemoryMode.ON_HEAP => onHeapExecutionMemoryPool.acquireMemory(numBytes, taskAttemptId)
case MemoryMode.OFF_HEAP => offHeapExecutionMemoryPool.acquireMemory(numBytes, taskAttemptId)
}
}
}
private[spark] object StaticMemoryManager {
private val MIN_MEMORY_BYTES = 32 * 1024 * 1024
/**
* 返回可用于存储区域的内存总量,以字节为单位。
*/
private def getMaxStorageMemory(conf: SparkConf): Long = {
val systemMaxMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)
val memoryFraction = conf.getDouble("spark.storage.memoryFraction", 0.6)
val safetyFraction = conf.getDouble("spark.storage.safetyFraction", 0.9)
(systemMaxMemory * memoryFraction * safetyFraction).toLong
}
/**
* 返回可用于执行区域的内存总量(以字节为单位)。
*/
private def getMaxExecutionMemory(conf: SparkConf): Long = {
val systemMaxMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)
if (systemMaxMemory < MIN_MEMORY_BYTES) {
throw new IllegalArgumentException(s"System memory $systemMaxMemory must " +
s"be at least $MIN_MEMORY_BYTES. Please increase heap size using the --driver-memory " +
s"option or spark.driver.memory in Spark configuration.")
}
if (conf.contains("spark.executor.memory")) {
val executorMemory = conf.getSizeAsBytes("spark.executor.memory")
if (executorMemory < MIN_MEMORY_BYTES) {
throw new IllegalArgumentException(s"Executor memory $executorMemory must be at least " +
s"$MIN_MEMORY_BYTES. Please increase executor memory using the " +
s"--executor-memory option or spark.executor.memory in Spark configuration.")
}
}
val memoryFraction = conf.getDouble("spark.shuffle.memoryFraction", 0.2)
val safetyFraction = conf.getDouble("spark.shuffle.safetyFraction", 0.8)
(systemMaxMemory * memoryFraction * safetyFraction).toLong
}
}