概述
MemoryManager
MemoryManager是spark的内存管理器,它定义了execution和storage之间共享内存的方式。execution memory指的是在shuffle、join、sort和aggregation过程中使用的内存;storage memory指的是缓存RDD和缓存broadcast广播变量占用的内存。每个JVM中都存在一个MemoryManager。
MemoryManager的继承关系
Spark从1.6.0版本开始,内存管理模块就发生了改变,旧版本的内存管理模块使用的是StaticMemoryManager,现在被称为"legacy"。"Legacy"模式默认不可用,即spark.memory.useLegacyMode参数默认为false。spark1.6.0版本后的新的内存管理模型,它实现的是UnifiedMemoryManager
UnifiedMemoryManager
UnifiedMemoryManager定义了Execution memory和storage memory之间的软边界,并且它们可以互相借用内存。
Execution memory和storage memory的总内存大小由spark.memory.fraction参数指定,默认为0.6,它表明了该总内存大小占jvm堆内存大小的比例。进一步地,storage memory的内存大小所占比例由spark.memory.storageFraction指定,默认为0.5。也就是说,默认的storage memory的内存大小为 = jvm heap size * 0.6 * 0.5。
当Execution memory是空闲时,storage memory可以借用Execution memory的内存,直至Execution memory要求回收它的内存。这个时候,storage memory中缓存的block会溢出到磁盘,直至Execution memory已经收回足够的内存。
当storage memory是空闲时,Execution memory也可以借用storage memory的内存,但是Execution memory属于有借无还的角色,即使这时storage memory内存紧张,也不会让出已经借用的内存。
acquireExecutionMemory方法
/**
* Try to acquire up to `numBytes` of execution memory for the current task and return the
* number of bytes obtained, or 0 if none can be allocated.
*
* This call may block until there is enough free memory in some situations, to make sure each
* task has a chance to ramp up to at least 1 / 2N of the total memory pool (where N is the # of
* active tasks) before it is forced to spill. This can happen if the number of tasks increase
* but an older task had a lot of memory already.
*/
override private[memory] def acquireExecutionMemory(
numBytes: Long,
taskAttemptId: Long,
memoryMode: MemoryMode): Long = synchronized {
assertInvariants()
assert(numBytes >= 0)
val (executionPool, storagePool, storageRegionSize, maxMemory) = memoryMode match {
case MemoryMode.ON_HEAP => (
onHeapExecutionMemoryPool,
onHeapStorageMemoryPool,
onHeapStorageRegionSize,
maxHeapMemory)
case MemoryMode.OFF_HEAP => (
offHeapExecutionMemoryPool,
offHeapStorageMemoryPool,
offHeapStorageMemory,
maxOffHeapMemory)
}
/**
* Grow the execution pool by evicting cached blocks, thereby shrinking the storage pool.
*
* When acquiring memory for a task, the execution pool may need to make multiple
* attempts. Each attempt must be able to evict storage in case another task jumps in
* and caches a large block between the attempts. This is called once per attempt.
*/
def maybeGrowExecutionPool(extraMemoryNeeded: Long): Unit = {
if (extraMemoryNeeded > 0) {
// There is not enough free memory in the execution pool, so try to reclaim memory from
// storage. We can reclaim any free memory from the storage pool. If the storage pool
// has grown to become larger than `storageRegionSize`, we can evict blocks and reclaim
// the memory that storage has borrowed from execution.
val memoryReclaimableFromStorage = math.max(
storagePool.memoryFree,
storagePool.poolSize - storageRegionSize)
if (memoryReclaimableFromStorage > 0) {
// Only reclaim as much space as is necessary and available:
val spaceToReclaim = storagePool.freeSpaceToShrinkPool(
math.min(extraMemoryNeeded, memoryReclaimableFromStorage))
storagePool.decrementPoolSize(spaceToReclaim)
executionPool.incrementPoolSize(spaceToReclaim)
}
}
}
/**
* The size the execution pool would have after evicting storage memory.
*
* The execution memory pool divides this quantity among the active tasks evenly to cap
* the execution memory allocation for each task. It is important to keep this greater
* than the execution pool size, which doesn't take into account potential memory that
* could be freed by evicting storage. Otherwise we may hit SPARK-12155.
*
* Additionally, this quantity should be kept below `maxMemory` to arbitrate fairness
* in execution memory allocation across tasks, Otherwise, a task may occupy more than
* its fair share of execution memory, mistakenly thinking that other tasks can acquire
* the portion of storage memory that cannot be evicted.
*/
def computeMaxExecutionPoolSize(): Long = {
maxMemory - math.min(storagePool.memoryUsed, storageRegionSize)
}
executionPool.acquireMemory(
numBytes, taskAttemptId, maybeGrowExecutionPool, () => computeMaxExecutionPoolSize)
}