spark2.3源码分析之UnifiedMemoryManager

概述

MemoryManager

MemoryManager是spark的内存管理器，它定义了execution和storage之间共享内存的方式。execution memory指的是在shuffle、join、sort和aggregation过程中使用的内存；storage memory指的是缓存RDD和缓存broadcast广播变量占用的内存。每个JVM中都存在一个MemoryManager。

MemoryManager的继承关系

Spark从1.6.0版本开始，内存管理模块就发生了改变，旧版本的内存管理模块使用的是StaticMemoryManager，现在被称为"legacy"。"Legacy"模式默认不可用，即spark.memory.useLegacyMode参数默认为false。spark1.6.0版本后的新的内存管理模型，它实现的是UnifiedMemoryManager

UnifiedMemoryManager

UnifiedMemoryManager定义了Execution memory和storage memory之间的软边界，并且它们可以互相借用内存。

Execution memory和storage memory的总内存大小由spark.memory.fraction参数指定，默认为0.6，它表明了该总内存大小占jvm堆内存大小的比例。进一步地，storage memory的内存大小所占比例由spark.memory.storageFraction指定，默认为0.5。也就是说，默认的storage memory的内存大小为 = jvm heap size * 0.6 * 0.5。

当Execution memory是空闲时，storage memory可以借用Execution memory的内存，直至Execution memory要求回收它的内存。这个时候，storage memory中缓存的block会溢出到磁盘，直至Execution memory已经收回足够的内存。

当storage memory是空闲时，Execution memory也可以借用storage memory的内存，但是Execution memory属于有借无还的角色，即使这时storage memory内存紧张，也不会让出已经借用的内存。

 

acquireExecutionMemory方法

  /**
   * Try to acquire up to `numBytes` of execution memory for the current task and return the
   * number of bytes obtained, or 0 if none can be allocated.
   *
   * This call may block until there is enough free memory in some situations, to make sure each
   * task has a chance to ramp up to at least 1 / 2N of the total memory pool (where N is the # of
   * active tasks) before it is forced to spill. This can happen if the number of tasks increase
   * but an older task had a lot of memory already.
   */
  override private[memory] def acquireExecutionMemory(
      numBytes: Long,
      taskAttemptId: Long,
      memoryMode: MemoryMode): Long = synchronized {
    assertInvariants()
    assert(numBytes >= 0)
    val (executionPool, storagePool, storageRegionSize, maxMemory) = memoryMode match {
      case MemoryMode.ON_HEAP => (
        onHeapExecutionMemoryPool,
        onHeapStorageMemoryPool,
        onHeapStorageRegionSize,
        maxHeapMemory)
      case MemoryMode.OFF_HEAP => (
        offHeapExecutionMemoryPool,
        offHeapStorageMemoryPool,
        offHeapStorageMemory,
        maxOffHeapMemory)
    }

    /**
     * Grow the execution pool by evicting cached blocks, thereby shrinking the storage pool.
     *
     * When acquiring memory for a task, the execution pool may need to make multiple
     * attempts. Each attempt must be able to evict storage in case another task jumps in
     * and caches a large block between the attempts. This is called once per attempt.
     */
    def maybeGrowExecutionPool(extraMemoryNeeded: Long): Unit = {
      if (extraMemoryNeeded > 0) {
        // There is not enough free memory in the execution pool, so try to reclaim memory from
        // storage. We can reclaim any free memory from the storage pool. If the storage pool
        // has grown to become larger than `storageRegionSize`, we can evict blocks and reclaim
        // the memory that storage has borrowed from execution.
        val memoryReclaimableFromStorage = math.max(
          storagePool.memoryFree,
          storagePool.poolSize - storageRegionSize)
        if (memoryReclaimableFromStorage > 0) {
          // Only reclaim as much space as is necessary and available:
          val spaceToReclaim = storagePool.freeSpaceToShrinkPool(
            math.min(extraMemoryNeeded, memoryReclaimableFromStorage))
          storagePool.decrementPoolSize(spaceToReclaim)
          executionPool.incrementPoolSize(spaceToReclaim)
        }
      }
    }

    /**
     * The size the execution pool would have after evicting storage memory.
     *
     * The execution memory pool divides this quantity among the active tasks evenly to cap
     * the execution memory allocation for each task. It is important to keep this greater
     * than the execution pool size, which doesn't take into account potential memory that
     * could be freed by evicting storage. Otherwise we may hit SPARK-12155.
     *
     * Additionally, this quantity should be kept below `maxMemory` to arbitrate fairness
     * in execution memory allocation across tasks, Otherwise, a task may occupy more than
     * its fair share of execution memory, mistakenly thinking that other tasks can acquire
     * the portion of storage memory that cannot be evicted.
     */
    def computeMaxExecutionPoolSize(): Long = {
      maxMemory - math.min(storagePool.memoryUsed, storageRegionSize)
    }

    executionPool.acquireMemory(
      numBytes, taskAttemptId, maybeGrowExecutionPool, () => computeMaxExecutionPoolSize)
  }