Spark 内存管理之StaticMemoryManager

概述

介绍StaticMemoryManager的创建以及内存的申请、释放等。

前言

  这部分内容相对较为简单，只需要知道这是Spark 1.6之前的唯一内存管理器，现在为了兼容性，依旧保留，但是默认使用的是统一内存管理器。
  关于原理，尤其是内存分配，在Spark内存管理之堆内/堆外内存原理详解中，我们已经详细介绍过了，而源码其实与Spark 内存管理之UnifiedMemoryManager中内存分配部分差不多，只是少了动态分配而已，这里不会细讲，只是附上源码。

1. 创建StaticMemoryManager

查看说明：

查看类定义：

相对于UnifiedMemoryManager 构造时调用工厂方法apply( )，StaticMemoryManager类中有一个this()方法，用于构造此对象（两个方法在后面）：

而由于StaticMemoryManager不支持堆外存储内存，因此会进行一下操作，将堆外存储内存转化为堆外执行内存：

最终内存分配情况如下

this()方法中用到的两个方法StaticMemoryManager.getMaxExecutionMemory()、StaticMemoryManager.getMaxStorageMemory():

1.1 StaticMemoryManager.getMaxStorageMemory()

1.2 StaticMemoryManager.getMaxExecutionMemory()

2. 实现内存操作方法

2.1 acquireStorageMemory()

由于StaticMemoryManager不支持堆外存储内存，因此只能从堆内存储内存来获取内存。

2.2 acquireExecutionMemory()

判断申请的执行内存种类，交给对应的 堆内/堆外 执行内存池 去申请。

2.3 acquireUnrollMemory()

由于静态方式下，StorageMemory是固定的，因此我们要限制UnrollMemory内存的大小。

总结

介绍StaticMemoryManager的创建以及内存的申请、释放等。

参考

• Spark 内存管理之StaticMemoryManager

附录

/**
  *一个[[MemoryManager]]将堆空间静态划分为不相交的区域。
  *
  *执行区和存储区的大小分别通过`spark.shuffle.memoryFraction`和`spark.storage.memoryFraction`确定。
	* 两个区域完全分开，因此任何一个使用都不能从另一个借用内存。
 */
private[spark] class StaticMemoryManager(
    conf: SparkConf,
    maxOnHeapExecutionMemory: Long,
    override val maxOnHeapStorageMemory: Long,
    numCores: Int)
  extends MemoryManager(
    conf,
    numCores,
    maxOnHeapStorageMemory,
    maxOnHeapExecutionMemory) {

  def this(conf: SparkConf, numCores: Int) {
    this(
      conf,
      StaticMemoryManager.getMaxExecutionMemory(conf),
      StaticMemoryManager.getMaxStorageMemory(conf),
      numCores)
  }

  // StaticMemoryManager不支持堆外存储内存：
  offHeapExecutionMemoryPool.incrementPoolSize(offHeapStorageMemoryPool.poolSize)
  offHeapStorageMemoryPool.decrementPoolSize(offHeapStorageMemoryPool.poolSize)



  override def maxOffHeapStorageMemory: Long = 0L

  override def acquireStorageMemory(
      blockId: BlockId,
      numBytes: Long,
      memoryMode: MemoryMode): Boolean = synchronized {
    require(memoryMode != MemoryMode.OFF_HEAP,
      "StaticMemoryManager does not support off-heap storage memory")
    if (numBytes > maxOnHeapStorageMemory) {
      // Fail fast if the block simply won't fit
      logInfo(s"Will not store $blockId as the required space ($numBytes bytes) exceeds our " +
        s"memory limit ($maxOnHeapStorageMemory bytes)")
      false
    } else {
      onHeapStorageMemoryPool.acquireMemory(blockId, numBytes)
    }
  }

  // unRoll时值得驱逐的最大字节数
  private val maxUnrollMemory: Long = {
    (maxOnHeapStorageMemory * conf.getDouble("spark.storage.unrollFraction", 0.2)).toLong
  }
  override def acquireUnrollMemory(
      blockId: BlockId,
      numBytes: Long,
      memoryMode: MemoryMode): Boolean = synchronized {
    require(memoryMode != MemoryMode.OFF_HEAP,
      "StaticMemoryManager does not support off-heap unroll memory")
    val currentUnrollMemory = onHeapStorageMemoryPool.memoryStore.currentUnrollMemory
    val freeMemory = onHeapStorageMemoryPool.memoryFree
    // 展开时，我们将使用所有现有的可用内存，并在必要时使用一些额外的空间来免除缓存的块。 
    // 我们必须限制要通过展开而退出的内存量，否则展开一个大块可能会耗尽整个缓存。
    val maxNumBytesToFree = math.max(0, maxUnrollMemory - currentUnrollMemory - freeMemory)
    // Keep it within the range 0 <= X <= maxNumBytesToFree
    val numBytesToFree = math.max(0, math.min(maxNumBytesToFree, numBytes - freeMemory))
    onHeapStorageMemoryPool.acquireMemory(blockId, numBytes, numBytesToFree)
  }

  private[memory]
  override def acquireExecutionMemory(
      numBytes: Long,
      taskAttemptId: Long,
      memoryMode: MemoryMode): Long = synchronized {
    memoryMode match {
      case MemoryMode.ON_HEAP => onHeapExecutionMemoryPool.acquireMemory(numBytes, taskAttemptId)
      case MemoryMode.OFF_HEAP => offHeapExecutionMemoryPool.acquireMemory(numBytes, taskAttemptId)
    }
  }
}


private[spark] object StaticMemoryManager {

  private val MIN_MEMORY_BYTES = 32 * 1024 * 1024

  /**
   * 返回可用于存储区域的内存总量，以字节为单位。
   */
  private def getMaxStorageMemory(conf: SparkConf): Long = {
    val systemMaxMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)
    val memoryFraction = conf.getDouble("spark.storage.memoryFraction", 0.6)
    val safetyFraction = conf.getDouble("spark.storage.safetyFraction", 0.9)
    (systemMaxMemory * memoryFraction * safetyFraction).toLong
  }

  /**
   * 返回可用于执行区域的内存总量（以字节为单位）。
   */
  private def getMaxExecutionMemory(conf: SparkConf): Long = {
    val systemMaxMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)

    if (systemMaxMemory < MIN_MEMORY_BYTES) {
      throw new IllegalArgumentException(s"System memory $systemMaxMemory must " +
        s"be at least $MIN_MEMORY_BYTES. Please increase heap size using the --driver-memory " +
        s"option or spark.driver.memory in Spark configuration.")
    }
    if (conf.contains("spark.executor.memory")) {
      val executorMemory = conf.getSizeAsBytes("spark.executor.memory")
      if (executorMemory < MIN_MEMORY_BYTES) {
        throw new IllegalArgumentException(s"Executor memory $executorMemory must be at least " +
          s"$MIN_MEMORY_BYTES. Please increase executor memory using the " +
          s"--executor-memory option or spark.executor.memory in Spark configuration.")
      }
    }
    val memoryFraction = conf.getDouble("spark.shuffle.memoryFraction", 0.2)
    val safetyFraction = conf.getDouble("spark.shuffle.safetyFraction", 0.8)
    (systemMaxMemory * memoryFraction * safetyFraction).toLong
  }

}