UnifiedMemoryManager

UnifiedMemoryManager 动态存储管理器

UnifiedMemoryManager 动态存储管理器是MemoryManager存储管理器的子类
它动态的管理执行内存和存储内存的大小,以便执行内存和存储内存可以在自身内存不够使用时占用对方的内存
先获取执行内存和存储内存的占用内存maxMemory大小，并默认执行内存和存储内存各占总内存的50%，不过可以通过配置参数
spark.memory.storageFraction设置

代码清单

在总内存中获取一部分内存用于执行内存和存储内存，另一部分内存用于非存储、非执行目的操作。
getMaxMemory()方法返回堆内存的大小,大小为(systemMemory - reservedMemory) * memoryFraction
systemMemory 默认为Runtime.getRuntime.maxMemory，也可通过配置属性spark.testing.memory设置
reservedMemory默认为RESERVED_SYSTEM_MEMORY_BYTES = 300 * 1024 * 1024=300MB，也可通过配置属性spark.testing.reservedMemory设置
memoryFraction默认为0.6，也可通过配置属性spark.memory.fraction设置

private val RESERVED_SYSTEM_MEMORY_BYTES = 300 * 1024 * 1024

  /**
   * Return the total amount of memory shared between execution and storage, in bytes.
   */
  private def getMaxMemory(conf: SparkConf): Long = {
   
    val systemMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)
    val reservedMemory = conf.getLong("spark.testing.reservedMemory",
      if (conf.contains("spark.testing")) 0 else RESERVED_SYSTEM_MEMORY_BYTES)
    val minSystemMemory = (reservedMemory * 1.5).ceil.toLong
    if (systemMemory < minSystemMemory) {
   
      throw new IllegalArgumentException(s"System memory $systemMemory must " +
        s"be at least $minSystemMemory. Please increase heap size using the --driver-memory " +
        s"option or spark.driver.memory in Spark configuration.")
    }
    // SPARK-12759 Check executor memory to fail fast if memory is insufficient
    if (conf.contains("spark.executor.memory")) {
   
      val exec