Spark 内存管理

Spark官网对内存管理的介绍如下地址：
http://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption
查看源码，内存管理的逻辑在MemoryManager.scala
SparkEnv.scala

    val useLegacyMemoryManager = conf.getBoolean("spark.memory.useLegacyMode", false)
    val memoryManager: MemoryManager =
      if (useLegacyMemoryManager) {
        new StaticMemoryManager(conf, numUsableCores)
      } else {
        UnifiedMemoryManager(conf, numUsableCores)
      }

    val blockManagerPort = if (isDriver) {
      conf.get(DRIVER_BLOCK_MANAGER_PORT)
    } else {
      conf.get(BLOCK_MANAGER_PORT)
    }

1.5及其以前的内存管理模式为如下，得到的内存大小为：(systemMaxMemory * memoryFraction * safetyFraction).toLong，如果内存设置为1G，那么真正的获得的执行内存为：1024M*0.2*0.8 = 163M，真正的获得的存储内存为：1024M*06*0.9 = 552.96
统一的内存管理，存储和内存是独立分开的

  def this(conf: SparkConf, numCores: Int) {
    this(
      conf,
      StaticMemoryManager.getMaxExecutionMemory(conf),
      StaticMemoryManager.getMaxStorageMemory(conf),
      numCores)
  }

  private def getMaxExecutionMemory(conf: SparkConf): Long = {
    // 默认如果未配置spark.testing.memory，则使用运行时的最大内存
    val systemMaxMemory = conf.getLong("spark.testing.memory", Runtime