Spark on YARN占用资源分析 - Spark 内存模型

Spark的Excutor的Container内存有两大部分组成：堆外内存和Excutor内存

A)   堆外内存(spark.yarn.executor.memoryOverhead)

  主要用于JVM自身的开销。默认：MAX(executorMemory*0.10,384m)

B)   Excutor内存(spark.executor.memory)

   Execution:shuffle、排序、聚合等用于计算的内存

    Storage：用于集群中缓冲和传播内部数据的内存（cache、广播变量）

上面的 堆外内存和excutor 内存是可以相互转换使用的

两个重要参数：

spark.memory.fraction

  用于设置Execution和Storage内存在内存（这个内存是JVM的堆内存-300M，这300M是预留内存）中

的占比，默认是60%(spark 2.x)。即Execution和Storage内存大小之和占堆内存比例。

            spark 1.6.x 的spark.memory.fraction 默认是75% 

       参考链接： http://spark.apache.org/docs/1.6.3/tuning.html  搜索 spspark.memory.fraction

                   http://spark.apache.org/docs/latest/tuning.html

剩下的40%用于用户的数据结构、Spark的元数据和预留防止OOM的内存。

pspark.memory.storageFraction

  表示Storage内存在Execution和Storage内存之和的占比。设置这个参数可避免缓冲的数据块被清理出

内存。

   spark  1.6.3  

Memory usage in Spark largely falls under one of two categories: execution and storage. Execution memory refers to that used for computation in shuffles, joins, sorts and aggregations, while storage memory refers to that used for caching and propagating internal data across the cluster. In Spark, execution and storage share a unified region (M). When no execution memory is used, storage can acquire all the available memory and vice versa. Execution may evict storage if necessary, but only until total storage memory usage falls under a certain threshold (R). In other words, R describes a subregion within M where cached blocks are never evicted. Storage may not evict execution due to complexities in implementation.

This design ensures several desirable properties. First, applications that do not use cach