问题现象一 :
Reason: Container killed by YARN for exceeding memory limits. 3.0 GB of 2.5 GB physical memory used
问题分析 :
此报错与堆外内存有关
解决方案 :
- 设置参数 --conf “spark.yarn.executor.memoryOverhead=4G”
- 程序里设置 (如下图代码)
private val conf = new SparkConf().set("spark.yarn.executor.memoryOverhead", "4096")