问题1:16/03/14 15:22:24 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 17.0 GB of 17 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
解决:
--conf spark.yarn.executor.memoryOverhead=4096 \
问题2:
30000元素list,Cn2组合,用array或者list,会out of memory或者导致GC时间太长,从而作业失败
解决:
val i1 = x._2.iterator val i2 = x._2.iterator.toArray i1.flatMap(y => i2.map(z => ((y, z), 1) ).filter(z => z._1._1 > z._1._2) )