spark sql insert 报错 Container killed on request. Exit code is 143

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 198 in stage 4.0 failed 4 times, most recent failure: Lost task 198.3 in stage 4.0 (TID 1722, hadoop-slave-17, executor 22): ExecutorLostFailure (executor 22 exited caused by one of the running tasks) Reason: Container marked as failed: container_e209_1579608513692_34016_01_000030 on host: hadoop-slave-17. Exit status: 143. Diagnostics: [2020-03-05 17:17:55.532]Container killed on request. Exit code is 143
[2020-03-05 17:17:55.532]Container exited with a non-zero exit code 143. 
[2020-03-05 17:17:55.532]Killed by external signal
Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1524)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1512)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1511)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1511)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1739)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1694)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1683)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2031)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:184)
	... 23 more

错误分析

从hive报错看是由于物理内存达到限制，导致container被kill掉报错。
从报错时刻看是执行reduce阶段报错；故可能reduce处理阶段container的内存不够导致。

解决方案

首先查看关于container内存的配置：

hive (default)> SET mapreduce.map.memory.mb;
mapreduce.map.memory.mb=4096
hive (default)> SET mapreduce.reduce.memory.mb;
mapreduce.reduce.memory.mb=4096
hive (default)> SET yarn.nodemanager.vmem-pmem-ratio;
yarn.nodemanager.vmem-pmem-ratio=4.2

因此，单个map和reduce分配物理内存4G；虚拟内存限制4*4.2=16.8G；

单个reduce处理数据量超过内存4G的限制导致；设置 mapreduce.reduce.memory.mb=8192 解决；

二、如果是spark执行

直接 增加资源 就可以 （亲测可用）

--executor-memory 4G

参考：
http://stackoverflow.com/questions/29001702/why-yarn-java-heap-space-memory-error?answertab=oldest#tab-top