【spark】spark historyserver OOM解决方案

前言：

我们的spark historyserver 有时会频繁宕机。

报错日志：

21/11/18 16:18:36 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[spark-history-task-0,5,main]
java.lang.OutOfMemoryError: GC overhead limit exceeded
21/11/18 16:18:37 INFO server.AbstractConnector: Stopped Spark@2234078{HTTP/1.1,[http/1.1]}{0.0.0.0:18088}
21/11/18 16:19:07 WARN thread.QueuedThreadPool: qtp358420513{STOPPING,8<=11<=200,i=0,q=2} Couldn't stop Thread[qtp358420513-2628,5,main]
21/11/18 16:19:07 WARN thread.QueuedThreadPool: qtp358420513{STOPPING,8<=11<=200,i=0,q=2} Couldn't stop Thread[qtp358420513-1018,5,main]
21/11/18 16:19:13 INFO util.ShutdownHookManager: Shutdown hook called

报错原因：

spark history 服务内存溢出。

解决方案：

1. 修改配置文件

vi ${SPARK_HOME}/conf/spark-env.sh

2. 添加

export SPARK_DAEMON_MEMORY=8192m

3. 重启 historyserver服务

./sbin/stop-history-server.sh
./sbin/start-history-server.sh

4. 查看是否修改成功

hdfs     10553  4846  0 17:18 pts/0    00:00:00 grep 27353
hdfs     27353     1 99 16:33 ?        01:44:46 /opt/jdk/bin/java -Xmx64m -cp /var/lib/hadoop-hdfs/spark-2.3.2-bin/conf/:/var/lib/hadoop-hdfs/spark-2.3.2-bin/jars/*:/etc/hadoop/conf/ -Dspark.history.ui.port=18088 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://nameservice1/tmp/spark/log -Xmx8192m org.apache.spark.deploy.history.HistoryServer

修改成功！