Yarn调优之调整ResourceManager堆内存大小

最新推荐文章于 2024-07-29 07:00:00 发布

宝罗Paul

最新推荐文章于 2024-07-29 07:00:00 发布

阅读量9.8k

点赞数

分类专栏：大数据文章标签： yarn hadoop JVM

本文链接：https://blog.csdn.net/qq_31598113/article/details/82973161

版权

本文记录了一次YarnResourceManager由于Java堆内存不足导致的故障，通过检查发现RM的最大堆内存仍为默认的1000Mb。为解决此问题，对`yarn-env.sh`配置文件进行修改，将`YARN_RESOURCEMANAGER_HEAPSIZE`调整为合适的值，并备份原配置。随后，使用工具同步配置到整个集群，并分别重启active和standby状态的RM，验证堆内存配置已生效，确保系统稳定运行。

摘要由CSDN通过智能技术生成

ResourceManager挂了。查看到active的ResourceManager日志有如下内容：
java.lang.OutOfMemoryError: Java heap space
故障的原因是RM的堆内存空间size不够了。

查看到活跃节点RM的最大堆内存大小仍然是默认的1000Mb

[hadoop@my-hdp-01 hadoop]$ ps aux | grep -i resourcemanager | grep -v grep | grep --color Xmx
hadoop    9075  0.0  0.4 2973936 596152 ?      Sl   Oct07   1:02 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/bin/java -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/home/hadoop/hadoop/logs -Dyarn.log.dir=/home/hadoop/hadoop/logs -Dhadoop.log.file=yarn-hadoop-resourcemanager-my-hdp-01.log -Dyarn.log.file=yarn-hadoop-resourcemanager-my-hdp-01.log -Dyarn.home.dir= -Dyarn.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop/lib/native -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/home/hadoop/hadoop/logs -Dyarn.log.dir=/home/hadoop/hadoop/logs -Dhadoop.log.file=yarn-hadoop-resourcemanager-my-hdp-01.log -Dyarn.log.file=yarn-hadoop-resourcemanager-my-hdp-01.log -Dyarn.home.dir=/home/hadoop/hadoop -Dhadoop.home.dir=/home/hadoop/hadoop -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop/lib/native -classpath /home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/lib/*:/home/hadoop/hadoop/share/hadoop/common/*:/home/hadoop/hadoop/share/hadoop/hdfs:/home/hadoop/hadoop/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop/share/hadoop/hdfs/*:/home/hadoop/hadoop/share/hadoop/yarn/lib/*:/home/hadoop/hadoop/share/hadoop/yarn/*:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop/share/hadoop/mapreduce/*:/home/hadoop/hadoop/share/hadoop/tools/lib/*::/home/hadoop/hadoop/share/hadoop/yarn/*:/home/hadoop/hadoop/share/hadoop/yarn/lib/*:/home/hadoop/hadoop/etc/hadoop/rm-config/log4j.properties org.apache.hadoop.yarn.server.resourcemanager.ResourceManager

查看到待命节点RM最大堆内存大小也是默认的1000Mb

[hadoop@my-hdp-01 hadoop]$ ssh my-hdp-02 ps aux | grep -i resourcemanager | grep -v grep | grep --color Xmx
hadoop    4919  7.8  0.9 3349280 1248308 ?     Sl   Oct07 120:31 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/bin/java -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/home/hadoop/hadoop/logs -Dyarn.log.dir=/home/hadoop/hadoop/logs -Dhadoop.log.file=yarn-hadoop-resourcemanager-my-hdp-02.log -Dyarn.log.file=yarn-hadoop-resourcemanager-my-hdp-02.log -Dyarn.home.dir= -Dyarn.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop/lib/native -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/home/hadoop/hadoop/logs -Dyarn.log.dir=/home/hadoop/hadoop/logs -Dhadoop.log.file=yarn-hadoop-resourcemanager-my-hdp-02.log -Dyarn.log.file=yarn-hadoop-resourcemanager-my-hdp-02.log -Dyarn.home.dir=/home/hadoop/hadoop -Dhadoop.home.dir=/home/hadoop/hadoop -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/home/had