浅谈Java OOM问题的几种解决办法

at org.apache.hadoop.examples.WordCount.main(WordCount.java:83)

at sun.reflect.NativeMethodAccessorlmpl.invoke©(Native Method)

at sun.reflect.NativeMethodAccessorlmpl.invoke(NativeMethodAccessorlmpl.java:62)

at sun.reflect.DelegatingMethodAccessorlmpl.invoke(DelegatingMethodAccessorlmpl.java:43)

at java.lang.reflect.Method.invoke (Method.java:498)

at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)

at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)

at org.apache.hadoop.examples .Exampl.eDriver.main(ExampleDriver, java:74)

at sun.reflect.NativeMethodAccessorlmpl.invoke©(Native Method)

at sun.reflect.NativeMethodAccessorlmpl.invoke(NativeMethodAccessorlmpl.java:62)

at sun.reflect.DelegatingMethodAccessorlmpl. invoke(DelegatingMethodAccessorlmpl.java:43)

若遇到上图问题，可以直接通过修改hadoop客户端的hadoop-env.sh文件中HADOOP_CLIENT_OPTS来永久的修改客户端HADOOP_CLIENT_OPTS JVM参数，也可以通过命令行临时修改：

export HADOOP_CLIENT_OPTS=“-Xmx128m $HADOOP_CLIENT_OPTS”

2、APPMaster阶段

APPMaster启动运行时遇到内存不足问题：

2021-12-13 14:46:27,465 ERROR [Listener at 0.0.0.0/30046]org.apache,hadoop.mapreduce.v2.app.MRAppMaster:Error starting MRAppMaster java.lang.OutOfMemoryError:GC overhead limit exceeded

at java.util.Arrays.copyOfRange(Arrays,java:3664)

at java.lang.String.(String,java:207)

at java.lang.StringBuilder.toString(StringBuilder.java:407)

at org.apache,hadoop.conf.Configuration$Parser.handleEndElement(Configuration,java:3246)

at org.apache,hadoop.conf.Configuration$Parser.parseNext(Configuration,java:3331)

at org.apache,hadoop.conf.Configuration$Parser.parse(Configuration,java :3114)

at org.apache,hadoop.conf.Configuration.loadResource(Configuration,java:3007)

at org.apache,hadoop.conf.Configuration.loadResources(Configuration,java:2973)

at org.apache,hadoop.conf.Configuration.getProps(Configuration,java:2848)

at org.apache,hadoop.conf.Configuration,get(Configuration.java:1460)

at org.apache,hadoop.security,authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)

at org.apache,hadoop.security,authorize.ServiceAuthorizationManager.refresh(ServiceAuthorizationManager.java:150)

at org.apache,hadoop.ipc.Server.refreshServiceAcl(Server,java :664)

at org.apache,hadoop.mapreduce.v2.app.client.MRClientService.refreshServiceAcls(MRClientService.java:156)

at org.apache,hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:131)

at org.apache,hadoop.service.AbstractService.start(AbstractService.java:194)

at org.apache,hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1290)

at org.apache,hadoop.service.AbstractService.start(AbstractService.java:194)

at org.apache,hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)

at java,security.AccessController.doPrivileged(Native Method)

at javax.security,auth.Subject.doAs(Subject, java:422)

at org.apache,hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)

at org.apache,hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)

at org.apache,hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)

2021-12-13 14:46:29,478 INFO [Listener at 0.0.0.0/30046]org.apache,hadoop.util.ExitUtil:Exiting with status 1:java.lang.OutOfMemoryError:GC overhead limit exceeded

若遇上述问题可修改客户端配置文件mapred-site.xml

##为AM分配的内存

yarn.app.mapreduce.am.resource.mb

516

##给AM分配的JVM启动参数

yarn.app.mapreduce.am.command-opts

-Xmx2048m

3、Map阶段

Map阶段运行时报出OOM问题截图:

2021-12-10 15:22:10, 475 ERROR [main] org. apache, hadoop. mapred. YarnChild: Error running child : java. lang. OutOfMemoryError: Java heap space

at org.apache,hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask. java:1000)

at org.apache,hadoop.mapred.MapTask.createSortingCollector(MapTask. java:408)

at org.apache,hadoop.mapred.MapTask.access$100(MapTask. java :82)

at org.apache,hadoop.mapred.MapTask$NewOutputCollector.(MapTask. java :710)

at org.apache,hadoop.mapred. MapTask.runNewMapper(MapTask. java :782)

at org.apache,hadoop.mapred. MapTask.run(MapTask. java:347)

at org.apache,hadoop.mapred.YarnChild$2.run(YarnChiId. java:174)

at java,security.AccessController.doPrivileged(Native Method)

at javax.ecurity,auth.Subject.doAs(Subject, java:422)

at org.apache,hadoop.security.UserGroupInformation.doAs(UserGroupInformation. java:1730)

at org.apache,hadoop.mapred.YarnChiId.main(YarnChiId. java:168)

若遇上述问题可修改客户端配置文件mapred-site.xml

mapreduce.map.memory.mb

2048

#Map Jvm启动参数

mapreduce.map.java.opts

-Xmx1024m

true

4、Reduce阶段

Reduce阶段运行时报出OOM问题截图:

2021-12-10 15:09:30,740 ERROR[main]org.apache,hadoop.mapred.YarnChild:Error running child:java.lang.OutOfMemoryError:Java heap space

at java.nio.CharBuffer.allocate(CharBuffer.java:335)

at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:795)

at org.apache,hadoop.io.Text,decode(Text,java:412)

at org.apache,hadoop.io.Text,decode(Text,java:389)

at org.apache,hadoop.io.Text.toString(Text,java:280)

at org.apache,hadoop.examples.WordCount1$IntSumReducer.reduce(WordCount1.java:68)

at org.apache,hadoop.examples.WordCount1$IntSumReducer.reduce(WordCount1.java:54)

at org.apache,hadoop.mapreduce.Reducer,run(Reducer,java:171)

at org.apache,hadoop.mapred.Task$NewCombinerRunner.combine(Task,java:1912)

at org.apache,hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1662)

at org.apache,hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1505)

at org.apache,hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:735)

at org.apache,hadoop.mapred.MapTask.closeQuietly(MapTask.java:2076)

at org.apache,hadoop.mapred.MapTask.runNewMapper(MapTask.java:809)

at org.apache,hadoop.mapred.MapTask.run(MapTask.java:347)

at org.apache,hadoop.mapred.YarnChild$2.run(YarnChiId.java:174)

at java,security.AccessController.doPrivileged(Native Method)

at javax.security,auth.Subject.doAs(Subject,java:422)

at org.apache,hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)

at org.apache,hadoop.mapred.YarnChiId.main(YarnChiId.java:168)

若遇上述问题可修改客户端配置文件mapred-site.xml

mapreduce.reduce.memory.mb

2048

#Reduce Jvm启动参数

最后

学习视频：

大厂面试真题：

若遇上述问题可修改客户端配置文件mapred-site.xml

mapreduce.reduce.memory.mb

2048

#Reduce Jvm启动参数

最后

学习视频：

[外链图片转存中…(img-olmZFs4i-1718927585771)]

大厂面试真题：

[外链图片转存中…(img-44RTfKY3-1718927585771)]