新建的hadoop集群提交yarn任务后报错,hive任务也执行不了。
报错如下:
ERROR [Listener at 0.0.0.0/45871] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:979)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:177)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:159)
... 14 more
看了下这个空指针MRClientService.getHttpPort(MRClientService.java:177)
@Override
public int getHttpPort() {
return webApp.port();
}
webApp的port没有获取到, 再看port()方法就是获取httpServer地址的端口
public int port() {
InetSocketAddress addr = checkNotNull(httpServer, “httpServer”)
.getConnectorAddress(0);
return addr == null ? -1 : addr.getPort();
}
看网上说是yarn.resourcemanager.webapp.address没配置导致没能获取到port。
在yarn-site.xml中配置了yarn.resourcemanager.webapp.address之后就正常了。
如果是HA机制下yarn-site.xml需要加入以下配置:
<!-- 指定 rm1 的主机名 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>xxx1</value>
</property>
<!-- 指定 rm1 的 web 端地址 -->
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>xxx1:8088</value>
</property>
<!-- 指定 rm2 的主机名 -->
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>hadoop103</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>xxx2:8088</value>
</property>
可以参考这个配置https://blog.csdn.net/m0_51111980/article/details/126085577
xxx1主机名的值从yarn.resourcemanager.hostname中找,如果是HA那么是类似这样的
参考:
[1] https://blog.csdn.net/weixin_44812855/article/details/112716670
[2] https://blog.csdn.net/m0_51111980/article/details/126085577
[3] https://www.cnblogs.com/dreamboy/archive/2018/07/10/9289631.html
[4] https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html