启动hadoop,刚解决完datanode未成功启动问题,又出来新问题,在datanode上用jps看不到NodeManager进程;查看日志,出现错误如下:
Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hbase-cluster-slave1-hx4lw doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager.
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:203)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:272)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:496)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:543)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hbase-cluster-slave1-hx4lw doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager.
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:278)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:197)
... 6 more
于是在网上搜索的相关问题,大部分都是修改 yarn-site.xml 中的yarn.scheduler.minimum-allocation-mb 内存大于1024,
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property>
借鉴该方法,重启hadoop,还是不管用;又借鉴另一网友方法,修改yarn-env.sh 中 JAVA_HEAP_MAX=-Xmx3072m 改为3G,重启hadoop,查看日志还是出现同样的问题;最后发现,除了内存问题,cpu也有限制,因此,在yarn-site.xml 文件中,添加下述属性,重启hadoop,NodeManager正常启动,问题解决。
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>
如果NodeManager启动成功,DataNode启动不了的话,参考上一博文,主要是通过删除/opt/hbase/hadoop-2.7.4/temp目录下文件和dfs目录下文件。(当集群已用时间较长,慎重删除)