NodeManager启动报错

最新推荐文章于 2022-08-26 10:38:36 发布
最新推荐文章于 2022-08-26 10:38:36 发布
阅读量1.3k 收藏
点赞数
分类专栏： Hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/kwame211/article/details/81057256
版权

Nodeemanager异常 org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch


系统环境：ubuntu14.04 server cloudera CDH 5.10 共计28个节点。单节点内存128G。有6台机器是24核，22台机器是32核。有10台机器磁盘容量是1.8T 18台机器磁盘容量是12.8T。
resourceManager,namenode 在同一节点上，且该节点不包括datanode和nodeManager服务。
剩余28节点均包含了datanode和nodeManager服务
最近集群做TB级数据随机化实验，hdfs各节点数据分布不太均衡，10台磁盘容量为1.8T的机器磁盘全部占满，导致这些节点上yarn的nodemanager 没有足够的空间，致使9个nodemanager 停止接受服务，1个nodemanager 服务终止，且无法启动。

考虑将10台低容量磁盘机器datanode角色停止授权，数据分发到其他大容量机器上，并删除hdfs数据。

在停止授权期间，我给集群又添加了一台32核，128G内存 1.8T磁盘的机器，分配了nodeManager角色。
上述工作顺利完成后，重新启动yarn.

大部分机器正常工作，只有上述服务终止的nodemanager依然无法启动，查询日志报错如下：
FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager
Error starting NodeManager
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:227)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:944)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:931)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 5 more

google搜索 org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch 给出的解决方案：
网址：https://community.hortonworks.com/questions/86301/nodemanager-fails-to-start.html
Maybe a sst file got corrupt can you try to remove the folder of /var/log/hadoop-yarn/nodemanager/recovery-state from failed nodemanagers and check if starts?
These files stays in the system even if you decomission the nodes.
上述文件在节点上并不存在。故继续查看日志：

上午9点56:15.339分 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager
registered UNIX signal handlers for [TERM, HUP, INT]
4月 14, 上午9点56:16.426分 INFO org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService
Using state database at /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state for recovery
4月 14, 上午9点56:16.457分 INFO org.apache.hadoop.service.AbstractService
Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:944)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:931)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:227)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
4月 14, 上午9点56:16.466分 INFO org.apache.hadoop.service.AbstractService
Service NodeManager failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:227)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:944)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:931)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 5 more

  1. 发现疑似目录：/var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm- state下存在：
  2. 005615.sst 005638. log 005640. log CURRENT LOCK MANIFEST- 004397
  3. 移除所有文件。重启nodemanager 成功。
  4. 回顾错误原因可能是，我在该nodemanager终止情况下，在集群中添加了新的nodemanager，使得角色数目增加，而启动失败的nodemanager时，它使用存储的状态来恢复，在和数据库校验过程中发现数目不符合而启动失败。因此删除上述目录下的文件。


系统环境：ubuntu14.04 server cloudera CDH 5.10 共计28个节点。单节点内存128G。有6台机器是24核，22台机器是32核。有10台机器磁盘容量是1.8T 18台机器磁盘容量是12.8T。
resourceManager,namenode 在同一节点上，且该节点不包括datanode和nodeManager服务。
剩余28节点均包含了datanode和nodeManager服务
最近集群做TB级数据随机化实验，hdfs各节点数据分布不太均衡，10台磁盘容量为1.8T的机器磁盘全部占满，导致这些节点上yarn的nodemanager 没有足够的空间，致使9个nodemanager 停止接受服务，1个nodemanager 服务终止，且无法启动。

考虑将10台低容量磁盘机器datanode角色停止授权，数据分发到其他大容量机器上，并删除hdfs数据。

在停止授权期间，我给集群又添加了一台32核，128G内存 1.8T磁盘的机器，分配了nodeManager角色。
上述工作顺利完成后，重新启动yarn.

大部分机器正常工作，只有上述服务终止的nodemanager依然无法启动，查询日志报错如下：
FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager
Error starting NodeManager
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:227)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:944)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:931)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 5 more

google搜索 org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch 给出的解决方案：
网址：https://community.hortonworks.com/questions/86301/nodemanager-fails-to-start.html
Maybe a sst file got corrupt can you try to remove the folder of /var/log/hadoop-yarn/nodemanager/recovery-state from failed nodemanagers and check if starts?
These files stays in the system even if you decomission the nodes.
上述文件在节点上并不存在。故继续查看日志：

上午9点56:15.339分 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager
registered UNIX signal handlers for [TERM, HUP, INT]
4月 14, 上午9点56:16.426分 INFO org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService
Using state database at /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state for recovery
4月 14, 上午9点56:16.457分 INFO org.apache.hadoop.service.AbstractService
Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:944)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:931)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:227)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
4月 14, 上午9点56:16.466分 INFO org.apache.hadoop.service.AbstractService
Service NodeManager failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:227)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: checksum mismatch
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:944)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:931)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 5 more

  1. 发现疑似目录：/var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm- state下存在：
  2. 005615.sst 005638. log 005640. log CURRENT LOCK MANIFEST- 004397
  3. 移除所有文件。重启nodemanager 成功。
  4. 回顾错误原因可能是，我在该nodemanager终止情况下，在集群中添加了新的nodemanager，使得角色数目增加，而启动失败的nodemanager时，它使用存储的状态来恢复，在和数据库校验过程中发现数目不符合而启动失败。因此删除上述目录下的文件。
个人分类：  Yarn
DemonHunter211
关注
专栏目录
Hadoop【问题记录 02】【hadoop-3.1.3 单机版】ResourceManager无法启动NodeManager启动后过自动关闭 javax/activation/DataSource
シ風
07-28 1269
报错 版本为 hadoop-3.1.3，这个问题出现的原因很多，首先是看报错日志： [root@tcloud sbin]# cd /usr/local/hadoop-3.1.3/logs/ [root@tcloud logs]# ll # 查看最新的 resourcemanager 日志 [root@tcloud logs]# cat ./hadoop-root-resourcemanager-tcloud.log 主要的报错信息如下： Error injecting constructor, java.
2023/09/10 --Hadoop103，104 上NodeManger 没有启动
m0_48214806的博客
09-10 296
问题 Hadoop103，104 上NodeManger 没有启动。2. 在hadoop103 添加 memory 和 CPU。1. 查看hadoop102 yarn 配置。3. 启动nodemanger。
resourcemanager和nodemanager启动失败
wu_noah的博客
03-04 4960
在启动hadoop3.0的过程中，通过jps指令发现进程中没有启动resourcemanager和nodemanager检查resourcemanager的log的文件发现原来是jdk9默认禁止访问许多javax的API导致hadoop3启动失败将jdk9换为jdk1.8.0_161则恢复正常...
hadoop节点nodemanager启动失败
XiaoYu的博客
04-01 4187
在正常启动的hadoop的时候系统没有报错，但是JPS查看进程的时候发现缺少了节点管理器，查看后台的日志进程...问题：发现如下错误：FATAL org.apache.hadoop.yarn.server.nodemanager.NodeMana ger：启动NodeManager时出错  org.apache.hadoop.yarn.exceptions.YarnRuntimeException...
service network restart重启失败_NodeManager 启动失败问题处理
weixin_39737492的博客
11-24 312
欢迎关注公众号NodeManager 启动失败一例：报错如下：20/09/16 15:47:20 ERROR nodemanager.NodeManager: Error starting NodeManagerjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException:...
NodeManager启动失败--防火墙篇
静下心来好好学习
02-10 5714
Hadoop环境CDH4.4 今天年后第一天上班（实习中），还过一个月，实习第一份实习合同就到期了~ Hadoop集群的虚拟环境看上去太乱，所以就将3个节点（1master + 2Slaves）重启，结果NodeManager启动失败。查看日志，记录错误如下： 2014-02-10 18:24:07,635 FATAL org.apache.hadoop.yarn.server.nodema
Hadoop NodeManager无法启动问题解决
晨之风的专栏
10-30 1万+
最近在学习Hadoop的时候， 用虚拟机在Ubuntu中按照官网教程搭建好伪集群环境。 hdfs可以正常启动， 但是在启动Yarn的时候， NodeManager一直无法启动 doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager
CDH集群nodemanager启动不了
purisuit_knowledge的专栏
01-12 3243
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverContainer(ContainerManagerImpl.java:289)
NodeManager节点启动报错内存不足 NodeManager from master doesn't satisfy minimum allocations
BENULL的博客
09-27 3080
启动hadoop后jps存在nodemanager进程，但马上会报错中断进程 查看日志后发现报的是启动nodemanager 所需内存不足 NodeManager from master doesn't satisfy minimum allocations
windows下hadoop的resourcemanager与nodemanager无法启动：Error starting NodeManager
weixin_47285493的博客
04-27 4622
hadoop的resourcemanager与nodemanager无法成功启动
Hadoop com/fasterxml/jackson/databind/ObjectMapper Error starting NodeManager
AntKengElephant的博客
11-12 664
2019-11-11 11:20:20,267 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: usercache path : file:/dat a/appcom/hadoop-2.7.4/tmp/nm-local-dir/usercac...
Error starting NodeManager-Service NodeManager failed in state INITED; cause: java.lang.Null
奔跑吧兄弟！
12-11 3548
版本：CDH 5.2 1、异常日志 2014-11-25 14:21:27,873 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Adding container_1415870304563_7717_01_000011 to recently stopped cont
NodeManager启动错误
潇水汀寒
07-25 1万+
NodeManager 没起来 2013-07-25 20:06:22,266 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop
hadoop节点nodemanager启动失败原因小结~
热门推荐
ling811的专栏
07-12 3万+
slaves节点报错，报的是启动nodemanager 所需内存不足 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Unexpected error starting NodeStatusUpdater org.apache.hadoop.yarn.exceptions.YarnRunt
hadoop集群运维的坑合集--datanode和nodemanager无法启动
weixin_44513718的博客
08-26 1514
后面因为服务器整体降配，需要重启，发现其他组件启动都没有问题，刚好在hdp100上这台的datanode和nodemanager均无法启动
hadoop-hadoop部署完成，启动集群后，NodeManager未启动， NodeManager from hd01 doesn't satisfy minimum allocations
lisery的博客
08-27 1251
hadoop部署过程结束，执行start-all.sh后，NodeManager为启动， 相关日志为： 2018-08-27 16:14:11,228 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.yarn.exceptions...
Hadoop2.0.0-CDH4中NodeManager启动不了的解决方法！
＂悸动的青春ゞ's专栏
07-11 1162
Hadoop2.2.0-CHD4中NameNode启动不了的解决方法！
CDH环境中NodeManager无法启动，ResourceManager无法启动
weixin_42578941的博客
05-05 5709
CDH环境中NodeManager无法启动，ResourceManager无法启动
resourcemanager和nodemanager启动不了
最新发布
09-15
当ResourceManager和NodeManager启动不了时，通常可能的原因有： 1. 配置错误：YARN的配置文件（如yarn-site.xml）中的设置可能不正确，比如资源限制、主机名、端口号等。 2. 环境问题：运行ResourceManager和...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值