HBASE重启报错add to deadNodes and continue.

最新推荐文章于 2023-09-26 16:36:03 发布
最新推荐文章于 2023-09-26 16:36:03 发布
阅读量1.2k 收藏 1
点赞数 1
分类专栏： 常见报错
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_44455125/article/details/108763070
版权

本来是服务器磁盘满了,虽然没有太多数据,但是大佬不在没人排查,然后我一个弱鸡运维就只能给扩了下容,一个悲伤的故事从此开始…

扩容之后,我先启动了Hadoop,检查HDFS节点及yarn一切正常后,启动了HBASE,然后执行了hbase shell,进入HBASE命令行,执行list查看一下表(因为有些情况HBASE会假死,进程还在但是不能查看表),在此就报错了…

我先是查看了下HBASE 60010端口,如图:
在这里插入图片描述
可以看到,RegionServer,一个都没注册进来…
我就再次去看了RegionServer节点日志,如下(部分截取):
在这里插入图片描述

2020-09-23 16:14:32,724 WARN  [regionserver/h3/20.88.0.204:16020] regionserver.HRegionServer: reportForDuty failed; sleeping and then retrying.
2020-09-23 16:14:35,725 INFO  [regionserver/h3/20.88.0.204:16020] regionserver.HRegionServer: reportForDuty to master=h1,16000,1600847708094 with port=16020, startcode=1600847708936
2020-09-23 16:14:35,726 WARN  [regionserver/h3/20.88.0.204:16020] regionserver.HRegionServer: reportForDuty failed; sleeping and then retrying.
2020-09-23 16:14:38,727 INFO  [regionserver/h3/20.88.0.204:16020] regionserver.HRegionServer: reportForDuty to master=h1,16000,1600847708094 with port=16020, startcode=1600847708936

看他的样子是连不上HMaster,但是我60010看着没啥啊,就再去看了Hmaster的日志(部分报错内容):

2020-09-23 10:56:12,957 INFO  [h1:16000.activeMasterManager] hdfs.DFSClient: Could not obtain BP-1598488466-192.168.1.202-1589938686053:blk_1075553367_1906199 from any node: java.io.IOException: No live nodes contain current block No live nodes contain current block Block locations: 20.88.0.203:50010 Dead nodes:  20.88.0.203:50010. Will get new block locations from namenode and retry...
2020-09-23 10:56:12,957 WARN  [h1:16000.activeMasterManager] hdfs.DFSClient: DFS chooseDataNode: got # 1 IOException, will wait for 2981.932633213984 msec.
2020-09-23 10:56:15,940 WARN  [h1:16000.activeMasterManager] hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.io.IOException: Got error for OP_READ_BLOCK, self=/20.88.0.202:53036, remote=/20.88.0.203:50010, for file /hbase/MasterProcWALs/state-00000000000000005943.log, for pool BP-1598488466-192.168.1.202-1589938686053 block 1075553367_1906199
	at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:445)
	at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:410)
	at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:787)
	at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:666)
	at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:326)
	at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:570)
	at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:793)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:648)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:232)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
	at org.apache.hadoop.hbase.protobuf.generated.ProcedureProtos$ProcedureWALHeader.parseDelimitedFrom(ProcedureProtos.java:3870)
	at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.readHeader(ProcedureWALFormat.java:138)
	at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.open(ProcedureWALFile.java:76)
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLog(WALProcedureStore.java:1027)
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:990)
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:302)
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:516)
	at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1253)
	at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1165)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749)
	at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:199)
	at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1871)
	at java.lang.Thread.run(Thread.java:748)
2020-09-23 10:56:15,940 WARN  [h1:16000.activeMasterManager] hdfs.DFSClient: Failed to connect to /20.88.0.203:50010 for block, add to deadNodes and continue. java.io.IOException: Got error for OP_READ_BLOCK, self=/20.88.0.202:53036, remote=/20.88.0.203:50010, for file /hbase/MasterProcWALs/state-00000000000000005943.log, for pool BP-1598488466-192.168.1.202-1589938686053 block 1075553367_1906199
java.io.IOException: Got error for OP_READ_BLOCK, self=/20.88.0.202:53036, remote=/20.88.0.203:50010, for file /hbase/MasterProcWALs/state-00000000000000005943.log, for pool BP-1598488466-192.168.1.202-1589938686053 block 1075553367_1906199

到这就奇了怪了,我的HDFS是正常的,但是hmaster报错说连接不上DataNode?我就再次去50070查看了HDFS状态,包括检查了HDFS文件完整性,都没有问题.

既然报错DataNode连接不上,那就是两个肯定有问题,百度上面的错误都是让你检查hosts文件什么的网络问题,我的肯定不是啊…都是一个服务器的虚拟机…虽然我也检查了.

既然HMaster看不出什么,就去看看DataNode日志,翻了翻果然有发现:
在这里插入图片描述

2020-09-23 15:49:54,726 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: h1/20.88.0.202:8020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-09-23 15:49:54,726 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
java.io.IOException: Failed on local exception: java.io.IOException: 打开的文件过多; Host Details : local host is: "h3/20.88.0.204"; destination host is: "h1":8020; 
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
	at org.apache.hadoop.ipc.Client.call(Client.java:1480)
	at org.apache.hadoop.ipc.Client.call(Client.java:1413)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy15.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:152)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:402)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:500)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: 打开的文件过多
	at sun.nio.ch.IOUtil.makePipe(Native Method)
	at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
	at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:409)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713)
	at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529)
	at org.apache.hadoop.ipc.Client.call(Client.java:1452)
	... 8 more
2020-09-23 15:49:55,727 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: h1/20.88.0.202:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-09-23 15:49:56,728 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: h1/20.88.0.202:8020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-09-23 15:49:57,523 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: meta file /data/hdfs/current/BP-1598488466-192.168.1.202-1589938686053/current/finalized/subdir27/subdir160/blk_1075552308_1905140.meta is missing!
2020-09-23 15:49:57,524 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opReadBlock BP-1598488466-192.168.1.202-1589938686053:blk_1075552308_1905140 received exception java.io.FileNotFoundException: /data/hdfs/current/BP-1598488466-192.168.1.202-1589938686053/current/finalized/subdir27/subdir160/blk_1075552308_1905140.meta (打开的文件过多)

打开的文件过多???
在这请教了下大佬,大佬表示也没见过,百度了下,说是修改文件最大打开数量(默认1024)
在此之前,有做过大数据量集群压测情况(大概几亿左右),也没有出现过现在的问题(现在只有十几万条数据),一般来说还是没问题的,不过还是听了大佬的.

按照es的优化,修改了最大打开数量,文件描述符等.执行如下:

echo '* soft nofile 65536
* hard nofile 131072
*               soft    nproc           4096
*               hard    nproc           4096'>>/etc/security/limits.conf
    echo 'vm.max_map_count=655360' >>/etc/sysctl.conf
    sysctl -p

执行完成当前窗口用limit -n 显示还是1024,需要断开重连下生效.

然后停掉HBASE和Hadoop,重启,果然好了不报这个错,但是遇到了新的问题:
在这里插入图片描述
他开始不停的加载HDFS中的日志,我复制了路径查看了下,发现从3000开始到13000结束,大概一万个日志文件…,而HBASE两秒加载一个…
Hadoop HDFS开始和结束截图:

在这里插入图片描述

在这里插入图片描述

瞬间感觉恍然大悟:HDFS小文件太多(大概30M一个),而小文件过多容易造成HDFS NameNode节点占用大量内存等,HBASE启动又要检查这些日志,然后DataNode应该就扛不住了…

但是不是很确定没有这些HBASE还是否能用,所以停掉HBASE,将这个目录移走
在这里插入图片描述
位置在/hbase/MasterProcEALs,我将其重命名为-bak.
然后再次启动了HBASE,监控日志发现HBASE找不到这些文件,报了大量的错,但是并没有影响启动,过了一会,两个Reginaserver也注册上了…

重新执行了hbase shell ,list,恢复正常
再次执行了 scan ‘table name’,{LIMIT=>1},数据查看正常,恢复.

再次百度了这个目录的作用,类似于MySQL的binlog,记录了HBASE所有的DDL操作.

荒-于嬉
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
HBase（hbase-2.4.9-bin.tar.gz）
01-10
HBase（hbase-2.4.9-bin.tar.gz）是一个分布式的、面向列的开源数据库，该技术来源于 Fay Chang 所撰写的Google论文“Bigtable：一个结构化数据的分布式存储系统”。就像Bigtable利用了Google文件系统（File System...
hdfs获取文件信息错误
wang1hong2lu3的专栏
08-03 983
本地下载hdfs文件出错问题
hadoop2.7.5伪分布式api操作时报错
qq_45626867的博客
03-29 293
报错 [LocalJobRunner Map Task Executor #0] WARN org.apache.hadoop.hdfs.DFSClient - Failed to connect to /172.17.0.15:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection timed out: no further information java.net.ConnectExcep
Spark | HDFS 本地调式时出现：Failed to connect to /192.168.0.53:9866 for block, add to deadNodes and continu
hello_heheha的博客
11-16 2968
出现原因 21/11/16 15:13:17 WARN BlockReaderFactory: I/O error constructing remote block reader. java.net.ConnectException: Connection timed out: no further information ... ... Failed to connect to /192.168.0.53:9866 for block, add to deadNodes and continue. ja
Failed to connect to /XXXfor block, add to deadNodes and continue. java.net.NoRouteToHostException:
热门推荐
KLordy的博客
10-08 1万+
15/10/09 00:18:18 INFO DFSClient: Could not obtain BP-1056880277-192.168.1.101-1444127746416:blk_1073741825_1001 from any node: java.io.IOException: No live nodes contain current block. Will get new b
spark 访问hdfs 50010 for block, add to deadNodes and continue.
wang1hong2lu3的专栏
12-02 701
Failed to connect to /192.168.1.125:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection timed out: no further information java.net.ConnectException: Connection timed out: no further information
hadoop错误Could not obtain block blk_XXX_YYY from any node：java.io.IOException：No live nodes contain c...
10-08 731
错误： 10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes contain current block. Will get new bl...
聊聊复杂网络环境下hdfs的BlockMissingException异常|参数dfs.client.use.datanode.hostname
最新发布
明哥的IT随笔
09-26 589
聊聊复杂网络环境下hdfs的BlockMissingException异常|参数dfs.client.use.datanode.hostname。
No live nodes contain current block. Will get new block locations from namenode and retry...
qiuchenl的专栏
07-23 8096
1   多个用户操作hdfs和hbase时，出现以下异常，大概意思就是无法连接datanode，获取不到数据 INFO hdfs.DFSClient: Could not obtain block blk_-3181406624357578636_19200 from any node: java.io.IOException: No live nodes contain current blo
用vscode写微信小程序时，设置自动将less文件编译成wxss文件
weixin_44218060的博客
01-09 2191
##第一步：下载easy less插件 ##第二步：在设置setting文件中进行配置 ###打开设置setting文件 ##第三步：在setting.xml中添加如下配置： "less.compile": { "outExt": ".wxss" } 重新新建less保存就可看到自动生成wxss文件了 ...
hbase-2.4.11-bin.tar.gz
04-19
https://dlcdn.apache.org/hbase/2.4.11/ hbase-2.4.11 稳定版安装部署包
hbase-1.2.1-bin.tar.gz.zip
11-06
hbase-1.2.1-bin.tar.gz.zip 提示：先解压再使用，最外层是zip压缩文件
hbase-2.2.6-bin.tar.gz
10-28
hbase-2.2.6-bin.tar.gz HBase是一个分布式的、面向列的开源数据库，该技术来源于 Fay Chang 所撰写的Google论文“Bigtable：一个结构化数据的分布式存储系统”。就像Bigtable利用了Google文件系统（File System）所...
hbase-1.6.0-bin.tar.gz
10-17
HBase1.6.0安装包
hbase-2.1.9-bin.tar.gz
05-19
hbase-2.1.9-bin.tar.gz
hbase-2.2.7-bin.tar.gz
05-21
hbase-2.2.7-bin.tar.gz
hbase-1.2.6.1-bin.tar.gz
08-09
hbase-1.2.6.1-bin.tar.gz,hbase-1.2.6.1-bin.tar.gz,hbase-1.2.6.1-bin.tar.gz,hbase-1.2.6.1-bin.tar.gz,hbase-1.2.6.1-bin.tar.gz,hbase-1.2.6.1-bin.tar.gz,hbase-1.2.6.1-bin.tar.gz,hbase-1.2.6.1-bin.tar.gz
hbase-2.0.5-bin.tar.gz
01-25
hbase-2.0.5-bin.tar.gz
hbase-1.2.4-bin.tar.gz.zip
06-07
HBase是一个分布式的、面向列的开源数据库，该技术来源于 Fay Chang 所撰写的Google论文“Bigtable：一个结构化数据的分布式存储系统”。就像Bigtable利用了Google文件系统（File System）所提供的分布式数据存储...
hbase启动报错 Failed to load class "org.slf4j.impl.StaticLoggerBinder".
09-21
当在启动HBase时遇到"Failed to load class org.slf4j.impl.StaticLoggerBinder"的错误时，这通常是由于缺少SLF4J的绑定引起的。SLF4J是一个为Java程序提供简单日志门面的库，它需要一个实际的日志记录实现作为绑定...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值