NAMENODE双活，数据不能写入

最新推荐文章于 2023-07-25 21:30:26 发布
最新推荐文章于 2023-07-25 21:30:26 发布
阅读量528 收藏
点赞数
分类专栏： linux 大数据 hdfs 文章标签： hadoop linux ambari
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/jzy3711/article/details/124123501
版权
大数据 同时被 3 个专栏收录
40 篇文章 1 订阅
订阅专栏
linux
11 篇文章 0 订阅
订阅专栏
hdfs
10 篇文章 0 订阅
订阅专栏
集群遇到HDFS安全模式问题，导致数据无法写入。原因是坏块导致NameNode进入安全模式，通过`hdfs fsck`查询并删除坏块，使用`hdfs debug recoverLease`修复。之后出现日志同步错误，调整`dfs.qjournal.write-txns.timeout.ms`和`ha.health-monitor.rpc-timeout.ms`参数解决。目前集群运行稳定。
摘要由CSDN通过智能技术生成

背景

公司在黑龙江的集群用了7年，最近总是遇到namenode双活，数据无法写入问题。

问题

2022-03-28 11:02:38,318 INFO  ipc.Server (Server.java:run(2172)) - IPC Server handler 49 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 136.192.59.173:43484 Call#4 Retry#3
org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/db_app/flg_asia_serv_info_d/day_id=20220224/000218_0.lzo_deflate. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1810)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/db_app/flg_asia_serv_info_d/day_id=20220224/000218_0.lzo_deflate. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1806)
	... 12 more
2022-03-28 11:02:38,319 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,319 INFO  ipc.Server (Server.java:run(2172)) - IPC Server handler 68 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 136.192.59.178:54519 Call#22650 Retry#2
org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/in_bil/contact_push_msg/day_id=20211120/000162_0.lzo_deflate. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1810)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/in_bil/contact_push_msg/day_id=20211120/000162_0.lzo_deflate. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1806)
	... 12 more
2022-03-28 11:02:38,319 INFO  ipc.Server (Server.java:run(2172)) - IPC Server handler 89 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 136.192.59.132:46251 Call#24119 Retry#1
org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/in_bil/contact_task/day_id=20210924/000197_0.lzo_deflate. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1810)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/in_bil/contact_task/day_id=20210924/000197_0.lzo_deflate. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1806)
	... 12 more
2022-03-28 11:02:38,320 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,320 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,320 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,320 INFO  ipc.Server (Server.java:run(2172)) - IPC Server handler 50 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.complete from 136.192.59.66:60785 Call#38290463 Retry#1
org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot complete file /data/oc_guoxin_self_data/boncdb/dwa/dwa_mobile_data_event_new/20220328/0469/20220328100252.txt. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1331)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3494)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:851)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:536)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot complete file /data/oc_guoxin_self_data/boncdb/dwa/dwa_mobile_data_event_new/20220328/0469/20220328100252.txt. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327)
	... 12 more
2022-03-28 11:02:38,320 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,320 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,320 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,320 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,383 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,322 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,385 INFO  ipc.Server (Server.java:run(2172)) - IPC Server handler 39 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 136.192.59.167:49984 Call#22762 Retry#2
org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/in_bil/contact_push_msg/day_id=20211224/000051_0.lzo_deflate. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1810)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/in_bil/contact_push_msg/day_id=20211224/000051_0.lzo_deflate. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1806)
	... 12 more
2022-03-28 11:02:38,321 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,385 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,386 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,384 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,383 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,386 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,386 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,386 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,386 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,387 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,387 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,387 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,387 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,390 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,390 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,390 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,390 INFO  ipc.Server (Server.java:run(2172)) - IPC Server handler 30 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.complete from 136.192.59.69:42099 Call#55478308 Retry#1
org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot complete file /data/oc_guoxin_self_data/boncdb/itf/t_4glog_roam_d/20220328/08/LOGROAM_23_5MIN_2022032808500_20220328080959_20220328080722_mbl_dpi_HrcWhlgq86699.gz. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1331)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3494)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:851)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:536)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot complete file /data/oc_guoxin_self_data/boncdb/itf/t_4glog_roam_d/20220328/08/LOGROAM_23_5MIN_2022032808500_20220328080959_20220328080722_mbl_dpi_HrcWhlgq86699.gz. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327)
	... 12 more
2022-03-28 11:02:38,390 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,390 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,390 INFO  ipc.Server (Server.java:run(2172)) - IPC Server handler 61 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.complete from 136.192.59.66:60785 Call#38290464 Retry#1
org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot complete file /data/oc_guoxin_self_data/boncdb/dwa/dwa_mobile_data_event_new/20220328/0452/20220328105703.txt. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1331)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3494)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:851)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:536)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot complete file /data/oc_guoxin_self_data/boncdb/dwa/dwa_mobile_data_event_new/20220328/0452/20220328105703.txt. Name node is in safe mode.
The reported blocks 22878610 needs additional 42322789 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327)
	... 12 more
2022-03-28 11:02:38,391 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,392 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,746 INFO  destination.HDFSAuditDestination (HDFSAuditDestination.java:createConfiguration(263)) - Returning HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml
2022-03-28 11:02:38,795 INFO  destination.HDFSAuditDestination (HDFSAuditDestination.java:getLogFileStream(224)) - Checking whether log file exists. hdfPath=hdfs://hljcluster/ranger/audit/hdfs/20220328/hdfs_ranger_audit_hljnn01.asiainfo.com.log, UGI=ocdp (auth:SIMPLE)
2022-03-28 11:02:38,818 INFO  BlockStateChange (BlockManager.java:processReport(1939)) - BLOCK* processReport: from storage DS-dab00d6e-491d-4155-9c29-2967706ef4ed node DatanodeRegistration(136.192.59.179:50010, datanodeUuid=53d93291-9230-4348-8f2d-58357969f186, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-64402ccb-cbc1-4522-bd2a-bf5883a9e04c;nsid=1379355448;c=0), blocks: 528412, hasStaleStorage: true, processing time: 426 msecs
2022-03-28 11:02:38,818 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,818 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,818 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,820 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,820 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_guoxin
2022-03-28 11:02:38,851 WARN  security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521)) - No groups available for user oc_asiainfo
2022-03-28 11:02:38,852 INFO  ipc.Server (Server.java:run(2172)) - IPC Server handler 87 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 136.192.59.178:54446 Call#4 Retry#3
org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/db_app/flg_asia_serv_info_m/month_id=202202/000257_0.lzo_deflate. Name node is in safe mode.
The reported blocks 23282231 needs additional 41919168 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1810)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /data/data_asiainfo/hive_data/db_app/flg_asia_serv_info_m/month_id=202202/000257_0.lzo_deflate. Name node is in safe mode.
The reported blocks 23282231 needs additional 41919168 blocks to reach the threshold 0.9900 of total blocks 65859998.
The number of live datanodes 53 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1806)
	... 12 more

请添加图片描述
集群有坏块，导致进入安全模式，无法写入数据，影响业务

解决

处理坏块

#查询坏块
hdfs fsck /user/hadoop-twq/cmd -list-corruptfileblocks```
#删除坏块
hdfs fsck /data/data_asiainfo/hive_data/in_bil/contact_task/day_id=20220204/ -delete
#修复坏块必须指定到文件
hdfs debug recoverLease -path /data/data_asiainfo/hive_data/in_bil/contact_task/day_id=20220204/1.file

追踪问题

上次改完好了一段时间，没面出现如下新的问题

2022-04-02 12:01:26,670 FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: flush failed for required journal (JournalAndStream(mgr=QJM to [136.192.59.131:8485, 136.192.59.133:8485, 136.192.59.132:8485], stream=QuorumOutputStream starting at txid 14906286046))
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 3 exceptions thrown:
136.192.59.132:8485: IPC's epoch 288 is less than the last promised epoch 289
	at org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:418)
	at org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:446)
	at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:341)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:148)
	at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
	at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

136.192.59.133:8485: IPC's epoch 288 is less than the last promised epoch 289
	at org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:418)
	at org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:446)
	at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:341)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:148)
	at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
	at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

136.192.59.131:8485: IPC's epoch 288 is less than the last promised epoch 289
	at org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:418)
	at org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:446)
	at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:341)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:148)
	at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
	at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

	at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
	at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
	at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
	at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:107)
	at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113)
	at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:533)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:57)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:529)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:647)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1660)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:760)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:453)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

解决

修改hdfs-site.xml

#Write timeout in milliseconds when writing to a quorum of remote journals.
dfs.qjournal.write-txns.timeout.ms=60000

修改core-site.xml

#Timeout for the actual monitorHealth() calls.
ha.health-monitor.rpc-timeout.ms=180000

结果

到目前为止稳定运行

jzy3711
关注
专栏目录
HDFS(七)：NameNode容灾
05-07 562
在主namenode发生故障时（假设没有及时备份数据），可以从SecondaryNameNode恢复数据。 方法一：将SecondaryNameNode中数据拷贝到namenode存储数据的目录； 模拟namenode故障，并采用方法一，恢复namenode数据 1.kill -9 namenode进程(jps查看进程) 2.删除namenode存储的数据（/opt/module/hadoo...
WordCount单词统计笔记
m0_52672980的博客
10-18 1275
1.在本机的/root目录下，依次创建文件夹data,文本文件word.txt. mkdir -p /root/data vim /root/data/word.txt 键入i,进入编辑模式，输入如下内容： hello world hadoop hdfs qingjiao hadoop hongya hdfs qingjiao qingjiao hadoop hongya 键入Esc，退出编辑模式，输入:wq保存退出。 终端中执行HDFS shell命令。 # 创建目录 hadoop fs -mkd
hadoop解决文件上传问题（DataStreamer Exceptionorg.apache.hadoop.ipc.RemoteException）
生夏夏的博客
04-26 7024
在进行Hadoop环境搭建后，使用Hadoop shell命令时---上传本地文件到HDFS的目录下，出现如下报错： 由于本人新手小白一枚
【已解决】Hadoop报错：Name node is in safe mode.
还有很长很长的路
04-27 1960
问题 C:\Users\Ashley>hadoop fs -checksum /user/root/temp/5.cpp checksum: Zero blocklocations for /user/root/temp/5.cpp. Name node is in safe mode. 原因 因为某些不当的操作，进入了安全模式，退出即可。 解决 C:\Users\Ashley...
Name node is in safe mode解决
微电子学与固体电子学-俞驰
04-25 1184
[2]"main"org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException):Zero blocklocations for /user/ds/hello.txt. Name node is in safe mode. 解决方案[1]: hdfs dfs...
Hadoop-2.7.0中HDFS NameNode HA实现之DFSZKFailoverController、ZKFailoverController（一）
辰辰爸的博客
12-14 1万+
一、简介      DFSZKFailoverController是Hadoop-2.7.0中HDFS NameNode HA实现的中心组件，它负责整体的故障转移控制等。它是一个守护进程，通过main()方法启动，继承自ZKFailoverController。二、实现流程      1、启动       通过main()方法启动，如下： /** * 进程启动的main()方法 */
理解hadoop的复制策略：数据高可用和容灾
## 第一章：Hadoop复制策略概述 ### 1.1 Hadoop的数据存储架构 ...- NameNode（主节点）：负责管理文件系统的命名空间和文件块的元数据信息。 - DataNode（数据节点）：负责存储和管理文件块的实际数据。 - Secondary
HDFS集群搭建，高可用双机热备模式(HA)自动切换，hdfs+zookeeper+journalnode，步骤分步原理详解(适合初学者)
CAM的博客
07-01 7522
前 言 作者也是初学hadoop，了解各组件的用途，并且项目中有些东西用不到，所以作者用最简化的组件搭建，避免多余资源的浪费，作者最初构想是zookeeper+hdfs搭建一个分布式文件系统，不用yarn和mapReduce，但是由于双机热备需要一个共享资源的目录，所以还是用到了yarn的journalnode。后续会进行详细说明。 作者会针对每一步进行详细说明以及原理解释，还有作者踩过的
大数据八股文（自用）
最新发布
m0_64735594的博客
07-25 1232
实现的逻辑是继承GenericUDF，重写evaluate方法，getdisplay方法。打包上传到hdfs路径上或者hive的lib目录 注册自定义的函数UDTF炸裂 一行多输出 TUDAF聚合多行输出一行Aggregate前台是和用户直接交互的界面和各种实时响应用户请求的业务逻辑。后台是并不直接面向用户的配置管理系统。中台就是各个项目业务中 重复的轮子，比如交易、支付。
大数据平台建设关键技术
leveretz的博客
02-22 8489
平台关键技术设计 架构风格 云计算 Google在2006年率先提出“云计算”的概念。所谓“云计算”，是一种大规模的分布式模型，通过网络将抽象的､可伸缩的､便于管理的数据能源､服务､存储方式等传递给终端用户。狭义云计算是指IT基础设施的交付和使用模式，指通过网络以按照需求量的方式和易扩展的方式获得所需资源。广义云计算指服务交付和使用模式，指通过网络以按照需求量和易扩展方式获得所需服务。 目前，云计算可以认为包含3个层次的内容：基础设施即服务（IaaS）､平台即服务（PaaS）和软件即服务（Sa.
org.apache.hadoop.ipc.RemoteException(java.io.IOException)异常
沧海笑007
11-20 1万+
       最近在调试flink程序时，发现程序起不来，查看错误日志和hadoop相关，我的程序与hadoop相关的只有设置了checkpoint的路径是hdfs的一个目录路径。 错误日志最后的错误大致是： org.apache.hadoop.ipc.RemoteException(java.io.IOException): File / 后查看机器进程情况发现：集群中datanode没有...
java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond
weixin_33861800的博客
07-19 1741
一：任务： /datacenter/shell/user/full/bdl_social_test1.sh 跑全量数据时！Namenode挂掉！ 二：报错日志： /var/log/hadoop-hdfs/hadoop-hdfs-namenode-yhbd01.log 内容：...
hadoop源码hdfs学习-基础知识
strideahead的博客
07-19 242
在HDFS中，Namenode是HDFS的Master节点，负责管理文件系统的命名空间（namespace），以及数据块到具体DataNode节点的映射信息，集群中的DataNode一般是一个节点一个，，负责他所在节点上的存储。从内部看，一个文件被分为多个数据块，存储在以组DataNode上，以本地文件的形式保存这些数据块以及数据块的校验信息。 用户能通过HDFS客户端发起读写HDFS文件的...
Spark学习笔记@第一个例子wordcount+Eclipse
骑猪喝咖啡
05-21 4338
使用Eclipse编辑wordcount程序，打包并运行。详细记录了打包发布过程中的问题。
Hadoop数据节点DataNode异常
Eddie-Wang的博客
06-13 1159
1、问题背景： 操作系统：CentOS7.664位 Hadoop版本：Hadoop2.8.5 JDK：1.8.0_201 2、问题描述 使用start-dfs.sh、start-yarn.sh脚本启动Hadoop后，通过jps命令检测DataNode节点进程发现没有DataNode进程，只有NameNode，SecondaryNameNode，NodeManager，ResourceMa...
hadoop常见错误及解决办法！
热门推荐
沉底的石头
05-30 1万+
1：Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out  Answer： 程序里面需要打开多个文件，进行分析，系统一般默认数量是1024，（用ulimit -a可以看到）对于正常使用是够了，但是对于程序来讲，就太少了。 修改办法： 修改2个文件。        /etc/security/limits.
hadoop异常：org.apache.hadoop.ipc.RemoteException
qq_41302130的博客
08-29 1万+
在使用自定义outputformat出现异常，异常值如下： hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(949)) - Failed to close inode 17333。 或者： org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.n...
hadoop拷贝文件时 org.apache.hadoop.ipc.RemoteException异常的解决
weixin_34357962的博客
06-20 314
1、系统或hdfs是否有空间 2、datanode数是否正常 3、是否在safemode 4、防火墙关闭 5、配置方面 6、把NameNode的tmp文件清空，然后重新格式化NameNode
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值