hadoop定位hdfs文件块异常和修复Cannot obtain block length for LocatedBlock

最新推荐文章于 2023-02-12 11:20:42 发布

jiandequn

最新推荐文章于 2023-02-12 11:20:42 发布

阅读量1.8k

点赞数 3

分类专栏： java hadoop

本文链接：https://blog.csdn.net/jiandequn/article/details/103292966

版权

当遇到Hadoop MapReduce任务处理时出现无法获取块长度的异常，可以使用`hdfs fsck /path -openforwrite`检查打开未关闭的文件。发现异常文件后，通过`hdfs debug recoverLease`命令尝试修复，例如对/data/logs/2019/11/27/16/data-log-20191127_16.1574841601929.log执行修复操作。参考相关教程进行操作。

摘要由CSDN通过智能技术生成

1、mapreduce任务处理时，产生异常信息，不能获取块的长度信息，错误日志如下：

 org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1574843806023_0008_m_000002_3: Error: java.lang.RuntimeException: org.apache.hadoop.hdfs.CannotObtainBlockLengthException: Cannot obtain block length for LocatedBlock{BP-814407275-192.168.65.10-1562911561103:blk_1073747950_7139; getBlockSize()=764; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[192.168.65.11:9866,DS-0380594d-516b-4ce5-87cc-7527d150eadc,DISK], DatanodeInfoWithStorage[192.168.65.10:9866,DS-249289b1-0ede-4ead-bb30-f8338af5c73e,DISK]]}
	at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:161)
	at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.nextKeyValue(CombineFileRecordReader.java:66)
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:568)
	at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hadoop.hdfs.CannotObtainBlockLengthException: Cann