【flinkx】【hdfs】【ing】Cannot obtain block length for LocatedBlock

一. 任务描述

使用flinkx去跑HDFS到HIVE的任务时，出现如下报错：

 

CannotObtainBlockLengthException

com.dtstack.flinkx.throwable.FlinkxRuntimeException: can't get file size from hdfs, file = hdfs://xxx/.data/540240453caeb6fe4b3f118410a05315_235_0
	at com.dtstack.flinkx.connector.hdfs.sink.BaseHdfsOutputFormat.getCurrentFileSize(BaseHdfsOutputFormat.java:169) ~[flinkx-connector-hdfs-1.12-SNAPSHOT.jar:?]
	at com.dtstack.flinkx.sink.format.BaseFileOutputFormat.checkCurrentFileSize(BaseFileOutputFormat.java:137) ~[flinkx-core-1.12-SNAPSHOT.jar:?]
	at com.dtstack.flinkx.sink.format.BaseFileOutputFormat.writeSingleRecordInternal(BaseFileOutputFormat.java:128) ~[flinkx-core-1.12-SNAPSHOT.jar:?]
	at com.dtstack.flinkx.sink.format.BaseRichOutputFormat.writeSingleRecord(BaseRichOutputFormat.java:427) ~[flinkx-core-1.12-SNAPSHOT.jar:?]
	at com.dtstack.flinkx.sink.format.BaseRichOutputFormat.writeRecord(BaseRichOutputFormat.java:244) ~[flinkx-core-1.12-SNAPSHOT.jar:?]
	at com.dtstack.flinkx.sink.format.BaseRichOutputFormat.writeRecord(BaseRichOutputFormat.java:84) ~[flinkx-core-1.12-SNAPSHOT.jar:?]
	at com.dtstack.flinkx.sink.DtOutputFormatSinkFunction.invoke(DtOutputFormatSinkFunction.java:117) ~[flinkx-core-1.12-SNAPSHOT.jar:?]
	at org.apache.flink.table.runtime.operators.sink.SinkOperator.processElement(SinkOperator.java:72) ~[flink-table-blink_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at StreamExecCalc$12.processElement(Unknown Source) ~[?:?]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at com.dtstack.flinkx.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:135) ~[flinkx-core-1.12-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:267) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
Caused by: org.apache.hadoop.hdfs.CannotObtainBlockLengthException: Cannot obtain block length for LocatedBlock{BP-908532681-10.203.2.20-1593930010000:blk_2513177302_1108300914318; getBlockSize()=0; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.203.70.78:9866,DS-9aa641e8-4ce8-436b-9eaa-fd3eb676185e,DISK], DatanodeInfoWithStorage[10.203.22.205:9866,DS-cb07688e-a349-46eb-8929-71d8fd6156dd,DISK], DatanodeInfoWithStorage[10.203.34.146:9866,DS-efa9e16d-405b-4a05-8e52-0e35d681c3d2,DISK]]}
	at org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:363) ~[hadoop-hdfs-client-3.2.1U20.jar:?]
	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270) ~[hadoop-hdfs-client-3.2.1U20.jar:?]
	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:201) ~[hadoop-hdfs-client-3.2.1U20.jar:?]
	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:185) ~[hadoop-hdfs-client-3.2.1U20.jar:?]
	at org.apache.hadoop.hdfs.DFSClient.openInternal(DFSClient.java:1048) ~[hadoop-hdfs-client-3.2.1U20.jar:?]
	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1011) ~[hadoop-hdfs-client-3.2.1U20.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:359) ~[hadoop-hdfs-client-3.2.1U20.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:355) ~[hadoop-hdfs-client-3.2.1U20.jar:?]
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.2.1U20.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:367) ~[hadoop-hdfs-client-3.2.1U20.jar:?]
	at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:164) ~[hadoop-common-3.2.1U20.jar:?]
	at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:285) ~[hadoop-common-3.2.1U20.jar:?]
	at org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:500) ~[hadoop-common-3.2.1U20.jar:?]
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:912) ~[hadoop-common-3.2.1U20.jar:?]
	at com.dtstack.flinkx.connector.hdfs.sink.BaseHdfsOutputFormat.getCurrentFileSize(BaseHdfsOutputFormat.java:166) ~[flinkx-connector-hdfs-1.12-SNAPSHOT.jar:?]
	... 24 more

看报错可以得到当时任务的一些消息

文件没有了 还去获取文件
文件还在写的时候去获取文件

暂时怀疑flinkx的hdfs写操作会偶发的出现，或者当达到什么机制时后，会出现。

 
 

二、google解决方案

Usually when you see “Cannot obtain block length for LocatedBlock”, this means the file is still in being-written state, i.e., it has not been closed yet, and the reader cannot successfully identify its current length by communicating with corresponding DataNodes. There are multiple possibilities here, .e.g., there may be temporary network connection issue between the reader and the DataNodes, or the original writing failed some while ago and the under-construction replicas are somehow missing.
 
In general you run fsck command to get more information about the file. You can also trigger lease recovery for further debugging. Run command:
 
hdfs debug recoverLease -path <path-of-the-file> -retries <retrytimes>
 
This command will ask the NameNode to try to recover the lease for the file, and based on the NameNode log you may track to detailed DataNodes to understand the states of the replicas. The command may successfully close the file if there are still healthy replicas. Otherwise we can get more internal details about the file/block state.

 
当看到这个消息时：Cannot obtain block length for LocatedBlock,说明文件仍然是写的状态，即写还没有关闭，client无法成功的和datanodes通讯获取到当前文件的长度。
 
有以下几种可能的原因会出现这种问题：

1. 可能是短暂的网络通讯问题导致client和dataNodes不能通讯 来获取到文件信息
2. flink写某些文件时，写失败了并且创建副本时，副本因为一些原因丢失了。

解决：

1. 可以通过run fsck相关命令获取文件更多的信息。
2. 或者执行如上debug命令来请求namenode来恢复文件的租约，并且基于namenode日志可以追踪到datanode的一些细节来理解副本的状态。如果文件有健康的副本，此时会关闭文件。

 

三、进一步逻辑探究

1. hdfs的租约机制

ing