写HDFS任务时不时发生Checkpoint超时,查看了日志发下如下异常:
location:org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1495)
throwable:java.io.IOException: Got error, status message , ack with firstBadLink as ip:port
at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:142)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1482)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1385)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:554)
同时排查发现问题出现时,对应的机器负载突然飙高
原因:对应的机器是HDFS DataNode所在的机器,由于该机器上跑了比较吃CPU的离线任务,DataNode读的压力很大,导致写不能及时响应,所以flink在写snapshot时有部分subtask响应超时,进而导致Checkpoint超时失败。