ERROR org.apache.flume.sink.hdfs.AbstractHDFSWriter: Error while trying to hflushOrSync!

2022-11-14 11:12:31,890 ERROR org.apache.flume.sink.hdfs.AbstractHDFSWriter: Error while trying to hflushOrSync!
2022-11-14 11:12:31,890 WARN org.apache.flume.sink.hdfs.BucketWriter: Closing file: /origin_data/ctp_info_45/2022-11-11/a30-ctp_info_45-data-2022-11-11-00-00-00.1668097963853.tmp failed. Will retry again in 180 seconds.
java.nio.channels.ClosedChannelException
    at org.apache.hadoop.hdfs.ExceptionLastSeen.throwException4Close(ExceptionLastSeen.java:73)
    at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:153)
    at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:625)
    at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:580)
    at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:134)
    at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:268)
    at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:133)
    at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:319)
    at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:316)
    at org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:727)
    at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
    at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:724)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748) 

业务影响：消费kafka数据，落地hdfs。装载到hive表中发现表缺失数据

临时解决方法是使用命令：hdfs debug recoverLease -path /xxx/xxx手工关闭打开状态的文件。

Flume版本：1.9.0

初步分析的原因有

1、网络传输问题

2、可能是namenode相应关闭文件的rpc请求有瓶颈

3、hdfs.callTimeout的默认值是30000毫秒，hdfs.retryInterval的默认值是180秒。调整为hdfs.callTimeout=60000，hdfs.retryInterval=360

#待更新补充