记一次mr中途落盘数据丢失 hdfs超租约异常 Error: java.io.FileNotFoundException: File does not exist:

现象为：mr任务偶尔中途失败，但执行结果yarn会判定为成功，具体可以看日志报错。

hive sql中途的报错，使用的是tez引擎：

sqoop同步mysql到hdfs中的报错：

Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /user/liveme/facebook/_temporary/1/_temporary/attempt_1666686642034_1714844_m_000001_0/part-m-00001 (inode 1448130627) Holder DFSClient_attempt_1666686642034_1714844_m_000001_0_275261276_1 does not have any open files.

Error: java.io.FileNotFoundException: File does not exist: /user/liveme/facebook/_temporary/1/_temporary/attempt_1666686642034_1714844_m_000001_0/part-m-00001 (inode 1448130627) Holder DFSClient_attempt_1666686642034_1714844_m_000001_0_275261276_1 does not have any open files.

这种问题首先定位的是hdfs和hadoop基础环境的问题，因为不是稳定发生，概率也不确定，所以非常不好复现，但却关键时刻会发生给你一记背刺，最关键的是失败了他会判定为成功，你就算监控都监控不到，就很让人麻。

经过查询资料
https://doc.hcs.huawei.com/zh-cn/usermanual/mrs/mrs_03_0084.html
https://blog.csdn.net/slx_2011/article/details/30286395
https://blog.csdn.net/slx_2011/article/details/30286395
https://community.pivotal.io/s/article/Hive-job-failed-with-error-org-apache-hadoop-ipc-RemoteException-org-apache-hadoop-hdfs-server-namenode-LeaseExpiredException?language=en_US

将DataNode max data transfer threads 参数从16401 翻倍 解决问题