hadoop append 追加文件错误

最新推荐文章于 2023-06-06 12:22:30 发布

修鹏李

最新推荐文章于 2023-06-06 12:22:30 发布

阅读量8.4k

点赞数

分类专栏： hadoop

本文链接：https://blog.csdn.net/map_lixiupeng/article/details/32153251

版权

在尝试使用Hadoop HDFS进行文件追加操作时遇到错误，问题定位到DFSOutputStream.java的531行。为解决此问题，在客户端配置中添加了两行代码，将`dfs.client.block.write.replace-datanode-on-failure.policy`设置为`NEVER`，并将`dfs.client.block.write.replace-datanode-on-failure.enable`设置为`true`，以防止数据节点替换并启用相关功能。

摘要由CSDN通过智能技术生成

Exception in thread "main" java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[10.10.22.17:50010, 10.10.22.18:50010], original=[10.10.22.17:50010, 10.10.22.18:50010]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:960)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1026)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1175)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:531)

在客户端的代码里面加入：