hdfs java 追加,java – 如何在非常小的集群上附加一个hdfs文件(3个节点或更少)

最新推荐文章于 2023-06-27 16:47:52 发布

小薇酱

最新推荐文章于 2023-06-27 16:47:52 发布

阅读量75

点赞数

文章标签： hdfs java 追加

我试图附加到单个节点集群上的hdfs上的文件.我也尝试了一个2节点集群,但得到相同的例外.

在hdfs-site中,我将dfs.replication设置为1.如果我将dfs.client.block.write.replace-datanode-on-failure.policy设置为DEFAULT,我得到以下异常

java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[10.10.37.16:50010],original=[10.10.37.16:50010]). The current Failed datanode replacement policy is DEFAULT,and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.

如果我按照configuration in hdfs-default.xml的评论为非常小的集群(3个节点或更少)的建议,并设置dfs.client.block.write.replace-datanode-on-failure.policy,以至于我得不到以下异常：

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot append to file/user/hadoop/test. Name node is in safe mode.

The reported blocks 1277 has reached the threshold 1.0000 of total blocks 1277. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 3 seconds.

这是我如何追加：

Configuration conf = new Configuration();

conf.set("fs.defaultFS","hdfs://MY-MACHINE:8020/user/hadoop");

conf.set("hadoop.job.ugi","hadoop");

FileSystem fs = FileSystem.get(conf);

OutputStream out = fs.append(new Path("/user/hadoop/test"));

PrintWriter writer = new PrintWriter(out);

writer.print("hello world");

writer.close();

有什么我在代码中做错了吗？

也许,在配置中有些东西丢失？

任何帮助将不胜感激！