org.apache.hadoop.mapred.Task - Failure committing: java.io.IOException: Could not rename

最新推荐文章于 2023-03-08 11:55:00 发布

Supper宝宝

最新推荐文章于 2023-03-08 11:55:00 发布

阅读量345

点赞数

分类专栏： hadoop输出多个文件文章标签： hadoop

本文链接：https://blog.csdn.net/kebexue/article/details/107303387

版权

hadoop输出多个文件专栏收录该内容

1 篇文章 0 订阅

订阅专栏

多个文件输出时报错：
06:20:30,429 [pool-3-thread-1] WARN org.apache.hadoop.fs.FileUtil - Failed to delete file or dir [F:\projects\output_temporary\0_temporary\attempt_local722729032_0001_r_000000_0.data-r-00000.crc]: it still exists.
06:20:30,430 [pool-3-thread-1] WARN org.apache.hadoop.fs.FileUtil - Failed to delete file or dir [F:\projects\output_temporary\0_temporary\attempt_local722729032_0001_r_000000_0.data0-r-00000.crc]: it still exists.
06:20:30,431 [pool-3-thread-1] WARN org.apache.hadoop.fs.FileUtil - Failed to delete file or dir [F:\projects\output_temporary\0_temporary\attempt_local722729032_0001_r_000000_0.data1-r-00000.crc]: it still exists.
06:20:30,433 [pool-3-thread-1] WARN org.apache.hadoop.fs.FileUtil - Failed to delete file or dir [F:\projects\output_temporary\0_temporary\attempt_local722729032_0001_r_000000_0\data-r-00000]: it still exists.
06:20:30,434 [pool-3-thread-1] WARN org.apache.hadoop.fs.FileUtil - Failed to delete file or dir [F:\projects\output_temporary\0_temporary\attempt_local722729032_0001_r_000000_0\data0-r-00000]: it still exists.
06:20:30,435 [pool-3-thread-1] WARN org.apache.hadoop.fs.FileUtil - Failed to delete file or dir [F:\projects\output_temporary\0_temporary\attempt_local722729032_0001_r_000000_0\data1-r-00000]: it still exists.
06:20:30,436 [pool-3-thread-1] WARN org.apache.hadoop.mapred.Task - Failure committing: java.io.IOException: Could not rename file:/F:/projects/output/_temporary/0/_temporary/attempt_local722729032_0001_r_000000_0 to file:/F:/projects/output/_temporary/0/task_local722729032_0001_r_000000
解决方案：
在reduce阶段重写cleanup方法，将其关闭

// 关闭资源
@Override
protected void cleanup(Context context) throws IOException, InterruptedException {
multipleOutputs.close();
}