// 限制spark输出往hdfs写._SUCCESS.crc文件
Configuration hconf = sc.hadoopConfiguration();
hconf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false");
// 限制spark输出往hdfs写.part-00000.crc文件
try {
FileSystem fileSystem = FileSystem.get(hconf);
fileSystem.setWriteChecksum(false);
} catch (IOException e) {
e.printStackTrace();
}
Spark禁止输出标记文件设置
最新推荐文章于 2024-07-18 16:36:56 发布