Spark 将输出数据保存到已经存在的目录
通过RDD的saveAsTextFile方法保存
String input = "args[0]";
String output = "args[1]";
SparkConf conf = new SparkConf();
JavaSparkContext jsc = new JavaSparkContext(conf);
JavaRDD<String> rdd = jsc.textFile(input);
JavaPairRDD<String, String> pairRdd = rdd.mapToPair(new PairFunction<String, String, String>() {
private static final long serialVersionUID = -4327506895954890835L;
@Override
public Tuple2<String, String> call(String t) throws Exception {
String[] lines = t.split(",");
return new Tuple2<String,String>(lines[0],lines[1]);
}
});
pairRdd.saveAsTextFile(output);
如果输出目录output已经存在,则会报异常:
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /output already