1,ERROR Utils: Aborting task java.io.IOException: key out of order: For after package
先上代码
public static void main(String[] args) throws ClassNotFoundException {
SparkConf sparkConf = new SparkConf().setAppName("SparkTest-MapFile").setMaster("local");
sparkConf.registerKryoClasses(new Class>[] { Class.forName("org.apache.hadoop.io.IntWritable"),
Class.forName("org.apache.hadoop.io.Text") });
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
JavaRDD jpr = ctx.textFile("/README.md");
jpr.foreach(v -> {
System.out.println(v);
});
JavaRDD words = jpr.flatMap(line -> Arrays.asList(line.split(" ")).iterator());
JavaPairRDD counts = words.mapToPair(w -> new Tuple2(w, 1))
.reduceByKey((x, y) -> x + y);
counts.mapToPair(v -> new Tuple2(new Text(v._1()), new IntWritable(v._2())))
.saveAsNewAPIHadoopFile("/tmp/test6", Text.class, IntWritable.class, MapFileOutputFormat.class);
}
按理说这样的写法是没有问题的,但是就是一直报错,不明原因,各种找也没有找到相应的解决办法,也没有找到有人跟我一样出现问题,然后自己就开始各种无头苍蝇乱找,后面MapFileOutputFormat改为了SequenceFileOutputFormat的方式保存就没问题,这就让纠结了,后面无意间看到 hadoop的Map存储方式有有序的,就想到是不是存储前要自己手动排序下,修改代码加入排序.sortByKey(),然后在进行运行,完美运行成功。
public static void main(String[] args) throws ClassNotFoundException {
SparkConf sparkConf = new SparkConf().setAppName("SparkTest-MapFile").setMaster("local");
sparkConf.registerKryoClasses(new Class>[] { Class.forName("org.apache.hadoop.io.IntWritable"),
Class.forName("org.apache.hadoop.io.Text") });
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
JavaRDD jpr = ctx.textFile("/README.md");
jpr.foreach(v -> {
System.out.println(v);
});
JavaRDD words = jpr.flatMap(line -> Arrays.asList(line.split(" ")).iterator());
JavaPairRDD counts = words.mapToPair(w -> new Tuple2(w, 1))
.reduceByKey((x, y) -> x + y);
counts.mapToPair(v -> new Tuple2(new Text(v._1()), new IntWritable(v._2())))
.sortByKey().saveAsNewAPIHadoopFile("/tmp/test6", Text.class, IntWritable.class, MapFileOutputFormat.class);
}
猜测可能JavaPairRDD对map方式的存储需要自己排序的,真正是不是这样的原因就不知道了,希望哪位大神知道的话可以告知下