Spark中wordCount的常见写法

Spark的wordCount是入门级程序，下面我以Java为基础，分享3种基础的写法

1.JavaRDD加reduceByKey写法

SparkConf conf = new SparkConf();
conf.setAppName("topN");
conf.setMaster("local");
JavaSparkContext jsc = new JavaSparkContext(conf);
JavaRDD<String> lines = jsc.textFile("E:\\spark\\learn\\test.txt", 3);
//先分割每一行的词
JavaPairRDD<String, Integer> pairRdd = lines.flatMapToPair(new PairFlatMapFunction<String, String, Integer>(){

    public Iterator<Tuple2<String, Integer>> call(String s) throws Exception {
        List<Tuple2<String, Integer>> list = new ArrayList<Tuple2<String, Integer>>();
        String[] arr = s.split(" ");
        for(String str : arr){
            list.add(new Tuple2<String, Integer>(str, 1));
        }
        return list.iterator();
    }
});
//用reduceByKey去聚合相同的词
//后面两步，可以直接用reduceByKeyLocally（reduceByKey+collectAsMap）算子替代，这里更清楚将其分开
JavaPairRDD<String, Integer> pairCount = pairRdd.reduceByKey(new Function2<Integer, Integer, Integer>(){
    public Integer call(Integer v1, Integer v2){
        return v1 + v2;
    }
});

List<Tuple2<String, Integer>> list = pairCount .collect();
for(Tuple2<String, Integer> tuple2 : list){
    System.out.println(tuple2._1+"-"+tuple2._2);
}

2.JavaRDD+groupByKey

第一步也是先分割词，将每个词转化为（word, 1）的元组形式

下面用groupByKey将所有的词放到一个迭代器中，然后加每个迭代器中的值

JavaPairRDD<String, Iterable<Integer>> wordGroup = wordPair.groupByKey();
JavaPairRDD<String, Integer> wordCount = wordGroup.mapToPair(new PairFunction<Tuple2<String, Iterable<Integer>>, String, Integer>(){
    public Tuple2<String, Integer> call(Tuple2<String, Iterable<Integer>> t){
        Iterable<Integer> it = t._2();
        int num = 0;
        for(Integer i : it){
            num = num + i;
        }
        return new Tuple2<String, Integer>(t._1(), num);
    }

});
List<Tuple2<String, Integer>> list = wordCount.collect();
for(Tuple2<String, Integer> t : list){
    System.out.println(t._1+"-"+t._2);
}

3.SparkSQL方式

第一步先切割词，不用转元组形式

JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>(){
    public Iterator<String> call(String str){
        String[] arr = str.split(" ");
        List<String> list = Arrays.asList(arr);
        return list.iterator();
    }
});
//把每个词转化为Row类型
JavaRDD<Row> wordRow = words.map(new Function<String, Row>(){
   public Row call(String s){
       return RowFactory.create(s);
   }
});
//下面需要构建结构化数据，用于sql语句的使用
//这里用的Spark2.0，是SparkSession和DataSet，之前版本的可能有些可以用SQLContext和DataFrame的替代
SparkSession ss = SparkSession.builder().config(conf).getOrCreate();
List<StructField> fields = new ArrayList<StructField>();
StructField f1 = DataTypes.createStructField("word", DataTypes.StringType, true);
fields.add(f1);
StructType st = DataTypes.createStructType(fields);
Dataset ds = ss.createDataFrame(wordRow, st);
ds.createOrReplaceTempView("allWord");
Dataset rst = ss.sql("select word, count(*) as count from allWord group by word");
List<Row> list = rst.collectAsList();
for(Row r : list){
    System.out.println(r.get(0)+"-"+r.get(1));
}