groupByKey是对单个 RDD 的数据进行分组
def groupByKey(): RDD[(K, Iterable[V])]
def groupByKey(numPartitions: Int): RDD[(K, Iterable[V])]
def groupByKey(partitioner: Partitioner): RDD[(K, Iterable[V])]
groupByKey会将RDD[key,value] 按照相同的key进行分组,形成RDD[key,Iterable[value]]的形式, 有点类似于sql中的groupby,例如类似于mysql中的group_concat
例如这个例子, 我们对学生的成绩进行分组
JavaRDD<Tuple2<String,Float>> scoreDetails = javaSparkContext.parallelize(Arrays.asList(new Tuple2("xiaoming", 75)
, new Tuple2("xiaoming", 90)
, new Tuple2("lihua", 95)
, new Tuple2("lihua", 188)));
//将JavaRDD<Tuple2<String,Float>> 类型转换为 JavaPairRDD<String, Float>
JavaPairRDD<String, Float> scoreMapRDD = JavaPairRDD.fromJavaRDD(scoreDetails);
JavaPairRDD<String, Iterable<Float>> stringIterableJavaPairRDD = scoreMapRDD.groupByKey(2);
System.out.println(stringIterableJavaPairRDD.collectAsMap());
//返回的结果
//{lihua=[95, 188], xiaoming=[75, 90]}