【SparkAPI JAVA版】JavaPairRDD——distinct（十四）

JavaPairRDD的distinct方法讲解

官方文档

 /**
   * Return a new RDD containing the distinct elements in this RDD.
   */

说明

返回去重的一个新的RDD

函数原型

// java
public JavaPairRDD<K,V> distinct()
public JavaPairRDD<K,V> distinct(int numPartitions)
// scala
def distinct(): JavaPairRDD[K, V]
def distinct(numPartitions: Int): JavaPairRDD[K, V]

示例

public class Distinct {
    public static void main(String[] args) {

        System.setProperty("hadoop.home.dir", "E:\\hadoop-2.7.1");
        SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("Spark_DEMO");
        JavaSparkContext sc = new JavaSparkContext(sparkConf);
        JavaPairRDD<String, String> javaPairRDD1 = sc.parallelizePairs(Lists.newArrayList(
                new Tuple2<String, String>("cat", "11"), new Tuple2<String, String>("dog", "22"),
                new Tuple2<String, String>("cat", "11"), new Tuple2<String, String>("pig", "44"),
                new Tuple2<String, String>("duck", "55"), new Tuple2<String, String>("cat", "11"),
                new Tuple2<String, String>("cat", "12"), new Tuple2<String, String>("dog", "23"),
                new Tuple2<String, String>("cat", "11"), new Tuple2<String, String>("pig", "22"),
                new Tuple2<String, String>("duck", "55"), new Tuple2<String, String>("cat", "15")), 2);
        javaPairRDD1.foreach(new VoidFunction<Tuple2<String, String>>() {
            public void call(Tuple2<String, String> stringStringTuple2) throws Exception {
                System.out.println(stringStringTuple2);
            }
        });
        
        // 去重
        JavaPairRDD<String,String> javaPairRDD = javaPairRDD1.distinct();
        javaPairRDD.foreach(new VoidFunction<Tuple2<String, String>>() {
            public void call(Tuple2<String, String> stringStringTuple2) throws Exception {
                System.out.println(stringStringTuple2);
            }
        });
        // 输出分区数
        System.out.println("分区的个数："+javaPairRDD.partitions().size());

        // 带有numPartitions参数的
        JavaPairRDD<String,String> javaPairRDD2 = javaPairRDD1.distinct(3);

        javaPairRDD2.foreach(new VoidFunction<Tuple2<String, String>>() {
            public void call(Tuple2<String, String> stringStringTuple2) throws Exception {
                System.out.println("-->"+stringStringTuple2);
            }
        });
        // 输出分区数
        System.out.println("分区的个数："+javaPairRDD2.partitions().size());
    }
}

结果

19/03/21 10:53:57 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
(cat,11)
(dog,22)
(cat,11)
(pig,44)
(duck,55)
(cat,11)
19/03/21 10:53:57 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
(cat,12)
(dog,23)
(cat,11)
(pig,22)
(duck,55)
(cat,15)
19/03/21 10:53:57 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 110 ms on localhost (executor driver) (1/2)
19/03/21 10:53:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 16 ms
(cat,15)
(cat,12)
(cat,11)
(pig,22)
(pig,44)
19/03/21 10:53:57 INFO Executor: Finished task 0.0 in stage 2.0 (TID 4). 1052 bytes result sent to driver
19/03/21 10:53:57 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 62 ms on localhost (executor driver) (1/2)
(dog,23)
(dog,22)
(duck,55)
分区的个数：2
19/03/21 10:53:57 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks



19/03/21 10:53:57 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 8) in 16 ms on localhost (executor driver) (1/3)
-->(cat,15)
-->(dog,22)
-->(duck,55)
-->(pig,22)
-->(pig,44)
19/03/21 10:53:57 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 2 blocks
19/03/21 10:53:57 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 9) in 16 ms on localhost (executor driver) (2/3)
19/03/21 10:53:57 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/03/21 10:53:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/03/21 10:53:57 INFO Executor: Finished task 2.0 in stage 4.0 (TID 10). 923 bytes result sent to driver
19/03/21 10:53:57 INFO TaskSetManager: Finished task 2.0 in stage 4.0 (TID 10) in 0 ms on localhost (executor driver) (3/3)
19/03/21 10:53:57 INFO DAGScheduler: ResultStage 4 (foreach at Distinct.java:45) finished in 0.032 s
-->(dog,23)
-->(cat,12)
-->(cat,11)
分区的个数：3
19/03/21 10:53:57 INFO DAGScheduler: Job 2 finished: foreach at Distinct.java:45, took 0.127207 s

从日志里可以看出来distinct不改变分区数，但是分区的数据会去重后改变，不是单独去重。而且参数numPartitions指定多少分区，就会生成多少分区。有可能会返回空数据的分区。