spark的topn问题

1、按第一个字段从大到小降序取出前三个字段

3,zhangsan
5,lisi
6,wangwu
7,wermaziang
1,bjsxt
4,shsxt
5,xiansxt
6,gzsxt
9,laogao
0,xiaogao
3,laoxiao
JavaRDD<String> linesRDD = sc.textFile("top.txt");
JavaPairRDD<Integer, String> pairRDD = linesRDD.mapToPair(new PairFunction<String, Integer, String>() {
    private static final long serialVersionUID = 1L;

    @Override
    public Tuple2<Integer, String> call(String val) throws Exception {
        String[] splited = val.split(",");
        Integer userId = Integer.valueOf(splited[0]);
        return new Tuple2<Integer, String>(userId,val);
    }
});
JavaPairRDD<Integer, String> sortByKeyRDD = pairRDD.sortByKey(false);
List<Tuple2<Integer, String>> top3 = sortByKeyRDD.take(3);
for (Tuple2<Integer, String> tuple2 : top3) {
    System.out.println(tuple2._2);
}

2、取出每门课程的分数前三个

class1  100
class2 85
class3 70
class1 102
class2 65
class1 45
class2 85
class3 70
class1 16
class2 88
class1 95
class2 37
class3 98
class1 99
class2 23
class4 2
JavaRDD<String> linesRDD = sc.textFile("scores.txt");
JavaPairRDD<String, Integer> pairRDD = linesRDD.mapToPair(new PairFunction<String, String, Integer>() {
    @Override
    public Tuple2<String, Integer> call(String s) throws Exception {
        String[] split = s.split("\t");
        String className = split[0];
        Integer scores = Integer.parseInt(split[1]);
        Tuple2<String, Integer> tuple2 = new Tuple2<String, Integer>(className, scores);
        return tuple2;
    }
});
JavaPairRDD<String, Iterable<Integer>> stringIterableJavaPairRDD = pairRDD.groupByKey();
stringIterableJavaPairRDD.foreach(new VoidFunction<Tuple2<String, Iterable<Integer>>>() {
    @Override
    public void call(Tuple2<String, Iterable<Integer>> tuple) throws Exception {
        String className = tuple._1;
        //Iterable<Integer> scores = tuple._2
        Iterator<Integer> scores = tuple._2.iterator();
        Integer[] scoreTop3 = new Integer[3];
        while (scores.hasNext()){
            Integer score = scores.next();
            for (int i = 0; i < scoreTop3.length; i++) {
                if(scoreTop3[i] == null){
                    scoreTop3[i] = score;
                    break;
                }else if(score > scoreTop3[i]){
                    for (int j = 2; j > i; j--) {
                        scoreTop3[j] = scoreTop3[j-1];
                    }
                    scoreTop3[i] = score;
                    break;
                }
            }
        }
        System.out.println("class Name:"+className);
        for(Integer sscore : scoreTop3){
            System.out.println(sscore);
        }
    }
});
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值