一、元数据如下:
20161123101523 http://h5.learn.com/h5/teacher.shtml
20161123101523 http://h5.learn.com/h5/course.shtml
20161123101523 http://bigdatalearn.com/bigdata/teacher.shtml
20161123101523 http://java.learn.com/java/video.shtml
20161123101523 http://bigdata.learn.com/bigdata/teacher.shtml
20161123101523 http://ui.learn.com/ui/course.shtml
20161123101523 http://bigdata.learn.com/bigdata/teacher.shtml
20161123101523 http://h5.learn.com/h5/course.shtml
20161123101523 http://java.learn.com/java/video.shtml
20161123101523 http://ui.learn.com/ui/video.shtml
20161123101523 http://h5.learn.com/h5/course.shtml
20161123101523 http://h5.learn.com/h5/teacher.shtml
20161123101523 http://bigdatalearn.com/bigdata/teacher.shtml
20161123101523 http://bigdata.learn.com/bigdata/video.shtml
20161123101523 http://ui.learn.com/ui/teacher.shtml
20161123101523 http://java.learn.com/java/video.shtml
二、目标:
统计出每个学科排名前3的,如:
三、思路:
1、先将所有Key相同的合并,得到如下结果
java-jdbc-8
java-js2e 18
java-高并发-100
java-中间件 15
h5-画布-18
h5-js-184
h5-事件驱动-20
h5-css-192
h5-easyui-3
大数据-kudu-38
大数据-Spark-300
大数据-Sqoop-9
大数据-Hbase-150
大数据-Hadoop 280
2、按照Key分组,然后排序取前3
四、代码如下
package scalapackage.testspark
import java.net.URL
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}
/**
* Created by Germmy on 2018/5/12.
*/
object TestTop3 {
def main(args: Array[String]) {
val sparkConf: SparkConf = new SparkConf().setAppName("SparkTop3").setMaster("local[*]")
val sc: SparkContext = new SparkContext(sparkConf)
val file: RDD[String] = sc.textFile("D:\\temp\\course.txt")
//1.先reduceByKey
val res=file.map(x=>{
val split: Array[String] = x.split(" ")
val url: String = split(1)
(url,1)
})
val sumedUrls: RDD[(String, Int)] = res.reduceByKey(_+_)
//取出学科
val res2=sumedUrls.map(x=>{
val url=x._1
val count=x._2
val project=new URL(url).getHost
(project,url,count)
})
//分组统计
val values: RDD[(String, List[(String, String, Int)])] = res2.groupBy(_._1).mapValues(_.toList.sortBy(_._3).reverse.take(3))
println(values.collect().toBuffer)
}
}
五、运行结果如下