Spark实例项目—每个域名下访问次数最多的URL

生活在于积累，你所遇到的困难，问题，，，，都将成为你生活的财富，而博客，就是记录你一路成长的最好见证。
————————送给正在写作业的你
Spark 项目分析网络URL数据。加深RDD理解
要求分析出每个域名的前三个访问量是哪些 URL
数据的格式：
https://blog.csdn.net/qq_43688472/article/details/84307884 [2015-07-07 13:52:58] 54
https://item.jd.com/26838388932.html [2008-03-04 15:47:37] 81
https://blog.csdn.net/qq_24073707/article/details/80665991 [5002-10-17 09:20:02] 73
https://hizero.taobao.com/?spm=a217m.8316598.682372.3.426d33d5KRgTIh [2009-05-29 20:02:43] 63
操作：

import java.net.URL

import org.apache.spark.{SparkConf, SparkContext}

object UrlCount01 {
  def main(args: Array[String]): Unit = {


      val sparkConf = new SparkConf().setMaster("local[2]").setAppName("UrlCount01")

      val sc = new SparkContext(sparkConf)

      val line = sc.textFile("file:///E:\\data.txt" )
    //筛选数据因为有的数据可能不散URL格式的数据
    //println("line: "+line)


    //    val rdd1 = line.filter(x=>{
    //      val tmp = x.split(":")
    //      if (tmp.length>=3||x.contains("[["))
    //        false
    //      else
    //        true
    //    }).map(
    //      x=>{
    //        val data = x.split("\t")
    //        val urls = new URL(data(0))
    //        val host = urls.getHost
    //        (data(0), 1)
    //      })

        val rdd1 = line.map(
          x=>{
            val data = x.split("\t")
            val urls = new URL(data(0))
            val host = urls.getHost
            (data(0), 1)
          })

    val rdd2 = rdd1.reduceByKey((x,y)=>x+y)
    //合并两个Map集合对象（将两个对应KEY的值累加）
    //( map1 /: map2 ) { case (map, (k,v)) => map + ( k -> (v + map.getOrElse(k, 0)) ) }
    val rdd3 = rdd2.map{case(d,t)=>{
      val urls = new URL(d)
      val host = urls.getHost
      (host,d, t)
    }}
    //把数据进行分组
    val rdd4 = rdd3.groupBy(_._1)
    //分组后进行排序操作
    val rdd5 = rdd4.map(sx=>{
      val key = sx._1
      val value = sx._2;
      val sorval = value.toList.sortBy(_._3).take(3)
      (key,sorval)
    })

      rdd5.foreach(println)

    //把操作完的数据存入本地文件
    //rdd5.saveAsTextFile("E:\\data2")
    sc.stop()
  }
  }

其中已经表明一些注释了，需要什么就打开什么吧，
让我们看一下结果：

(wengna.taobao.com,List((wengna.taobao.com,https://wengna.taobao.com/?spm=a217m.8316598.682375.7.426d33d5KRgTIh,2)))
(takefired.taobao.com,List((takefired.taobao.com,https://takefired.taobao.com/?spm=a217m.8316598.711275.5.426d33d5KRgTIh,1)))
(blog.csdn.net,List((blog.csdn.net,https://blog.csdn.net/qq_43688472/article/details/84940873,1), (blog.csdn.net,https://blog.csdn.net/qq_24073707/article/details/80988329,1), (blog.csdn.net,https://blog.csdn.net/qq_24073707/article/details/80658301,1)))
(12cmlook.taobao.com,List((12cmlook.taobao.com,https://12cmlook.taobao.com/?spm=a217m.8316598.682375.9.426d33d5KRgTIh,1)))
(item.jd.com,List((item.jd.com,https://item.jd.com/25619900612.html,1), (item.jd.com,https://item.jd.com/19997245287.html,1), (item.jd.com,https://item.jd.com/100001625726.html,1)))
(unawares.taobao.com,List((unawares.taobao.com,https://unawares.taobao.com/?spm=a217m.8316598.682348.7.426d33d5KRgTIh,2)))
(mp.csdn.net,List((mp.csdn.net,https://mp.csdn.net/mdeditor/84307884#,1)))

付出总会有回报，相信自己，即使你很慢，只要你在前进就好额，不要害怕，加油！
——————————————送给努力的你