combineByKey之Spark中一个比较核心高级函数

最新推荐文章于 2022-02-09 12:25:56 发布

搬砖快乐～

最新推荐文章于 2022-02-09 12:25:56 发布

阅读量209

点赞数 1

分类专栏： scala spark

本文链接：https://blog.csdn.net/demon_LL/article/details/84336052

版权

spark 同时被 2 个专栏收录

8 篇文章 0 订阅

订阅专栏

scala

6 篇文章 0 订阅

订阅专栏

import org.apache.spark.{HashPartitioner, Partitioner, SparkConf, SparkContext}


/**
  * Created by Administrator on 2018/7/3.
  */
object test {


   def main(args: Array[String]): Unit = {
      val conf = new SparkConf().setAppName("test").setMaster("local[12]")
      val sc = new SparkContext(conf)
      val book=Array(("大冰","阿弥陀佛么么哒"),("大冰","我不"),("柴静","看见"),("知乎","知乎"),
("胡赛尼","追风筝的人"),("大冰","你坏"),("钱钟书","围城"),("钱钟书","猫"),("大冰","好吗好的"))
      sc.makeRDD(book).combineByKey(
         v => (1,v),
         (acc:(Int,String),v:String) => (acc._1 + 1,v),
         (acc:(Int,String),acc1:(Int,String)) => (acc._1+acc1._1 , acc1._2),
         new HashPartitioner(1),
      false
      ).map {
         case (k, (k1, v)) => Array(k, k1, v).mkString("\t")
      }.foreach(println)
      sc.stop()

   }




}

理解：https://blog.csdn.net/Gpwner/article/details/73349589