Spark常用函数讲解

1.mapValus

2.flatMapValues

3.comineByKey

4.foldByKey

5.reduceByKey

6.groupByKey

7.sortByKey

8.cogroup

9.join

10.LeftOutJoin

11.RightOutJoin

 

1.mapValus(fun):对[K,V]型数据中的V值map操作

object MapValues {

  def main(args: Array[String]) {

    val conf = new SparkConf().setMaster( "local" ).setAppName( "map" )

    val sc = new SparkContext(conf)

    val list = List(( "mobin" , 22 ),( "kpop" , 20 ),( "lufei" , 23 ))

    val rdd = sc.parallelize(list)

    val mapValuesRDD = rdd.mapValues(_+ 2 )

    mapValuesRDD.foreach(println)

  }

}

输出： (mobin, 24) (kpop,22) (lufei,25)

 

2. flatMapValues(fun)：对[K,V]型数据中的V值flatmap操作

//省略<br>val list = List(("mobin",22),("kpop",20),("lufei",23))

val rdd = sc.parallelize(list)

val mapValuesRDD = rdd.flatMapValues(x => Seq(x, "male" ))

mapValuesRDD.foreach(println)

 

(mobin, 22)
(mobin,male)
(kpop,20)
(kpop,male)
(lufei,23)
(lufei,male)

如果是mapValues会输出：

(mobin,List(22, male))
(kpop,List(20, male)) (lufei,List(23, male))

 

3. comineByKey(createCombiner,mergeValue,mergeCombiners,partitioner,mapSideCombine)

createCombiner:在第一次遇到Key时创建组合器函数，将RDD数据集中的V类型值转换C类型值（V => C），

mergeValue：合并值函数，再次遇到相同的Key时，将createCombiner道理的C类型值与这次传入的V类型值合并成一个C类型值（C,V）=>C，

mergeCombiners:合并组合器函数，将C类型值两两合并成一个C类型值

 

object CombineByKey {

  def main(args: Array[String]) {

    val conf = new SparkConf().setMaster( "local" ).setAppName( "combinByKey" )

    val sc = new SparkContext(conf)

    val people = List(( "male" , "Mobin" ), ( "male" , "Kpop" ), ( "female" , "Lucy" ), ( "male" , "Lufei" ), ( "female" , "Amy" ))

    val rdd = sc.parallelize(people)

    val combinByKeyRDD = rdd.combineByKey(

      (x: String) => (List(x), 1 ),

      (peo: (List[String], Int), x : String) => (x :: peo._1, peo._2 + 1 ),

      (sex1: (List[String], Int), sex2: (List[String], Int)) => (sex1._1 ::: sex2._1, sex1._2 + sex2._2))

    combinByKeyRDD.foreach(println)

    sc.stop()

  }

}

(male,(List(Lufei, Kpop, Mobin), 3))
(female,(List(Amy, Lucy),2))

 

4.foldByKey(zeroValue)(func)

  foldByKey函数是通过调用CombineByKey函数实现的

func: Value将通过func函数按Key值进行合并（实际上是通过CombineByKey的mergeValue，mergeCombiners函数实现的，只不过在这里，这两个函数是相同的）

//省略

    val people = List(( "Mobin" , 2 ), ( "Mobin" , 1 ), ( "Lucy" , 2 ), ( "Amy" , 1 ), ( "Lucy" , 3 ))

    val rdd = sc.parallelize(people)

    val foldByKeyRDD = rdd.foldByKey( 2 )(_+_)

    foldByKeyRDD.foreach(println)

 

5.reduceByKey(func,numPartitions):按Key进行分组，使用给定的func函数聚合value值, numPartitions设置分区数，提高作业并行度

6.groupByKey(numPartitions):按Key进行分组，返回[K,Iterable[V]]，numPartitions设置分区数，提高作业并行度

 

7.sortByKey(accending，numPartitions):返回以Key排序的（K,V）键值对组成的RDD，accending为true时表示升序，为false时表示降序，numPartitions设置分区数，提高作业并行度

8.cogroup(otherDataSet，numPartitions)：对两个RDD(如:(K,V)和(K,W))相同Key的元素先分别做聚合，最后返回(K,Iterator<V>,Iterator<W>)形式的RDD,numPartitions设置分区数，提高作业并行度

val arr = List(( "A" , 1 ), ( "B" , 2 ), ( "A" , 2 ), ( "B" , 3 ))

val arr1 = List(( "A" , "A1" ), ( "B" , "B1" ), ( "A" , "A2" ), ( "B" , "B2" ))

val rdd1 = sc.parallelize(arr, 3 )

val rdd2 = sc.parallelize(arr1, 3 )

val groupByKeyRDD = rdd1.cogroup(rdd2)

groupByKeyRDD.foreach(println)

sc.stop

(B,(CompactBuffer( 2, 3),CompactBuffer(B1, B2)))
(A,(CompactBuffer(1, 2),CompactBuffer(A1, A2)))

 

9. join(otherDataSet,numPartitions):对两个RDD先进行cogroup操作形成新的RDD，再对每个Key下的元素进行笛卡尔积，numPartitions设置分区数，提高作业并行度

  10.LeftOutJoin(otherDataSet，numPartitions):左外连接，包含左RDD的所有数据，如果右边没有与之匹配的用None表示,numPartitions设置分区数，提高作业并行度

 

11.RightOutJoin(otherDataSet, numPartitions):右外连接，包含右RDD的所有数据，如果左边没有与之匹配的用None表示,numPartitions设置分区数，提高作业并行度

 

 

转载于:https://www.cnblogs.com/energy1010/p/6992860.html