spark的算子比较多,但只有部分是常用的,特此记录下来,如需补充更多算子,请在评论留言。
常见RDD算子(非Key-Vaue型)
- map
map作用在RDD分区的每一个元素上
scala> val nums=sc.parallelize(List(1,2,3))
nums: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[64] at parallelize at <console>:24
scala> nums.collect()
res43: Array[Int] = Array(1, 2, 3)
scala> nums.map
def map[U](f: Int => U)(implicit evidence$3: scala.reflect.ClassTag[U]): org.apache.spark.rdd.RDD[U]
scala> nums.map(x=>x+1).collect()
res45: Array[Int] = Array(2, 3, 4)
- flatmap
和map类似,但多了flat操作。其内部实现为,先对数据进行map操作,返回List或Seq,然后对返回的List(Seq)进行拍扁操作,形成一个大List。
scala> val ff=sc.parallelize(Array((1,2),(3,4),(5,6)))
ff: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[71] at parallelize at <console>:24
scala> ff.flatMap(x=>Seq(x._1)).collect()
res67: Array[Int] = Array(1, 3, 5)
- mapPartitions
操作对象为整个分区。通过获取分区的迭代器,在函数中通过这个分区整体的迭代器进行操作。
scala> val a = sc.parallelize(1 to 10, 3)
a: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[99] at parallelize at <console>:24
scala> a.collect()
res84: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
scala> def myfuncPerElement(e:Int):Int = {
|
| println("e="+e)
|
| e*2
|
| }
myfuncPerElement: (e: Int)Int
scala> def myfuncPerPartition ( iter : Iterator [Int] ) : Iterator [Int] = {
|
| pri