RDDs基本操作之Transformations（一）

一 Transformations

Transformations（转换）：从之前的RDD构建一个新的RDD，像map()和filter()

二 逐元素Transformations

1 map函数

map():map()接收函数，把函数应用到RDD的每一个元素，返回新RDD。

scala> val lines = sc.parallelize(Array("hello","spark","hello","world","!"))
lines: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[14] at parallelize at <console>:24
scala> lines.collect()
res8: Array[String] = Array(hello, spark, hello, world, !)
scala> val lines2=lines.map(word=>(word,1))
lines2: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[15] at map at <console>:26
scala> lines2.collect()
res9: Array[(String, Int)] = Array((hello,1), (spark,1), (hello,1), (world,1), (!,1))

2 filter函数

filter()接收函数，返回只包含满足filter()函数的元素的新RDD

scala> val lines3=lines.filter(word=>word.contains("hello"))
lines3: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[16] at filter at <console>:26
scala> lines3.collect()
res10: Array[String] = Array(hello, hello)

3 flatMap函数

对每个输入元素，输出多个输出元素。

flat压扁的意思，将RDD中元素压扁后返回一个新的RDD

scala> val inputs = sc.textFile("/root/helloSpark.txt")
inputs: org.apache.spark.rdd.RDD[String] = /root/helloSpark.txt MapPartitionsRDD[1] at textFile at <console>:24
scala> inputs.collect()
res0: Array[String] = Array(go to home hello java, so many to hello word kafka java, go to so)
scala> val lines=inputs.flatMap(line=>line.split(" "))
lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at flatMap at <console>:26
scala> lines.collect()
res1: Array[String] = Array(go, to, home, hello, java, so, many, to, hello, word, kafka, java, go, to, so)