RDD基本转化操作：filter、union、intersection、subtract、map

1、filter
简单说就是过滤操作，得到自己想要的元素。

 val inputRDD = sc.textFile("README.md")
inputRDD: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[1] at textFile at <console>:24

scala> val pythonRDD = inputRDD.filter(line => line.contains("Python"))
pythonRDD: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at filter at <console>:26

scala> pythonRDD.first
res2: String = high-level APIs in Scala, Java, Python, and R, and an optimized engine that

得到文本里面包含“Python”的字符串。

2、union
def union(other: RDD[T]): RDD[T]
该函数比较简单，就是将两个RDD进行合并，不去重。
来看下面例子：

scala> val item1 = sc.parallelize(1 to 3)
item1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala> val item2 = sc.parallelize(2 to 4)
item2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at <console>:24

scala> item1.union(item2).collect
res0: Array[Int] = Array(1, 2, 3, 2, 3, 4)

intersection

def intersection(other: RDD[T]): RDD[T]
def intersection(other: RDD[T], numPartitions: Int): RDD[T]
def intersection(other: RDD[T], partitioner: Partitioner)(implicit ord: Ordering[T] = null): RDD[T]

该函数返回两个RDD的交集，并且去重。参数numPartitions指定返回的RDD的分区数。参数partitioner用于指定分区函数

scala> val rdd1 = sc.makeRDD(1 to 2,1)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[3] at makeRDD at <console>:24

scala> rdd1.collect
res1: Array[Int] = Array(1, 2)

scala> val rdd2 = sc.makeRDD(2 to 3 ,1)
rdd2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[4] at makeRDD at <console>:24

scala> rdd2.collect
res2: Array[Int] = Array(2, 3)

scala> rdd1.intersection(rdd2).collect
res3: Array[Int] = Array(2)

scala> var rdd3 = rdd1.intersection(rdd2)
rdd3: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[16] at intersection at <console>:28

scala> rdd3.partitions.size
res4: Int = 1

scala> var rdd3 = rdd1.intersection(rdd2,2)
rdd3: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[22] at intersection at <console>:28

scala> rdd3.partitions.size
res5: Int = 2

subtract:
从单词意思可以看出，就是要求两个rdd中不同的元素，不去重

def subtract(other: RDD[T]): RDD[T]
def subtract(other: RDD[T], numPartitions: Int): RDD[T]
def subtract(other: RDD[T], partitioner: Partitioner)(implicit ord: Ordering[T] = null): RDD[T]

参数和intersection是一样的。

scala> var rdd4 = sc.makeRDD(Seq(1,2,2,3))
rdd4: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[24] at makeRDD at <console>:24

scala> rdd4.collect
res6: Array[Int] = Array(1, 2, 2, 3)

scala> var rdd5 = sc.makeRDD(3 to 4)
rdd5: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[25] at makeRDD at <console>:24

scala> rdd5.collect
res7: Array[Int] = Array(3, 4)

scala> rdd4.subtract(rdd5).collect
collect   collectAsync

scala> rdd4.subtract(rdd5).collect
res8: Array[Int] = Array(1, 2, 2)

注意：只有当你的整个数据能在但台机器的内存中放的下时，才能使用collect（），因此，collect（）不能用在大规模数据集上。
map()
接收一个函数，并且把这个函数应用到RDD中的每个元素中。

scala> rdd1.collect
res21: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

scala> val result = rdd1.map(x=>x*x)
result: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[35] at map at <console>:26

scala> result.collect
res25: Array[Int] = Array(1, 4, 9, 16, 25, 36, 49, 64, 81, 100)

flatMap()
和map类似，也是将RDD中的每个元素应用到flatMap中的函数中，不过返回的不是一个元素而是一个返回值序列的迭代器。一个比较简单的用途是可以讲字符串切分成单词。

scala> val lines = sc.parallelize(List("Hello world","hi")) 
lines: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala> val words = lines.flatMap(line => line.split(" "))
words: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at flatMap at <console>:26

scala> words.first
res0: String = Hello