-------------RDD、iterator、partition、collection、element直观认识-----------------------------------
RDD、iterator、partition、collection、element直观认识
RDD的内部元素是iterator的形式,RDD的分区等信息不在iterator中。
一个RDD、一个iterator,多个partition。
---------------------------------------------------------------------------------------------------------------------------
1基本操作:map、filter和flatmap
2、伪集合操作(distinct,union,intersection,substract,cartesian)
实验:
//伪集合操作:返回的都是RDD val rdd2 = sc.parallelize(List(5,6,5,6,7,8,9,10,11,12,13)) //rdd2.distinct()生产一个包含不同元素的RDD rdd2.distinct().collect().foreach(print) //13 11 6 7 8 9 12 10 5 //rdd2.union(rdd1) 联合2个RDD,不去重 rdd2.union(rdd1).collect.foreach(print) //5 6 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 //rdd2.intersection(rdd1) 找出两个共有的元素 rdd2.intersection(rdd1).collect.foreach(print) // 6 7 8 9 5 //rdd2.subtract(rdd1) 返回只有rdd2中有的元素 rdd2.subtract(rdd1).collect.foreach(print) //10 11 12 13 //rdd1和rdd2的笛卡尔积 rdd2.cartesian(rdd1).collect.foreach(print) //(5,1)(5,2)(5,3)(5,4)(5,5)(5,6)(5,7)(5,8)(5,9)(6,1)(6,2)(6,3)(6,4)(6,5)(6,6)(6,7)(6,8)(6,9)(5,1)(5,2)(5,3)(5,4)(5,5)(5,6)(5,7)(5,8)(5,9)(6,1)(6,2)(6,3)(6,4)(6,5)(6,6)(6,7)(6,8)(6,9)(7,1)(7,2)(7,3)(7,4)(7,5)(7,6)(7,7)(7,8)(7,9)(8,1)(8,2)(8,3)(8,4)(8,5)(8,6)(8,7)(8,8)(8,9)(9,1)(9,2)(9,3)(9,4)(9,5)(9,6)(9,7)(9,8)(9,9)(10,1)(10,2)(10,3)(10,4)(10,5)(10,6)(10,7)(10,8)(10,9)(11,1)(11,2)(11,3)(11,4)(11,5)(11,6)(11,7)(11,8)(11,9)(12,1)(12,2)(12,3)(12,4)(12,5)(12,6)(12,7)(12,8)(12,9)(13,1)(13,2)(13,3)(13,4)(13,5)(13,6)(13,7)(13,8)(13,9)16/04/26 21:25:28 INFO SparkContext: Invoking stop() from shutdown hook
3、基本action(count,reduce,fold,aggregate)
val rdd1 = sc.parallelize(List(1,2,3,Array(1,2),2)); println(rdd1.count()) //5
实验: