Spark PruneDependency 依赖关系 Filter
Represents a dependency between the PartitionPruningRDD and its parent. In this case, the child RDD contains a subset of partitions of the parents’.
更多资源
youtub视频演示
width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=65822402&page=3" scrolling="no" border="0" allowfullscreen="true">
输入数据
List(( "a" ,2) ,( "d" ,1) ,( "b" ,8) ,( "d" ,3)
处理程序scala
package com.opensource.bigdata.spark.local.rdd.operation.dependency.narrow.n_03_pruneDependency.n_03_filterByRange_filter
import com.opensource.bigdata.spark.local.rdd.operation.base.BaseScalaSparkContext
object Run extends BaseScalaSparkContext{
def main(args: Array[String]): Unit = {
val sc = pre()
val rdd1 = sc.parallelize(List(("a",2),("d",1),("b",8),("d",3)),2) //ParallelCollectionRDD
val rdd2 =rdd1.filterByRange("a","b") //MapParttionsRDD
println("rdd \n" + rdd2.collect().mkString("\n"))
sc.stop()
}
}
数据处理图