1 RDD 算子
1.1 transformation
2.2 action
1.1 transformation
1.1.1 value transformation
map flatMap filter disinct mapPartition mapPartitionsWithIndex
union intersection substract coalesce repartition
randomSplit glom zip zipPartitions zipWithIndex zipWithUniqueId sortBy
repartitionAndSortWithinPartitions(partitioner)//再分区并在分区内进行排序
1.1.2 key-value transformation
reduceByKey(shuffle之前有combine) groupByKey combaineByKey
aggregateByKey foldByKey sortByKey partitionBy join cogroup mapValues flatMapValues
1.2 action
action 会触发Spark生成Job,Action不会生成新的RDD,而是将RDD封装在Scala实例中,或直接输出到外部系统
1.2.1 value action
collect first take top takeOrdered reduce aggregate fold foreach foreachPartition count
1.2.2 key-value action
lookup countBykey
1.2.3 数值型 action
stats sum max min mean
variance方差 sampleVariance抽样方差
stdev标准差 sampleStdev抽样标准差
1.2.4 输出操作
saveAsTextFile
...
2 RDD高级
2.1 RDD的缓存及持久化:cache/persist
2.2 RDD的检查点(Checkpoint)
2.3 RDD的依赖关系(dependencies):宽、窄依赖(wide/shuffle、narrow)
2.4 广播变量(BroadCast)
2.5 累加器(Accumulator)
2.1 RDD的缓存及持久化
cache()
persist()
2.2 RDD的检查点 Checkpoint
Checkpoint(没有上游依赖源)
2.3 RDD 的宽窄依赖
宽依赖 wide/shuffle
窄依赖 narrow
2.1 广播变量 BroadCast
BroadCast 将数据块缓存到所有节点
value: Broadcast.value()
2.3 累加器 Accumulator
Accumulator
【参考】
1. Spark大数据分析 源码解析与实战详解 刘景泽 编著
2. Spark实时大数据分析 基于Spark Streaming框架 祖贝尔•纳比 著 王胜夏 景玉 译
3. Spark数据分析 基于Python语言 Jeffrey Aven 著 王道远 译