SparkCore

SparkCore总结

1. RDD

1.1定义：

··········1.1.1数据集：存储数据的计算逻辑

··········1.1.2分布式：数据的来源&计算都是分布式的

··········1.1.3弹性：

································血缘（依赖关系）：Spark可以通过特殊的处理方案简化依赖关系
································计算：Spark的计算是基于内存的，所以性能很高，可以和磁盘灵活切换
································分区：Spark在创建默认分区后，可以通过指定的算子来改变分区数量
································容错：Spark在执行计算时，如果发生了错误，需要进行容错重试处理

··········1.1.4Spark中分区的数量：

································Executor:可以通过提交应用的参数进行设定
································partition:默认情况下，读取文件采用的是hadoop的切片规则，如果读取内存中的数据,可以根据特点的算子进行设定。可以通过其他的算子进行改变。多个阶段的场合，下个阶段的分区数量取决于上个阶段最后的分区数量，但是可以在相应的算子中进行修改
································Stage:1（resultStage）+shuffle依赖的数量（shuffleMapStage）划分任务的目的就是为了任务执行的等待，因为Shuffle的过程需要落盘
································Task:原则上一个分区就是一个任务但是实际应用中，可以动态调整

1.2创建：

··········1.2.1:从内存中创键

··········1.2.2:从存储中创键

··········1.2.3:从其他RDD中创键

1.3属性

··········1.3.1分区

··········1.3.2依赖关系

··········1.3.3分区器

··········1.3.4优先位置

··········1.3.5计算函数

1.4使用

··········1.4.1转换

································单value类型
································双value类型
································K-V类型

··········1.4.2行动

································runJob

2. 广播变量：分布式共享只读数据

3.累加器：分布式共享只写数据