举例说明Spark RDD的分区、依赖

最新推荐文章于 2022-09-20 13:18:44 发布

bigbigdata

最新推荐文章于 2022-09-20 13:18:44 发布

阅读量1k

点赞数

分类专栏： spark 文章标签： spark RDD

本文链接：https://blog.csdn.net/bigbigdata/article/details/47292119

版权

本文通过实例展示了Spark中RDD的依赖关系，包括NarrowDependency和ShuffleDependency。每个RDD的dependencies属性显示其直接依赖的RDD，dependency类型揭示依赖性质。通过dependency的compute函数，可以了解如何从父RDD计算出子RDD的分区。当RDD分区计算失败时，系统会回溯到最近的已缓存RDD或数据源。

摘要由CSDN通过智能技术生成

例子如下:

scala> val textFileRDD = sc.textFile("/Users/zhuweibin/Downloads/hive_04053f79f32b414a9cf5ab0d4a3c9daf.txt")
15/08/03 07:00:08 INFO MemoryStore: ensureFreeSpace(57160) called with curMem=0, maxMem=278019440
15/08/03 07:00:08 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 55.8 KB, free 265.1 MB)
15/08/03 07:00:08 INFO MemoryStore: ensureFreeSpace(17237) called with curMem=57160, maxMem=278019440
15/08/03 07:00:08 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 16.8 KB, free 265.1 MB)
15/08/03 07:00:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:51675 (size: 16.8 KB, free: 265.1 MB)
15/08/03 07:00:08 INFO SparkContext: Created broadcast 0 from textFile at <console>:21
textFileRDD: org.apache.spark.rdd.RDD[