Spark2.3 RDD之flatMap源码解析

最新推荐文章于 2022-02-20 13:52:12 发布

DPnice

最新推荐文章于 2022-02-20 13:52:12 发布

阅读量2.7k

点赞数

分类专栏： spark 文章标签： spark scala flatMap

本文链接：https://blog.csdn.net/DPnice/article/details/80093370

版权

本文深入解析Spark2.3中RDD的flatMap源码，阐述flatMap如何将RDD内的每个元素通过自定义函数转化为iterator，进而组合成一个新的RDD。flatMap实现过程涉及多次回调，直至RDD所有元素处理完毕。

摘要由CSDN通过智能技术生成

Spark flatMap 源码：

  /**
   *  Return a new RDD by first applying a function to all elements of this
   *  RDD, and then flattening the results.
   */
  def flatMap[U: ClassTag](f: T => TraversableOnce[U]): RDD[U] = withScope {
    val cleanF = sc.clean(f)
    new MapPartitionsRDD[U, T](this, (context, pid, iter) => iter.flatMap(cleanF))
  }

Scala flatMap 源码：

  /** Creates a new iterator by applying a function to all values produced by this iterator
   *  and concatenating the results.
   *
   *  @param f the function to apply on each element.
   *  @return  the iterator resulting from applying the given iterator-valued function
   *           `f` to each value produced by this iterator and concatenating the results.
   *  @note    Reuse: $consumesAndProducesIterator
   */
  def flatMap[B](f: A =&g