Spark处理数据如何获得行号

Spark处理数据如何获得行号

因为Spark并行的处理数据，所以你不能在自己的driver program中计数到底是处理到第几个。Spark提供了zipWithIndex可以给你提供索引号。这个索引号是全局有序和唯一的。

public RDD<scala.Tuple2<T,Object>> zipWithIndex()
Zips this RDD with its element indices. The ordering is first based on the partition index and then the ordering of items within each partition. So the first item in the first partition gets index 0, and the last item in the last partition receives the largest index.
This is similar to Scala's zipWithIndex but it uses Long instead of Int as the index type. This method needs to trigger a spark job when this RDD contains more than one partitions.


Note that some RDDs, such as those returned by groupBy(), do not guarantee order of elements in a partition. The index assigned to each element is therefore not guaranteed, and may even change if the RDD is reevaluated. If a fixed ordering is required to guarantee the same index assignments, you should sort the RDD with sortByKey() or save it to a file.

另一个可以让你得到唯一标示符的函数是zipWithUniqueId,跟zipWithIndex不同的是，zipWithUniqueId只保证唯一，不保证连续性，中间可能有gap.