scala 与 spark 并行化

1. .par普通集合转换为并行集合

scala.collection:

scala> (1 to 5).foreach(println(_))
12345

scala> (1 to 5).par.foreach(println(_))
13254

• 注意既然是并行化处理，即一个集合不同的元素由不同的线程操作，不同线程执行的顺序是无法保证的；

• 获取线程名：

• scala> (0 to 10000).collect { case _ => Thread.currentThread.getName}.distinct
  scala.collection.immutable.IndexedSeq[String] = Vector(main)
  
  scala> (0 to 10000).par.collect { case _ => Thread.currentThread.getName}.distinct
  scala.collection.parallel.immutable.ParSeq[String] = ParVector(ForkJoinPool-1-worker-15, ForkJoinPool-1-worker-19, ForkJoinPool-1-worker-23, ForkJoinPool-1-worker-3, ForkJoinPool-1-worker-17, ForkJoinPool-1-worker-21
  ...