Spark修炼之道（进阶篇）——Spark入门到精通：第四节 Spark编程模型（一)

本节主要内容

1. Spark重要概念
2. 弹性分布式数据集（RDD）基础

1. Spark重要概念

本节部分内容源自官方文档：http://spark.apache.org/docs/latest/cluster-overview.html

（1）Spark运行模式

目前最为常用的Spark运行模式有： 
- local：本地线程方式运行，主要用于开发调试Spark应用程序 
- Standalone：利用Spark自带的资源管理与调度器运行Spark集群，采用Master/Slave结构，为解决单点故障，可以采用ZooKeeper实现高可靠（High Availability，HA) 
- Apache Mesos ：运行在著名的Mesos资源管理框架基础之上，该集群运行模式将资源管理交给Mesos，Spark只负责进行任务调度和计算 
- Hadoop YARN : 集群运行在Yarn资源管理器上，资源管理交给Yarn，Spark只负责进行任务调度和计算 
Spark运行模式中Hadoop YARN的集群运行方式最为常用，本课程中的第一节便是采用Hadoop YARN的方式进行Spark集群搭建。如此Spark便与Hadoop生态圈完美搭配，组成强大的集群，可谓无所不能。

（2）Spark组件（Components）

一个完整的Spark应用程序，如前一节当中SparkWordCount程序，在提交集群运行时，它涉及到如下图所示的组件： 

各Spark应用程序以相互独立的进程集合运行于集群之上，由SparkContext对象进行协调，SparkContext对象可以视为Spark应用程序的入口，被称为driver program，SparkContext可以与不同种类的集群资源管理器(Cluster Manager），例如Hadoop Yarn、Mesos等 进行通信，从而分配到程序运行所需的资源，获取到集群运行所需的资源后，SparkContext将得到集群中其它工作节点（Worker Node） 上对应的Executors （不同的Spark应用程序有不同的Executor，它们之间也是独立的进程，Executor为应用程序提供分布式计算及数据存储功能），之后SparkContext将应用程序代码分发到各Executors，最后将任务（Task）分配给executors执行。

Term（术语)	Meaning（解释)
Application（Spark应用程序）	运行于Spark上的用户程序，由集群上的一个driver program（包含SparkContext对象）和多个executor线程组成
Application jar（Spark应用程序JAR包）	Jar包中包含了用户Spark应用程序，如果Jar包要提交到集群中运行，不需要将其它的Spark依赖包打包进行，在运行时
Driver program	包含main方法的程序，负责创建SparkContext对象
Cluster manager	集群资源管理器，例如Mesos，Hadoop Yarn
Deploy mode	部署模式，用于区别driver program的运行方式:集群模式(cluter mode)，driver在集群内部启动；客户端模式（client mode），driver进程从集群外部启动
Worker node	工作节点， 集群中可以运行Spark应用程序的节点
Executor	Worker node上的进程，该进程用于执行具体的Spark应用程序任务，负责任务间的数据维护（数据在内存中或磁盘上)。不同的Spark应用程序有不同的Executor
Task	运行于Executor中的任务单元，Spark应用程序最终被划分为经过优化后的多个任务的集合（在下一节中将详细阐述）
Job	由多个任务构建的并行计算任务，具体为Spark中的action操作，如collect,save等)
Stage	每个job将被拆分为更小的task集合，这些任务集合被称为stage，各stage相互独立（类似于MapReduce中的map stage和reduce stage），由于它由多个task集合构成，因此也称为TaskSet

2. 弹性分布式数据集（RDD）基础

弹性分布式数据集（RDD，Resilient Distributed Datasets），由Berkeley实验室于2011年提出，原始论文名字：Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing 原始论文非常值得一读，是研究RDD的一手资料，本节内容大部分将基于该论文。

（1）RDD设计目标

RDD用于支持在并行计算时能够高效地利用中间结果，支持更简单的编程模型，同时也具有像MapReduce等并行计算框架的高容错性、能够高效地进行调度及可扩展性。RDD的容错通过记录RDD转换操作的lineage关系来进行，lineage记录了RDD的家族关系，当出现错误的时候，直接通过lineage进行恢复。RDD最合数据挖掘, 机器学习及图计算，因此这些应用涉及到大家的迭代计算，基于内存能够极大地提升其在分布式环境下的执行效率；RDD不适用于诸如分布式爬虫等需要频繁更新共享状态的任务。

下面给出的是在spark-shell中如何查看RDD的Lineage

<code class="hljs livecodeserver has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">//textFile读取hdfs根目录下的README.md文件，然后筛选出所有包括Spark的行
scala> val rdd2=sc.textFile(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/README.md"</span>).<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">filter</span>(<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">line</span> => <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">line</span>.<span class="hljs-operator" style="box-sizing: border-box;">contains</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Spark"</span>))
rdd2: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>] <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">at</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">filter</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">at</span> <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">21</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">
//toDebugString方法会打印出RDD的家族关系</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">
//可以看到textFile方法会生成两个RDD，分别是HadoopRDD</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">
//MapPartitionsRDD，而filter同时也会生成新的MapPartitionsRDD</span>
scala> rdd2.toDebugString
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">35</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">27</span> INFO mapred.FileInputFormat: Total input paths <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">to</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">process</span> : <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
res0: String = 
(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>) MapPartitionsRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>] <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">at</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">filter</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">at</span> <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">21</span> []
 |  MapPartitionsRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>] <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">at</span> textFile <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">at</span> <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">21</span> []
 |  /README.md HadoopRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>] <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">at</span> textFile <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">at</span> <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">21</span> []</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li></ul>

（2）RDD抽象

RDD在Spark中是一个只读的（val类型）、经过分区的记录集合。RDD在Spark中只有两种创建方式：（1）从存储系统中创建；（2）从其它RDD中创建。从存储中创建有多种方式，可以是本地文件系统，也可以是分布式文件系统，还可以是内存中的数据。 
下面的代码演示的是从HDFS中创建RDD

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scala> sc<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.textFile</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/README.md"</span>)
<span class="hljs-label" style="box-sizing: border-box;">res1:</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rdd</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RDD</span>[String] = MapPartitionsRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>] at textFile at <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">22</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

下面的代码演示的是从内存中创建RDD

<code class="hljs haskell has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">//内存中定义了一个数组
<span class="hljs-title" style="box-sizing: border-box;">scala</span>> val <span class="hljs-typedef" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">data</span> = <span class="hljs-type" style="box-sizing: border-box; color: rgb(102, 0, 102);">Array</span><span class="hljs-container" style="box-sizing: border-box;">(1, 2, 3, 4, 5)</span></span>
<span class="hljs-typedef" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">data</span>: <span class="hljs-type" style="box-sizing: border-box; color: rgb(102, 0, 102);">Array</span>[<span class="hljs-type" style="box-sizing: border-box; color: rgb(102, 0, 102);">Int</span>] = <span class="hljs-type" style="box-sizing: border-box; color: rgb(102, 0, 102);">Array</span><span class="hljs-container" style="box-sizing: border-box;">(1, 2, 3, 4, 5)</span></span>
//通过parallelize方法创建<span class="hljs-type" style="box-sizing: border-box; color: rgb(102, 0, 102);">ParallelCollectionRDD</span>
<span class="hljs-title" style="box-sizing: border-box;">scala</span>> val distData = sc.parallelize(<span class="hljs-typedef" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">data</span>)</span>
<span class="hljs-title" style="box-sizing: border-box;">distData</span>: org.apache.spark.rdd.<span class="hljs-type" style="box-sizing: border-box; color: rgb(102, 0, 102);">RDD</span>[<span class="hljs-type" style="box-sizing: border-box; color: rgb(102, 0, 102);">Int</span>] = <span class="hljs-type" style="box-sizing: border-box; color: rgb(102, 0, 102);">ParallelCollectionRDD</span>[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>] at parallelize at <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">23</span>
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul>

下面的代码演示的是从其它RDD创建新的RDD

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">//filter函数将distData RDD转换成新的RDD
scala> val distDataFiletered=distData<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.filter</span>(e=>e><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)
<span class="hljs-label" style="box-sizing: border-box;">distDataFiletered:</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rdd</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RDD</span>[Int] = MapPartitionsRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>] at filter at <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">25</span>
//触发action操作（后面我们会讲)，查看过滤后的内容
//注意collect只适合数据量较少时使用
scala> distDataFiltered<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collect</span>
<span class="hljs-label" style="box-sizing: border-box;">res3:</span> Array[Int] = Array(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>)
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul>

（3）RDD编程模型

在前面的例子中，我们已经接触过到如何利用RDD进行编程，前面我们提到的

<code class="hljs fsharp has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//filter函数将distData RDD转换成新的RDD</span>
scala> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> distDataFiletered=distData.filter(e=>e><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//触发action操作（后面我们会讲)，查看过滤后的内容</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//注意collect只适合数据量较少时使用</span>
scala> distDataFiltered.collect</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

这段代码它已经给我们解释了RDD编程模型的核心思想：“filter函数将distData RDD转换成新的RDD”，“触发action操作”。也就是说RDD的操作包括Transformations（转换)、Actions两种。

transformations操作会将一个RDD转换成一个新的RDD，需要特别注意的是所有的transformation都是lazy的，如果对scala中的lazy了解的人都知道，transformation之后它不会立马执行，而只是会记住对相应数据集的transformation，而到真正被使用的时候才会执行，例如distData.filter(e=>e>2) transformation后，它不会立即执行，而是等到distDataFiltered.collect方法执行时才被执行，如下图所示 
 
从上图可以看到，在distDataFiltered.collect方法执行之后，才会触发最终的transformation执行。

从transformation的介绍中我们知道，action是解决程序最终执行的诱因，action操作会返回程序执行结果如collect操作或将运行结果保存，例如SparkWordCount中的saveAsTextFile方法。

Spark 1.5.0支持的transformation包括：

（1）map 
map函数方法参数：

<code class="hljs markdown has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/**
   * Return a new RDD by applying a function to all elements of this RDD.
   */
  def map[<span class="hljs-link_label" style="box-sizing: border-box;">U: ClassTag</span>](<span class="hljs-link_url" style="box-sizing: border-box;">f: T => U</span>): RDD[U]</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

//使用示例

<code class="hljs php has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scala> val rdd1=sc.parallelize(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">Array</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>)).map(x=><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>*x).collect
rdd1: <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">Array</span>[Int] = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">Array</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

（2）filter 
方法参数：

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/**
   * Return a new RDD containing only the elements that satisfy a predicate.
   */
  <span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">filter</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(f: T => Boolean)</span>:</span> RDD[T]</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

使用示例

<code class="hljs vbscript has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scala> val rdd1=sc.parallelize(<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">Array</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>)).<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">filter</span>(x=>x><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>).collect
rdd1: <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">Array</span>[<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">Int</span>] = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">Array</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>)
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

（3）flatMap 
方法参数：

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/**
   *  Return a new RDD by first applying a function to all elements of this
   *  RDD, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> then flattening the results.
   */
  <span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">flatMap</span>[<span class="hljs-title" style="box-sizing: border-box;">U</span>:</span> ClassTag](f: T => TraversableOnce[U]): RDD[U] </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

使用示例：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scala>  val data =Array(Array(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>),Array(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>))
<span class="hljs-label" style="box-sizing: border-box;">data:</span> Array[Array[Int]] = Array(Array(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>), Array(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>))

scala> val rdd1=sc<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.parallelize</span>(data)
<span class="hljs-label" style="box-sizing: border-box;">rdd1:</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rdd</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RDD</span>[Array[Int]] = ParallelCollectionRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>] at parallelize at <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">23</span>

scala> val rdd2=rdd1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.flatMap</span>(<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">x</span>=><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">x</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.map</span>(<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">y</span>=><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">y</span>))
<span class="hljs-label" style="box-sizing: border-box;">rdd2:</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rdd</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RDD</span>[Int] = MapPartitionsRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>] at flatMap at <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">25</span>

scala> rdd2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collect</span>
<span class="hljs-label" style="box-sizing: border-box;">res0:</span> Array[Int] = Array(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>)


</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul>

（4）mapPartitions(func) 
本mapPartitions例子来源于：https://www.zybuluo.com/jewes/note/35032 
mapPartitions是map的一个变种。map的输入函数是应用于RDD中每个元素，而mapPartitions的输入函数是应用于每个分区，也就是把每个分区中的内容作为整体来处理的。它的函数定义为：

def mapPartitions[U: ClassTag](f: Iterator[T] => Iterator[U], preservesPartitioning: Boolean = false): RDD[U]

f即为输入函数，它处理每个分区里面的内容。每个分区中的内容将以Iterator[T]传递给输入函数f，f的输出结果是Iterator[U]。最终的RDD由所有分区经过输入函数处理后的结果合并起来的。

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scala> val a = sc.parallelize(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> to <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)
scala> <span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">myfunc</span>[<span class="hljs-title" style="box-sizing: border-box;">T</span>]<span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(iter: Iterator[T])</span> :</span> Iterator[(T, T)] = {
    var res = List[(T, T)]() 
    var pre = iter.next 
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">while</span> (iter.hasNext) {
        val cur = iter.next; 
        res .::= (pre, cur)
        pre = cur;
    } 
    res.iterator
}
scala> a.mapPartitions(myfunc).collect
res0: Array[(Int, Int)] = Array((<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>))</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li></ul>

上述例子中的函数myfunc是把分区中一个元素和它的下一个元素组成一个Tuple。因为分区中最后一个元素没有下一个元素了，所以(3,4)和(6,7)不在结果中。 
mapPartitions还有些变种，比如mapPartitionsWithContext，它能把处理过程中的一些状态信息传递给用户指定的输入函数。还有mapPartitionsWithIndex，它能把分区的index传递给用户指定的输入函数。

（5）mapPartitionsWithIndex

mapPartitionsWithIndex函数是mapPartitions函数的一个变种，它的函数参数如下：

def mapPartitionsWithIndex[U: ClassTag]( 
f: (Int, Iterator[T]) => Iterator[U], 
preservesPartitioning: Boolean = false): RDD[U]

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">```
scala> val a = sc.parallelize(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> to <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)
//函数带分区索引，返回的集合第一个元素为分区索引
scala> <span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">myfunc</span>[<span class="hljs-title" style="box-sizing: border-box;">T</span>]<span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(index:T,iter: Iterator[T])</span> :</span> Iterator[(T,T,T)] = {
    var res = List[(T,T, T)]() 
    var pre = iter.next 
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">while</span> (iter.hasNext) {
        val cur = iter.next
        res .::= (index,pre, cur) 
        pre = cur
    } 
    res.iterator
}
scala> a.mapPartitionsWithIndex(myfunc).collect
res11: Array[(Int, Int, Int)] = Array((<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>), (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>))
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul>

（6）sample 
方法参数：

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"> /**
   * Return a sampled subset of this RDD.
   *
   * <span class="hljs-decorator" style="color: rgb(0, 102, 102); box-sizing: border-box;">@param withReplacement can elements be sampled multiple times (replaced when sampled out)</span>
   * <span class="hljs-decorator" style="color: rgb(0, 102, 102); box-sizing: border-box;">@param fraction expected size of the sample as a fraction of this RDD's size</span>
   *  without replacement: probability that each element <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">is</span> chosen; fraction must be [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>]
   *  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span> replacement: expected number of times each element <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">is</span> chosen; fraction must be >= <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>
   * <span class="hljs-decorator" style="color: rgb(0, 102, 102); box-sizing: border-box;">@param seed seed for the random number generator</span>
   */
  <span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">sample</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(
      withReplacement: Boolean,
      fraction: Double,
      seed: Long = Utils.random.nextLong)</span>:</span> RDD[T] </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li></ul>

使用示例：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scala> val a = sc<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.parallelize</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> to <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)
<span class="hljs-label" style="box-sizing: border-box;">a:</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rdd</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RDD</span>[Int] = ParallelCollectionRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">12</span>] at parallelize at <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">21</span>

scala> val smapledA=a<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sample</span>(true,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.5</span>)
<span class="hljs-label" style="box-sizing: border-box;">smapledA:</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rdd</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RDD</span>[Int] = PartitionwiseSampledRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">13</span>] at sample at <console>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">23</span>
scala> smapledA<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collect</span>
<span class="hljs-label" style="box-sizing: border-box;">res12:</span> Array[Int] = Array(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>)

scala> val smapledA2=a<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sample</span>(false,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.5</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collect</span>
<span class="hljs-label" style="box-sizing: border-box;">smapledA2:</span> Array[Int] = Array(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li></ul>

转载：http://blog.csdn.net/lovehuangjiaju/article/details/48580863