Spark map与flatmap区别

案例说明1：

步骤一：将测试数据放到hdfs上面

hadoopdfs -put data1/test1.txt /tmp/test1.txt

该测试数据有两行文本：

步骤二：在Spark中创建一个RDD来读取hdfs文件/tmp/test1.txt

步骤三：查看map函数的返回值

得到map函数返回的RDD: 

查看map函数的返回值——文件中的每一行数据返回了一个数组对象

步骤四：查看flatMap函数的返回值

得到flatMap函数返回的RDD:

查看flatMap函数的返回值——文件中的所有行数据仅返回了一个数组对象

总结：

- Spark 中 map函数会对每一条输入进行指定的操作，然后为每一条输入返回一个对象；

- 而flatMap函数则是两个操作的集合——正是“先映射后扁平化”：

   操作1：同map函数一样：对每一条输入进行指定的操作，然后为每一条输入返回一个对象

   操作2：最后将所有对象合并为一个对象

案例说明2：

假设存在如下文件：

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@bluejoe0</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># cat categories.csv </span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,生活用品,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,数码用品,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>,手机,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>,华为<span class="hljs-constant" style="box-sizing: border-box;">Mate7</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

每一行的格式为：类别ID，类别名称，父类ID

现在欲输出每个类别的父类别的名称，类似于SQL的自连接，注意到join的外键其实是父类ID。

首先生成“父类ID->子类ID，子类名称”

<code class="hljs coffeescript has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">val categories=sc.textFile(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/root/categories.csv"</span>) val left = categories.map<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(_.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>))</span>.<span class="hljs-title" style="box-sizing: border-box;">map</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(x=>(x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)->Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"id"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))))</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

left的内容为：

Array((0,Map(id -> 1, name -> 生活用品)), (1,Map(id -> 2, name -> 数码用品)), (2,Map(id -> 3, name -> 手机)), (3,Map(id -> 4, name -> 华为Mate7)))

接着生成“父类ID->父类ID，父类名称”

<code class="hljs coffeescript has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">val right = categories.map<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(_.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>))</span>.<span class="hljs-title" style="box-sizing: border-box;">map</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(x=>(x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>)->Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pid"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pname"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))))</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

right的内容为：

Array((1,Map(pid -> 1, pname -> 生活用品)), (2,Map(pid -> 2, pname -> 数码用品)), (3,Map(pid -> 3, pname -> 手机)), (4,Map(pid -> 4, pname -> 华为Mate7)))

接下来，将这两个RDD进行合并，并按照key（key都是父类ID）进行reduce：

<code class="hljs brainfuck has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">val</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">merged</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">=</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">(left</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">+</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">+</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">right)</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">reduceByKey(_</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">+</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">+</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">_)</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

merged的内容为：

Array((4,Map(pid -> 4, pname -> 华为Mate7)), (0,Map(id -> 1, name -> 生活用品)), (1,Map(id -> 2, name -> 数码用品, pid -> 1, pname -> 生活用品)), (2,Map(id -> 3, name -> 手机, pid -> 2, pname -> 数码用品)), (3,Map(id -> 4, name -> 华为Mate7, pid -> 3, pname -> 手机)))

搞定！！

可以采用flatMap来简化以上的写法：

<code class="hljs coffeescript has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">val merged = categories.map<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(_.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>))</span>.<span class="hljs-title" style="box-sizing: border-box;">flatMap</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">((x)=>Array(x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)->Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"id"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)), x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>)->Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pid"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pname"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))))</span>.<span class="hljs-title" style="box-sizing: border-box;">reduceByKey</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(_++_)</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

结果是一样一样的！！当然代码的可读性大打折扣了~~~