深入理解与应用Hadoop中的MapReduce

最新推荐文章于 2022-10-29 12:48:59 发布

00u0o

最新推荐文章于 2022-10-29 12:48:59 发布

阅读量218

点赞数

现在大数据是越来越火了，而我自己研究这方面也很长时间了，今天就根据我自己的经验教会大家学会如何使用MapReduce，下文中将MapReduce简写为MR。
本篇博客将结合实际案例来具体说明MR的每一个知识点。

1、本篇博客核心内容:

这里写图片描述

2、MR的基本概念

这里写图片描述

3、MR中map()函数和reduce()函数如何编写

这里写图片描述

4、MR程序的基本编写流程(MR的基本执行过程)

这里写图片描述

下面将用一个具体的电信业务说明MR最基本的编写过程：
实验所用数据:
这里写图片描述
具体字段描述:

业务要求:统计同一个用户的上行总流量和，下行总流量和以及上下总流量和
例如：

代码示例：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">package <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">com</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.appache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.celephone</span>3<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>


import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.conf</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Configuration</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FSDataInputStream</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileSystem</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Path</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.io</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.IOUtils</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.io</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Text</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Job</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.input</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileInputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.input</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TextInputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.output</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileOutputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.output</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TextOutputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>


public class FlowCount
{
    public static String path1 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dir/flowdata.txt"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
    public static String path2 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dirout/"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
    public static void main(String[] args) throws Exception
    {
        Configuration conf = new Configuration()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.set</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"fs.default.name"</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        FileSystem fileSystem = FileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.get</span>(conf)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        if(fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.exists</span>(new Path(path2)))
        {
            fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.delete</span>(new Path(path2), true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        }       
        Job job = new Job(conf,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"FlowCount"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setJarByClass</span>(FlowCount<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        //编写驱动
        FileInputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setInputPaths</span>(job, new Path(path1))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setInputFormatClass</span>(TextInputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapperClass</span>(MyMapper<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapOutputKeyClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapOutputValueClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        //shuffle洗牌阶段
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setReducerClass</span>(MyReducer<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputKeyClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputValueClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputFormatClass</span>(TextOutputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        FileOutputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputPath</span>(job, new Path(path2))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;   </span>
        //将任务提交给JobTracker
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.waitForCompletion</span>(true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        //查看程序的运行结果
        FSDataInputStream fr = fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.open</span>(new Path(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dirout/part-r-00000"</span>))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        IOUtils<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.copyBytes</span>(fr,System<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.out</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1024</span>,true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
    }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li></ul>

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.appache.celephone3;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.LongWritable;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.Text;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.mapreduce.Mapper;

public <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyMapper</span>   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Mapper</span><<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">LongWritable</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>></span>
{
    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">protected</span> void map(LongWritable k1, Text v1,Context context)<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException, InterruptedException
    {
        String line = v1.toString();<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//拿到日志中的一行数据</span>
        String[] splited = line.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>);<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//切分各个字段</span>
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//获取我们所需要的字段</span>
        String msisdn = splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>];
        String upFlow = splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>];
        String downFlow = splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>];
        long flowsum = Long.parseLong(upFlow) + Long.parseLong(downFlow);
        context.write(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Text(msisdn), <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Text(upFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+downFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+String.valueOf(flowsum)));
    }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li></ul>

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.appache.celephone3;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.Text;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.mapreduce.Reducer;

public <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyReducer</span>  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Reducer</span><<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>></span>
{
    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">protected</span> void reduce(Text k2, Iterable<Text> v2s,Context context)<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException, InterruptedException
    {
       long upFlowSum = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L;
       long downFlowSum = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L;
       long FlowSum = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L;
       <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span>(Text v2:v2s)
       {
           String[] splited = v2.toString().split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>);
           upFlowSum += Long.parseLong(splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]);
           downFlowSum += Long.parseLong(splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>]);
           FlowSum += Long.parseLong(splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>]);
       }
       String data = String.valueOf(upFlowSum)+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+String.valueOf(downFlowSum)+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+String.valueOf(FlowSum);
       context.write(k2,<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Text(data));
    }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li></ul>

运行结果：
这里写图片描述

这里写图片描述

5、MR程序的优化方式1—分区

这里写图片描述

具体业务描述：对于上面的电信数据，统计同一个用户的上行总流量和，下行总流量和以及上下总流量和，并且手机号(11位)的信息输出到一个文件中，非手机号(8位)的信息输出到一个文件中
代码示例：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">package <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">com</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.appache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.partitioner</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>


import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.conf</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Configuration</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileSystem</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Path</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.io</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Text</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Job</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.input</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileInputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.input</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TextInputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.output</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileOutputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.output</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TextOutputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>


public class FlowCount
{
   public static String path1 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dir/flowdata.txt"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
   public static String path2 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dirout/"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
   public static void main(String[] args) throws Exception
   {
       Configuration conf = new Configuration()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.set</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"fs.default.name"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       FileSystem fileSystem = FileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.get</span>(conf)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">; </span>
       if(fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.exists</span>(new Path(path2)))
       {
           fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.delete</span>(new Path(path2), true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       }

       Job job = new Job(conf,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"FlowCount"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setJarByClass</span>(FlowCount<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       FileInputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setInputPaths</span>(job, new Path(path1))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setInputFormatClass</span>(TextInputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;//<k1,v1></span>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapperClass</span>(MyMapper<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapOutputKeyClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapOutputValueClass</span>(FlowBean<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;//<k2,v2></span>
       //shuffle阶段:分区、排序、分组、本地归并
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setPartitionerClass</span>(MyPartitioner<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setNumReduceTasks</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       //<k2,v2s>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setReducerClass</span>(MyReducer<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputKeyClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputValueClass</span>(FlowBean<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputFormatClass</span>(TextOutputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       FileOutputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputPath</span>(job, new Path(path2))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">; </span>
       //提交作业
       job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.waitForCompletion</span>(true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;      </span>
   }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li></ul>

<code class="hljs java has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.appache.partitioner;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.DataInput;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.DataOutput;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.Writable;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlowBean</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">implements</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Writable</span>
{</span>
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> upFlow ;  <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//上行流量</span>
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> downFlow; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//下行流量</span>
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> flowSum; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//总流量</span>

   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-title" style="box-sizing: border-box;">FlowBean</span>() {}
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-title" style="box-sizing: border-box;">FlowBean</span>(String upFlow,String downFlow)
   {
       <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.upFlow = Long.parseLong(upFlow);
       <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.downFlow = Long.parseLong(downFlow);
       <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.flowSum = Long.parseLong(upFlow) + Long.parseLong(downFlow);
   }

   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">getupFlow</span>()  
   {<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> upFlow;}
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">getdownFlow</span>() 
   {<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> downFlow;}
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">getflowSum</span> () 
   {<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> flowSum;}

   <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">write</span>(DataOutput out) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException
   {
      out.writeLong(upFlow);
      out.writeLong(downFlow);
      out.writeLong(flowSum);   
   }
   <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">readFields</span>(DataInput in) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException
   {
      upFlow = in.readLong();
      downFlow = in.readLong();
      flowSum = in.readLong();  
   }  
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> String <span class="hljs-title" style="box-sizing: border-box;">toString</span>()
   {
       <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> upFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+downFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+flowSum;  
   }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li></ul>

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.appache.partitioner;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.LongWritable;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.Text;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.mapreduce.Mapper;

public <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyMapper</span>  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Mapper</span><<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">LongWritable</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlowBean</span>></span>
{
    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">protected</span> void map(LongWritable k1, Text v1,Context context)<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException, InterruptedException
    {
        String line = v1.toString();<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//拿到日志中的一行数据</span>
        String[] splited = line.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>);<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//切分各个字段</span>
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//获取我们所需要的字段</span>
        String msisdn = splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>];<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//手机号 k2</span>
        FlowBean flowData = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> FlowBean(splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>],splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>]);<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//<100,200></span>
        context.write(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Text(msisdn), flowData);
    }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul>

<code class="hljs axapta has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">package com.appache.partitioner;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Partitioner;



<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyPartitioner</span> <span class="hljs-inheritance" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span></span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Partitioner</span><<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>,<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlowBean</span>>  //分区<18330267966,{</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">200</span>}>
{
    @Override
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> getPartition(Text k2, FlowBean v2, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> numPartitions)
    {
       String tele = k2.toString();
       <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span>(tele.length() ==  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span>)
           <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>;  <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//手机号的信息输出到0区</span>
       <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> 
           <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>;  <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//非手机号的信息输出到1区</span>
    }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li></ul>

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.appache.partitioner;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.Text;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.mapreduce.Reducer;

public <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyReducer</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Reducer</span><<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlowBean</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlowBean</span>></span>
{
    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">protected</span> void reduce(Text k2, Iterable<FlowBean> v2s,Context context)<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException, InterruptedException
    {
        long upFlow = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L;
        long downFlow = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L;
        long flowSum = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L;
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span>(FlowBean v2: v2s)
        {
            upFlow += v2.getupFlow();
            downFlow += v2.getdownFlow();
            flowSum += v2.getflowSum();
        }
        context.write(k2, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> FlowBean(upFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">""</span>,downFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">""</span>));  <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//将数据输出到指定的文件当中</span>
    }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li></ul>

运行结果：
这里写图片描述

6、MR程序的优化方式2—自定义排序

这里写图片描述

业务描述:
对于上面业务得到的统计结果:
这里写图片描述

先按照总流量由低到高排序，在总流量相同的情况下，按照下行流量和从低到高排序:
实例代码:

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">package <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">com</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.appache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sort</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>

import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.conf</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Configuration</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FSDataInputStream</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileSystem</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Path</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.io</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.IOUtils</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.io</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.NullWritable</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Job</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.input</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileInputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.input</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TextInputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.output</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileOutputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.output</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TextOutputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>


public class FlowCount  
{
       public static String path1 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/flowCount.txt"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       public static String path2 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dirout/"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       public static void main(String[] args) throws Exception
       {
           Configuration conf = new Configuration()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.set</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"fs.default.name"</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           FileSystem fileSystem = FileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.get</span>(conf)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           if(fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.exists</span>(new Path(path2)))
           {
               fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.delete</span>(new Path(path2), true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           }

           Job job = new Job(conf, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"FlowCount"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setJarByClass</span>(FlowCount<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>

           //编写驱动
           FileInputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setInputPaths</span>(job,new Path(path1))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;  //输入文件的路径</span>
           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setInputFormatClass</span>(TextInputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;//<k1,v1></span>
           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapperClass</span>(MyMapper<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapOutputKeyClass</span>(FlowBean<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapOutputValueClass</span>(NullWritable<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           //shuffle优化阶段
           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setReducerClass</span>(MyReducer<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputKeyClass</span>(FlowBean<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputValueClass</span>(NullWritable<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputFormatClass</span>(TextOutputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           FileOutputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputPath</span>(job,new Path(path2))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>

           job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.waitForCompletion</span>(true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;    </span>
           //查看运行结果:
           FSDataInputStream fr = fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.open</span>(new Path(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dirout/part-r-00000"</span>))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
           IOUtils<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.copyBytes</span>(fr,System<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.out</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1024</span>,true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
       }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li></ul>

<code class="hljs cs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">package com.appache.sort;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

import org.apache.hadoop.io.WritableComparable;



<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> FlowBean  implements  WritableComparable<FlowBean>
{
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> String msisdn;  <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//获取我们所需要的字段</span>
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> upFlow;
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> downFlow;
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> flowSum;

   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-title" style="box-sizing: border-box;">FlowBean</span>(){}
   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-title" style="box-sizing: border-box;">FlowBean</span>(String msisdn,String upFlow,String downFlow,String flowSum)
   {
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.msisdn = msisdn;
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.upFlow = Long.parseLong(upFlow);
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.downFlow = Long.parseLong(downFlow);
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.flowSum = Long.parseLong(flowSum); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//通过构造函数自动求取总流量</span>
   }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> String <span class="hljs-title" style="box-sizing: border-box;">getMsisdn</span>()
    {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> msisdn;
    }
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">getUpFlow</span>()
    {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> upFlow;
    }
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">getDownFlow</span>()
    {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> downFlow;
    }
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">getFlowSum</span>()
    {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> flowSum;
    }

    @Override   <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//所谓序列化就是将对象写到字节输出流当中</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">write</span>(DataOutput <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">out</span>) throws IOException
    {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">out</span>.writeUTF(msisdn);
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">out</span>.writeLong(upFlow);
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">out</span>.writeLong(downFlow);
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">out</span>.writeLong(flowSum);
    }
    @Override  <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//所谓反序列化就是将对象从输入流当中给读取出来</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">readFields</span>(DataInput <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span>) throws IOException
    {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.msisdn = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span>.readUTF();
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.upFlow = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span>.readLong();
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.downFlow = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span>.readLong();
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.flowSum = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span>.readLong();
    }

    @Override   <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//指定比较的标准</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> <span class="hljs-title" style="box-sizing: border-box;">compareTo</span>(FlowBean obj)
    {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span>(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.flowSum == obj.flowSum)   
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span>)(obj.downFlow - <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.downFlow);  <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//下行流量由高到底</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> 
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span>)(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.flowSum - obj.flowSum);   <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//总流量由低到高  </span>
    }
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> String <span class="hljs-title" style="box-sizing: border-box;">toString</span>()
    {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.msisdn+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.upFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.downFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.flowSum;
    }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li></ul>

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.appache.sort;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.LongWritable;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.NullWritable;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.Text;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.mapreduce.Mapper;

public <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyMapper</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Mapper</span><<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">LongWritable</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlowBean</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">NullWritable</span>></span>
{
    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">protected</span> void map(LongWritable k1, Text v1,Context context)<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException, InterruptedException
    {
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//拿到日志中的一行数据</span>
        String line = v1.toString();
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//切分各个字段</span>
        String[] splited = line.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>);
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//获取我们所需要的字段---并用FlowBean存储我们所需要的字段</span>
        FlowBean flowdata = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> FlowBean(splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>],splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>],splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>],splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>]);
        context.write(flowdata, NullWritable.get()); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//<{18330267966,100,200},null></span>
    }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul>

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.appache.sort;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.NullWritable;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.mapreduce.Reducer;

public <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyReducer</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Reducer</span><<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlowBean</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">NullWritable</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlowBean</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">NullWritable</span>></span>
{
    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>  
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">protected</span> void reduce(FlowBean k2, Iterable<NullWritable> v2s,Context context)<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException, InterruptedException
    {  
       <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span>(NullWritable v2:v2s)
       {
           context.write(k2, NullWritable.get());
       }
    }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li></ul>

运行结果:
这里写图片描述

7、MR程序的优化方式3—本地归并Combiner

这里写图片描述

具体业务描述：对于上面的电信数据，统计同一个用户的上行总流量和，下行总流量和以及上下总流量和，代码中要求加入本地归并优化方式:
代码示例：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">package <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">com</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.appache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.celephone</span>3<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>


import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.conf</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Configuration</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FSDataInputStream</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileSystem</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Path</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.io</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.IOUtils</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.io</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Text</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Job</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.input</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileInputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.input</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TextInputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.output</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileOutputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hadoop</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.mapreduce</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.lib</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.output</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TextOutputFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>


public class FlowCount
{
    public static String path1 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dir/flowdata.txt"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
    public static String path2 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dirout/"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
    public static void main(String[] args) throws Exception
    {
        Configuration conf = new Configuration()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.set</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"fs.default.name"</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        FileSystem fileSystem = FileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.get</span>(conf)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        if(fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.exists</span>(new Path(path2)))
        {
            fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.delete</span>(new Path(path2), true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        }       
        Job job = new Job(conf,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"FlowCount"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setJarByClass</span>(FlowCount<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        //编写驱动
        FileInputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setInputPaths</span>(job, new Path(path1))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setInputFormatClass</span>(TextInputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapperClass</span>(MyMapper<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapOutputKeyClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMapOutputValueClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        //加入本地归并优化方式:
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setCombinerClass</span>(MyReducer<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setNumReduceTasks</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        //shuffle洗牌阶段
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setReducerClass</span>(MyReducer<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputKeyClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputValueClass</span>(Text<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputFormatClass</span>(TextOutputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        FileOutputFormat<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setOutputPath</span>(job, new Path(path2))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;   </span>
        //将任务提交给JobTracker
        job<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.waitForCompletion</span>(true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        //查看程序的运行结果
        FSDataInputStream fr = fileSystem<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.open</span>(new Path(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://hadoop:9000/dirout/part-r-00000"</span>))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
        IOUtils<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.copyBytes</span>(fr,System<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.out</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1024</span>,true)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>
    }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li></ul>

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.appache.celephone3;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.LongWritable;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.Text;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.mapreduce.Mapper;

public <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyMapper</span>   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Mapper</span><<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">LongWritable</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>></span>
{
    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">protected</span> void map(LongWritable k1, Text v1,Context context)<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException, InterruptedException
    {
        String line = v1.toString();<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//拿到日志中的一行数据</span>
        String[] splited = line.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>);<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//切分各个字段</span>
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//获取我们所需要的字段</span>
        String msisdn = splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>];
        String upFlow = splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>];
        String downFlow = splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>];
        long flowsum = Long.parseLong(upFlow) + Long.parseLong(downFlow);
        context.write(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Text(msisdn), <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Text(upFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+downFlow+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+String.valueOf(flowsum)));
    }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul>

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.appache.celephone3;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.io.Text;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.hadoop.mapreduce.Reducer;

public <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyReducer</span>  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Reducer</span><<span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>, <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Text</span>></span>
{
    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">protected</span> void reduce(Text k2, Iterable<Text> v2s,Context context)<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> IOException, InterruptedException
    {
       long upFlowSum = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L;
       long downFlowSum = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L;
       long FlowSum = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L;
       <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span>(Text v2:v2s)
       {
           String[] splited = v2.toString().split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>);
           upFlowSum += Long.parseLong(splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]);
           downFlowSum += Long.parseLong(splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>]);
           FlowSum += Long.parseLong(splited[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>]);
       }
       String data = String.valueOf(upFlowSum)+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+String.valueOf(downFlowSum)+<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>+String.valueOf(FlowSum);
       context.write(k2,<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Text(data));
    }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li></ul>

运行结果：
这里写图片描述

8、MR程序的优化方式4—自定义计数器

这里写图片描述

9、如何用MR实现某个业务的方式总结

这里写图片描述

00u0o

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
深入理解与应用Hadoop中的MapReduce

现在大数据是越来越火了，而我自己研究这方面也很长时间了，今天就根据我自己的经验教会大家学会如何使用MapReduce，下文中将MapReduce简写为MR。本篇博客将结合实际案例来具体说明MR的每一个知识点。1、本篇博客核心内容:2、MR的基本概念3、MR中map()函数和reduce()函数
复制链接

扫一扫