hadoop wordcount 示例

本示例假设已经基本配置好hadoop
如何配置hadoop不是本文的重点
如果在终端中输入hadoop没有报错就可以继续往下看

1 查看classpath

hadoop classpath

2 设定环境变量classpath

在终端输入vim ~/.bashrc
在文件中添加export classpath=你的classpath
重启终端

3 创建WordCount.java

    package org.myorg;  
    import java.io.IOException;
    import java.util.*;

    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.conf.*;
    import org.apache.hadoop.io.*;
    import org.apache.hadoop.mapred.*;
    import org.apache.hadoop.util.*;

    public class WordCount {

       public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
         private final static IntWritable one = new IntWritable(1);
         private Text word = new Text();

         public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
           String line = value.toString();
           StringTokenizer tokenizer = new StringTokenizer(line);
           while (tokenizer.hasMoreTokens()) {
             word.set(tokenizer.nextToken());
             output.collect(word, one);
           }
         }
       }

       public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
         public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
           int sum = 0;
           while (values.hasNext()) {
             sum += values.next().get();
           }
           output.collect(key, new IntWritable(sum));
         }
       }

       public static void main(String[] args) throws Exception {
         JobConf conf = new JobConf(WordCount.class);
         conf.setJobName("wordcount");

         conf.setOutputKeyClass(Text.class);
         conf.setOutputValueClass(IntWritable.class);

         conf.setMapperClass(Map.class);
         conf.setCombinerClass(Reduce.class);
         conf.setReducerClass(Reduce.class);

         conf.setInputFormat(TextInputFormat.class);
         conf.setOutputFormat(TextOutputFormat.class);

         FileInputFormat.setInputPaths(conf, new Path(args[0]));
         FileOutputFormat.setOutputPath(conf, new Path(args[1]));

         JobClient.runJob(conf);
       }
    }

4 生成class文件

首先在终端输入mkdir wordcount_classes
现在的目录大概是这样的
WordCount.java(我们现在在这个目录)
—–wordcount_classes
然后执行

javac -classpath ${classpath} -d wordcount_classes WordCount.java
// 把WordCount.java生成class放到wordcount_classes目录下

5 打包.class生成.jar

在终端输入

jar -cvf wc.jar wordcount_classes/ .
//将wordcount_classes目录下所有文件打包成jar文件，jar名字不要与当前存在的目录重名即可

现在的目录
wordcount.java（我们在这里
wc.jar
—-wordcount_classes

6 生成一些测试数据样例

在终端中输入

mkdir sample
cd sample
mkdir input
cd input
touch file1
echo helloword >> file1
cd ../..

现在的目录
wordcount.java（我们在这里
wc.jar
—-wordcount_classes
—-sample
——–input
————file1

7用hadoop运行jar文件

终端输入
hadoop jar wc.jar org.myorg.WordCount sample/input sample/output
解释一下
wc.jar是我们的包文件 org.myorg.WordCount是我们的class名字 后面两个目录是输入输出

（截图中目录名字与本示例不一样，我在本文中将wordcount目录换为sample因为这个名字更能让人知道目录的作用)

8 查看结果

终端输入
cat sample/output/part-0000

(具体数字不太一样是因为我在示例中使用的数据和我真实使用的不一样)