第一个Hadoop程序

hadoop资料之前倒是有看过，只是工作一直没涉及，自己也没有动手过。之前尝试高版本的apache版本机编译，居然都不成功。初衷只是要搭建环境，却由于都放在业余时间搞，时间不连续，尝试的方法不同(中间尝试安装cdh4.40,结果发现面对一堆文档还不是自己想要的，最后还是回到hadoop权威指南附录)，导致这个工作一直难产。虽说最后结果简单，还是记录一下。

下载的是apache稳定版hadoop-0.23.9,0.23之后apache版本往两个分支走:hadoop 1.0和hadoop 2.0。由于这个版本较新却和0.20接近，配置上相似一些，有了之前尝试的教训，就是这个了。解压之后要设定好path，path加上hadoop bin目录。默认配置文件在hadoop etc/hadoop下。

公共配置:core-site.xml

hdfs配置:hdfs-site.xml

mapreduce配置: mapred-site.xml

有三种模式,standalone模式，伪分布式,和分布式。什么都不改启动的是本机模式。这里就是跑在本机模式。

WordCount2.java:

package com.imd.examples;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;

public class WordCount2 {
  public static void main(String[] args) throws Exception {
    // 不配置一个入口类会导致底层无法调用setJarByClass方法,运行时ClassNotFound.
    JobConf conf = new JobConf(WordCount2.class);
    conf.setMapperClass(WordCountMapper.class);
    conf.setReducerClass(WordCountReducer.class);
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);

    // 输入参数除最后一个外都是input path,最后一个是output path
    for (int i = 0; i < args.length - 1; ++i) {
      FileInputFormat.addInputPath(conf, new Path(args[i]));
    }

    FileOutputFormat.setOutputPath(conf, new Path(args[args.length - 1]));
    JobClient.runJob(conf);
  } 
}

WordCountMapper.java:

package com.imd.examples;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class WordCountMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
  private IntWritable one = new IntWritable(1);

  @Override
  // 怎么切分的呢?key是什么?
  public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
    StringTokenizer st = new StringTokenizer(value.toString());
    Text text = new Text();
    while (st.hasMoreTokens()) {
      text.set(st.nextToken());
      output.collect(text, one);
    }
  }
}

WordCountReducer.java:

package com.imd.examples;

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;

public class WordCountReducer extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
  @Override
  public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
    Integer sum = 0;
    while (values.hasNext()) {
      sum += values.next().get();
    }
    output.collect(key, new IntWritable(sum));
  }
}

然后compile:

javac -classpath /home/junhaozhang/hadoop-0.23.9/share/hadoop/mapreduce/hadoop-mapreduce-client-core-0.23.9.jar:/home/junhaozhang/hadoop-0.23.9/share/hadoop/common/hadoop-common-0.23.9.jar *.java -d .

jar -cvf WordCount2.jar com

运行:

hadoop jar WordCount2.jar com.imd.examples.WordCount2 releases.html output

output下看到结果了。

下面就需要尝试下伪分布式和分布式了。