ubuntu下的hadoop安装

1. 环境

ubuntu 14.04 64位
java 1.8.0_45（1.7即可）
ssh（sshd运行）

2. 安装版本

hadoop2.7.2（http://hadoop.apache.org/releases.html）

3. 安装过程

1.下载hadoop2.7.2的binary版本 http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
2.解压，并将解压得到的hadoop-2.7.2文件夹移到/usr/local中
3.cd 到/usr/local/hadoop-2.7.2目录
4.设置JAVA_HOME参数：在当前目录的etc/hadoop/文件末尾添加

# set to the root of your Java installation
export JAVA_HOME=/usr/local/jdk1.8.0/

5.执行命令bin/hadoop，打印hadoop用法。（注意：不是/bin/hadoop）
6.此时，hadoop默认是Standalone单机模式，测试hadoop如下：

$ mkdir input
$ cp etc/hadoop/*.xml input
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep input output 'dfs[a-z.]+'
$ cat output/*

4. mapreduce简单示例

a、编写map，reduce和配置，保存成WordCount.java

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
      }
    }
  }

  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

b、编译WordCount.java

先设置环境变量：

export JAVA_HOME=/usr/local/jdk1.8.0
export PATH=${JAVA_HOME}/bin:${PATH}
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar

其中，HADOOP_CLASSPATH是指定hadoop搜索哪些路径下的.class文件，下面用到的com.sun.tools.javac.Main就在${JAVA_HOME}/lib/tools.jar中。

然后，编译：

${HADOOP_HOME}/bin/hadoop com.sun.tools.javac.Main WordCount.java
jar cf wc.jar WordCount*.class

当前目录下生成WordCount$IntSumReducer.class、WordCount$TokenizerMapper.class和WordCount.class三个文件；再用jar打包这三个文件，生成wc.jar。

c、运行

${HADOOP_HOME}/bin/hadoop jar wc.jar WordCount ./input ./output

输入文件放在input文件夹中，hadoop生成output目录保存结果。

d、歪楼

2中可以不打包class文件，只要把class所在文件夹的路径(当前路径)加到HADOOP_CLASSPATH中即可，也即

export HADOOP_CLASSPATH=.:${HADOOP_CLASSPATH}

然后直接运行即可：

${HADOOP_HOME}/bin/hadoop WordCount ./input ./output

参考：

[1] Hadoop: Setting up a Single Node Cluster.
[2] ubuntu下搭建JAVA开发环境
[3] MapReduce Tutorial