Hadoop权威指南环境搭建+实例运行

基于题目, 记录整个过程中所遇到的问题.

环境: windows10中创建的虚拟机Ubuntu16.04(VMware)

1.安装JDK, 安装目录: /usr/lib/jvm/java-8-oracle , 此JDK为以前所装,具体方式不详.
2.安装Hadoop(源代码编译等涉及到的目录放在后图):
(1) 官网下载Hadoop 2.8.5 二进制包
(2) 解压包: sudo tar zxvf hadoop-2.8.5.tar.gz -C /opt
(3) 创建软链接: sudo ln -snf /opt/hadoop-2.8.5 /opt/hadoop
(4) 更改环境变量: sudo vim /etc/profile
开头添加如下内容:

export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export CLASSPATH=.:$HADOOP_HOME/share/hadoop/common/hadoop-common-2.8.5.jar:$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.5.jar:$HADOOP_HOME/share/hadoop/common/lib/commons-cli-1.2.jar:$CLASSPATH

注意: 最后一个export如果没有,在编译书中三个Java源码时javac *.java,便会找不到与Hadoop相关的类,详见文章最后错误1, 经网上多个教程踩坑, 得出上述结论.可参考:错误分析
(5) 更新环境变量: source /etc/profile
注意: 这样更新只会起临时效果, 再开一个Terminal将不起作用, 输入reboot重启即可解决.
(6) 将编译前后的文件整体打包jar -cvf MaxTemperature.jar . // 最后的 . 一定不要遗漏, 否则失败
(7) 准备数据: 因为书中数据是随便举例的,因此我在天气数据下载了008260-99999-2010.gz, 解压到 temdata重命名为008260-99999-2010.txt
(8) 新建目录, 然后建立HDFS输入路径: hadoop fs -mkdir -p example_1/data
(9) 将天气数据(008260-99999-2010.txt)复制到HDFS输入路径: hadoop fs -put temdata/008260-99999-2010.txt example_1/data
(10) 运行: hadoop jar MaxTemperature.jar MaxTemperature example_1/data/008260-99999-2010.txt example_1/dataoutput 也可以采用书中运行的方式, 更改为自己对应的文件即可.
(11) 查看两个生成文件(part-r-00000 和 _SUCCESS): hadoop fs -ls example_1/dataoutput
(12) 打印结果: hadoop fs -cat example_1/dataoutput/part-r-00000
最终显示2010 380 , 至此, 第一个例子完成!
如果对你有用, 欢迎点赞, 也欢迎评论交流.

源代码编译等涉及到的目录:

错误1:

MaxTemperature.java:3: error: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
                           ^
MaxTemperature.java:4: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.IntWritable;
                           ^
MaxTemperature.java:5: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
MaxTemperature.java:6: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Job;
                                  ^
MaxTemperature.java:7: error: package org.apache.hadoop.mapreduce.lib.input does not exist
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
                                            ^
MaxTemperature.java:8: error: package org.apache.hadoop.mapreduce.lib.output does not exist
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
                                             ^
MaxTemperatureMapper.java:5: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.IntWritable;
                           ^
MaxTemperatureMapper.java:6: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.LongWritable;
                           ^
MaxTemperatureMapper.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
MaxTemperatureMapper.java:8: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Mapper;
                                  ^
MaxTemperatureMapper.java:11: error: cannot find symbol
  extends Mapper<LongWritable, Text, Text, IntWritable> {
          ^
  symbol: class Mapper
MaxTemperatureMapper.java:11: error: cannot find symbol
  extends Mapper<LongWritable, Text, Text, IntWritable> {
                 ^
  symbol: class LongWritable
MaxTemperatureMapper.java:11: error: cannot find symbol
  extends Mapper<LongWritable, Text, Text, IntWritable> {
                               ^
  symbol: class Text
MaxTemperatureMapper.java:11: error: cannot find symbol
  extends Mapper<LongWritable, Text, Text, IntWritable> {
                                     ^
  symbol: class Text
MaxTemperatureMapper.java:11: error: cannot find symbol
  extends Mapper<LongWritable, Text, Text, IntWritable> {
                                           ^
  symbol: class IntWritable
MaxTemperatureMapper.java:16: error: cannot find symbol
  public void map(LongWritable key, Text value, Context context)
                  ^
  symbol:   class LongWritable
  location: class MaxTemperatureMapper
MaxTemperatureMapper.java:16: error: cannot find symbol
  public void map(LongWritable key, Text value, Context context)
                                    ^
  symbol:   class Text
  location: class MaxTemperatureMapper
MaxTemperatureMapper.java:16: error: cannot find symbol
  public void map(LongWritable key, Text value, Context context)
                                                ^
  symbol:   class Context
  location: class MaxTemperatureMapper
MaxTemperatureReducer.java:5: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.IntWritable;
                           ^
MaxTemperatureReducer.java:6: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
MaxTemperatureReducer.java:7: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Reducer;
                                  ^
MaxTemperatureReducer.java:10: error: cannot find symbol
  extends Reducer<Text, IntWritable, Text, IntWritable> {
          ^
  symbol: class Reducer
MaxTemperatureReducer.java:10: error: cannot find symbol
  extends Reducer<Text, IntWritable, Text, IntWritable> {
                  ^
  symbol: class Text
MaxTemperatureReducer.java:10: error: cannot find symbol
  extends Reducer<Text, IntWritable, Text, IntWritable> {
                        ^
  symbol: class IntWritable
MaxTemperatureReducer.java:10: error: cannot find symbol
  extends Reducer<Text, IntWritable, Text, IntWritable> {
                                     ^
  symbol: class Text
MaxTemperatureReducer.java:10: error: cannot find symbol
  extends Reducer<Text, IntWritable, Text, IntWritable> {
                                           ^
  symbol: class IntWritable
MaxTemperatureReducer.java:13: error: cannot find symbol
  public void reduce(Text key, Iterable<IntWritable> values,
                     ^
  symbol:   class Text
  location: class MaxTemperatureReducer
MaxTemperatureReducer.java:13: error: cannot find symbol
  public void reduce(Text key, Iterable<IntWritable> values,
                                        ^
  symbol:   class IntWritable
  location: class MaxTemperatureReducer
MaxTemperatureReducer.java:14: error: cannot find symbol
      Context context)
      ^
  symbol:   class Context
  location: class MaxTemperatureReducer
MaxTemperature.java:18: error: cannot find symbol
    Job job = new Job();
    ^
  symbol:   class Job
  location: class MaxTemperature
MaxTemperature.java:18: error: cannot find symbol
    Job job = new Job();
                  ^
  symbol:   class Job
  location: class MaxTemperature
MaxTemperature.java:22: error: cannot find symbol
    FileInputFormat.addInputPath(job, new Path(args[0]));
                                          ^
  symbol:   class Path
  location: class MaxTemperature
MaxTemperature.java:22: error: cannot find symbol
    FileInputFormat.addInputPath(job, new Path(args[0]));
    ^
  symbol:   variable FileInputFormat
  location: class MaxTemperature
MaxTemperature.java:23: error: cannot find symbol
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
                                            ^
  symbol:   class Path
  location: class MaxTemperature
MaxTemperature.java:23: error: cannot find symbol
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    ^
  symbol:   variable FileOutputFormat
  location: class MaxTemperature
MaxTemperature.java:28: error: cannot find symbol
    job.setOutputKeyClass(Text.class);
                          ^
  symbol:   class Text
  location: class MaxTemperature
MaxTemperature.java:29: error: cannot find symbol
    job.setOutputValueClass(IntWritable.class);
                            ^
  symbol:   class IntWritable
  location: class MaxTemperature
MaxTemperatureMapper.java:15: error: method does not override or implement a method from a supertype
  @Override
  ^
MaxTemperatureMapper.java:29: error: cannot find symbol
      context.write(new Text(year), new IntWritable(airTemperature));
                        ^
  symbol:   class Text
  location: class MaxTemperatureMapper
MaxTemperatureMapper.java:29: error: cannot find symbol
      context.write(new Text(year), new IntWritable(airTemperature));
                                        ^
  symbol:   class IntWritable
  location: class MaxTemperatureMapper
MaxTemperatureReducer.java:12: error: method does not override or implement a method from a supertype
  @Override
  ^
MaxTemperatureReducer.java:18: error: cannot find symbol
    for (IntWritable value : values) {
         ^
  symbol:   class IntWritable
  location: class MaxTemperatureReducer
MaxTemperatureReducer.java:21: error: cannot find symbol
    context.write(key, new IntWritable(maxValue));
                           ^
  symbol:   class IntWritable
  location: class MaxTemperatureReducer
43 errors

参考1
参考2
参考3
参考4
参考5