win7 eclipse调用虚拟机ubuntu部署的hadoop2.2.0伪分布（1）_虚拟机的 eclipse import org apache-CSDN博客

所用软件下载网址：链接：http://pan.baidu.com/s/1bn4IIQF密码：ramg

win7环境下jdk下载路径（/jdk/jdk-7u71-windows-i586.exe）

eclipse下载路径（/eclipse/eclipse-jee-indigo-SR2-win32.zip）

hadoop插件下载路径（/hadoop/eclipse插件/hadoop2x-eclipse-plugin.zip）

此片文章原文链接：http://www.linuxidc.com/Linux/2014-09/106148p2.htm，

注：

（1）本人使用的是/home/tom/hadoop-2.2.0，/usr/mywind/hadoop路径替换为/home/tom/hadoop-2.2.0即可

（2）创建用户此文用的是a01513，替换为tom即可

（3）IP地址自行修改

1、基于Eclipse的Hadoop2.x开发环境配置

关于JDK及ECLIPSE的安装我就不再介绍了，相信能玩Hadoop的人对这种配置都已经再熟悉不过了，如果实在不懂建议到谷歌百度去搜索一下教程。假设你已经把Hadoop的Eclipse插件下载下来了，然后解压把jar文件放到Eclipse的plugins文件夹里面：

重启Eclipse即可。

然后我们再安装Hadoop到Win7下，在这不再详细说明，跟安装JDK大同小异，在这个例子中我安装到了E:\hadoop。

启动Eclipse，点击菜单栏的【Windows/窗口】→【Preferences/首选项】→【Hadoop Map/Reduce】,把Hadoop Installation Directory设置成开发机上的Hadoop主目录：

点击OK。

开发环境配置完成，下面我们可以新建一个测试Hadoop项目，右键【NEW/新建】→【Others、其他】，选择Map/Reduce Project

输入项目名称点击【Finish/完成】：

创建完成后可以看到如下目录：

然后在SRC下建立下面包及类：

以下是代码内容：

TestMapper.java

package com.my.hadoop.mapper;
  
import java.io.IOException;
  
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
  
public class TestMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
        private static final int MISSING = 9999;
        private static final Log LOG = LogFactory.getLog(TestMapper.class);
  
          public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output,Reporter reporter)
              throws IOException {
            String line = value.toString();
            String year = line.substring(15, 19);
            int airTemperature;
            if (line.charAt(87) == '+') { // parseInt doesn't like leading plus signs
              airTemperature = Integer.parseInt(line.substring(88, 92));
            } else {
              airTemperature = Integer.parseInt(line.substring(87, 92));
            }
            LOG.info("loki:"+airTemperature);
            String quality = line.substring(92, 93);
            LOG.info("loki2:"+quality);
            if (airTemperature != MISSING && quality.matches("[012459]")) {
              LOG.info("loki3:"+quality);
              output.collect(new Text(year), new IntWritable(airTemperature));
            }
          }
  
}

TestReducer.java

package com.my.hadoop.reducer;
  
import java.io.IOException;
import java.util.Iterator;
  
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.Reducer;
  
public class TestReducer extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
  
        @Override
          public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output,Reporter reporter)
              throws IOException{
            int maxValue = Integer.MIN_VALUE;
            while (values.hasNext()) {
              maxValue = Math.max(maxValue, values.next().get());
            }
            output.collect(key, new IntWritable(maxValue));
          }
  
}

TestHadoop.java

package com.my.hadoop.test.main;
  
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
  
import com.my.hadoop.mapper.TestMapper;
import com.my.hadoop.reducer.TestReducer;
  
public class TestHadoop {
        
        public static void main(String[] args) throws Exception{
                  
                  if (args.length != 2) {
                        System.err
                            .println("Usage: MaxTemperature <input path> <output path>");
                        System.exit(-1);
                      }
                  JobConf job = new JobConf(TestHadoop.class);
            job.setJobName("Max temperature");
            FileInputFormat.addInputPath(job, new Path(args[0]));
            FileOutputFormat.setOutputPath(job, new Path(args[1]));
            job.setMapperClass(TestMapper.class);
            job.setReducerClass(TestReducer.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class);
            JobClient.runJob(job);
        }
        
}

为了方便对于Hadoop的HDFS文件系统操作，我们可以在Eclipse下面的Map/Reduce Locations窗口与Hadoop建立连接，直接右键新建Hadoop连接即可：

连接配置如下：

其中，Location name可任意填写，Mapreduce Master中Host为resourcemanager机器ip，Port为resourcemanager接受任务的端口号，即yarn-site.xml文件中yarn.resourcemanager.scheduler.address配置项中端口号。DFS Master中的Host为namenode机器ip，Port为core-site.xml文件中fs.defaultFS配置项中端口号。

然后点击完成即可，新建完成后，我们可以在左侧目录中看到HDFS的文件系统目录：

这里不仅可以显示目录结构，还可以对文件及目录进行删除、新增等操作，非常方便。

当上面的工作都做好之后，就可以把这个项目导出来了（导成jar文件放到Hadoop服务器上运行）：

点击完成，然后把这个testt.jar文件上传到Hadoop服务器（192.168.8.184）上，目录（其实可以放到其他目录，你自己喜欢）是：

/usr/mywind/hadoop/share/hadoop/mapreduce

如下图：

2、运行Hadoop程序及查看运行日志

当上面的工作准备好了之后，我们运行自己写的Hadoop程序很简单：

$ hadoop jar /usr/mywind/hadoop/share/hadoop/mapreduce/testt.jar com.my.hadoop.test.main.TestHadoop input output

注意这是output文件夹名称不能重复哦，假如你执行了一次，在HDFS文件系统下面会自动生成一个output文件夹，第二次运行时，要么把output文件夹先删除（$ hdfs dfs -rmr /user/a01513/output），要么把命令中的output改成其他名称如output1、output2等等。

如果看到以下输出结果，证明你的运行成功了：

a01513@hadoop :~$ hadoop jar /usr/mywind/hadoop/share/hadoop/mapreduce/testt.jar                                                                              com.my.hadoop.test.main.TestHadoop input output
14/09/02 11:14:03 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0                                                                            :8032
14/09/02 11:14:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0                                                                            :8032
14/09/02 11:14:04 WARN mapreduce.JobSubmitter: Hadoop command-line option parsin                                                                            g not performed. Implement the Tool interface and execute your application with                                                                              ToolRunner to remedy this.
14/09/02 11:14:04 INFO mapred.FileInputFormat: Total input paths to process : 1
14/09/02 11:14:04 INFO mapreduce.JobSubmitter: number of splits:2
14/09/02 11:14:05 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_14                                                                            09386620927_0015
14/09/02 11:14:05 INFO impl.YarnClientImpl: Submitted application application_14                                                                            09386620927_0015
14/09/02 11:14:05 INFO mapreduce.Job: The url to track the job: http://hadoop:80                                                                            88/proxy/application_1409386620927_0015/
14/09/02 11:14:05 INFO mapreduce.Job: Running job: job_1409386620927_0015
14/09/02 11:14:12 INFO mapreduce.Job: Job job_1409386620927_0015 running in uber mode : false
14/09/02 11:14:12 INFO mapreduce.Job:  map 0% reduce 0%
14/09/02 11:14:21 INFO mapreduce.Job:  map 100% reduce 0%
14/09/02 11:14:28 INFO mapreduce.Job:  map 100% reduce 100%
14/09/02 11:14:28 INFO mapreduce.Job: Job job_1409386620927_0015 completed successfully
14/09/02 11:14:29 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=105
                FILE: Number of bytes written=289816
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=1638
                HDFS: Number of bytes written=10
                HDFS: Number of read operations=9
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Launched map tasks=2
                Launched reduce tasks=1
                Data-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=14817
                Total time spent by all reduces in occupied slots (ms)=4500
                Total time spent by all map tasks (ms)=14817
                Total time spent by all reduce tasks (ms)=4500
                Total vcore-seconds taken by all map tasks=14817
                Total vcore-seconds taken by all reduce tasks=4500
                Total megabyte-seconds taken by all map tasks=15172608
                Total megabyte-seconds taken by all reduce tasks=4608000
        Map-Reduce Framework
                Map input records=9
                Map output records=9
                Map output bytes=81
                Map output materialized bytes=111
                Input split bytes=208
                Combine input records=0
                Combine output records=0
                Reduce input groups=1
                Reduce shuffle bytes=111
                Reduce input records=9
                Reduce output records=1
                Spilled Records=18
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=115
                CPU time spent (ms)=1990
                Physical memory (bytes) snapshot=655314944
                Virtual memory (bytes) snapshot=2480295936
                Total committed heap usage (bytes)=466616320
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=1430
        File Output Format Counters
                Bytes Written=10
a01513@hadoop :~$

我们可以到Eclipse查看输出的结果：