Win7下提交MapReduceJob到集群环境

1、下载hadoop-2.5.2.tar.gz，解压到D:\hadoop目录下

2、下载hadoop-common-2.2.0-bin-master.zip，将其中的hadoop.dll和winutils.exe复制到D:\hadoop\hadoop-2.5.2\bin目录下

3、配置环境变量HADOOP_HOME= D:\hadoop\hadoop-2.5.2，在PATH中添加%HADOOP_HOME%\bin

4、复制org.apache.hadoop.mapred.YARNRunner.java，修改如下三个地方

1) 替换Line390：391

    //vargs.add(MRApps.crossPlatformifyMREnv(jobConf, Environment.JAVA_HOME) + "/bin/java");

    vargs.add("$JAVA_HOME/bin/java");

2) 在Line456插入

    replaceEnvironment(environment);

3) 在Line638插入

  private void replaceEnvironment(Map<String, String> environment) {

    String tmpClassPath = (String)environment.get("CLASSPATH");
    tmpClassPath = tmpClassPath.replaceAll(";", ":");
    tmpClassPath = tmpClassPath.replaceAll("%PWD%", "\\$PWD");
    tmpClassPath = tmpClassPath.replaceAll("%HADOOP", "\\$HADOOP");
    tmpClassPath = tmpClassPath.replaceAll("\\\\", "/").replaceAll("%", "");
    environment.put("CLASSPATH", tmpClassPath);
  }

5、用新的YARNRunner.class替换hadoop-mapreduce-client-jobclient-2.5.2.jar中的同名文件

6、在Eclipse中创建一个JavaProject，新增一个用户库hadoop_2.5.2，将hadoop-2.5.2中的jar添加至这个用户库，在Project中引用这个用户库

7、编写wordcount程序，并提交到Hadoop集群，即可得到正确结果，不会报如下错误

1. Job job_1418033228434_0012 failed with state FAILED due to: Application application_1418033228434_0012 failed 2 times due to AM Container for appattempt_1418033228434_0012_000002 exited with  exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control
   
   
   ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control
   
   
          at org.apache.hadoop.util.Shell.runCommand( Shell.java:538 )
          at org.apache.hadoop.util.Shell.run( Shell.java:455 )
          at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( Shell.java:702)
          at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:300)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:81)
          at java.util.concurrent.FutureTask.run( FutureTask.java:262 )
          at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615)
          at java.lang.Thread.run( Thread.java:744 )
   
   
   
   
   Container exited with a non-zero exit code 1
   .Failing this attempt.. Failing the application.
   2014-12-09 13:42:05,383 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380) ) - Counters: 0

8、如果报下面的错误，请注意可能是你修改了hadoop-mapreduce-client-core-2.5.2.jar中的mapred-default.xml，比如在其中添加了mapreduce.application.classpath，请将修改还原。

1. Job job_1418033228434_0013 failed with state FAILED due to: Application application_1418033228434_0013 failed 2 times due to AM Container for appattempt_1418033228434_0013_000002 exited with  exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1:

   ExitCodeException exitCode=1:

          at org.apache.hadoop.util.Shell.runCommand( Shell.java:538 )

          at org.apache.hadoop.util.Shell.run( Shell.java:455 )

          at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( Shell.java:702)

          at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195)

          at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:300)

          at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:81)

          at java.util.concurrent.FutureTask.run( FutureTask.java:262 )

          at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615)

          at java.lang.Thread.run( Thread.java:744 )

   
   

   
   

   Container exited with a non-zero exit code 1

   .Failing this attempt.. Failing the application.

   2014-12-09 13:45:02,949 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380) ) - Counters: 0