Win7下提交MapReduceJob到集群环境

1、下载hadoop-2.5.2.tar.gz,解压到D:\hadoop目录下

2、下载hadoop-common-2.2.0-bin-master.zip,将其中的hadoop.dll和winutils.exe复制到D:\hadoop\hadoop-2.5.2\bin目录下


4、复制org.apache.hadoop.mapred.YARNRunner.java,修改如下三个地方
1) 替换Line390:391
    //vargs.add(MRApps.crossPlatformifyMREnv(jobConf, Environment.JAVA_HOME) + "/bin/java");
    vargs.add("$JAVA_HOME/bin/java");
2) 在Line456插入
    replaceEnvironment(environment);
3) 在Line638插入
  private void replaceEnvironment(Map<String, String> environment) {
    String tmpClassPath = (String)environment.get("CLASSPATH");
    tmpClassPath = tmpClassPath.replaceAll(";", ":");
    tmpClassPath = tmpClassPath.replaceAll("%PWD%", "\\$PWD");
    tmpClassPath = tmpClassPath.replaceAll("%HADOOP", "\\$HADOOP");
    tmpClassPath = tmpClassPath.replaceAll("\\\\", "/").replaceAll("%", "");
    environment.put("CLASSPATH", tmpClassPath);
  }
5、用新的YARNRunner.class替换hadoop-mapreduce-client-jobclient-2.5.2.jar中的同名文件

6、在Eclipse中创建一个JavaProject,新增一个用户库hadoop_2.5.2,将hadoop-2.5.2中的jar添加至这个用户库,在Project中引用这个用户库

7、编写wordcount程序,并提交到Hadoop集群,即可得到正确结果,不会报如下错误
  1. Job job_1418033228434_0012 failed with state FAILED due to: Application application_1418033228434_0012 failed 2 times due to AM Container for appattempt_1418033228434_0012_000002 exited with  exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control

    ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control

           at org.apache.hadoop.util.Shell.runCommand( Shell.java:538 )
           at org.apache.hadoop.util.Shell.run( Shell.java:455 )
           at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( Shell.java:702)
           at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195)
           at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:300)
           at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:81)
           at java.util.concurrent.FutureTask.run( FutureTask.java:262 )
           at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615)
           at java.lang.Thread.run( Thread.java:744 )


    Container exited with a non-zero exit code 1
    .Failing this attempt.. Failing the application.
    2014-12-09 13:42:05,383 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380) ) - Counters: 0

8、如果报下面的错误,请注意可能是你修改了hadoop-mapreduce-client-core-2.5.2.jar中的mapred-default.xml,比如在其中添加了mapreduce.application.classpath,请将修改还原。
  1. Job job_1418033228434_0013 failed with state FAILED due to: Application application_1418033228434_0013 failed 2 times due to AM Container for appattempt_1418033228434_0013_000002 exited with  exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1:
    ExitCodeException exitCode=1:
           at org.apache.hadoop.util.Shell.runCommand( Shell.java:538 )
           at org.apache.hadoop.util.Shell.run( Shell.java:455 )
           at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( Shell.java:702)
           at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195)
           at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:300)
           at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:81)
           at java.util.concurrent.FutureTask.run( FutureTask.java:262 )
           at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615)
           at java.lang.Thread.run( Thread.java:744 )


    Container exited with a non-zero exit code 1
    .Failing this attempt.. Failing the application.
    2014-12-09 13:45:02,949 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380) ) - Counters: 0
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值