win7上eclipse提交mr至hadoop2.6分布式运行错误集锦

最新推荐文章于 2021-11-15 11:20:37 发布

weipanp

最新推荐文章于 2021-11-15 11:20:37 发布

阅读量5.2k

点赞数

分类专栏： hadoop 文章标签： hadoop集群 mapreduce HA

本文链接：https://blog.csdn.net/weipanp/article/details/42713121

版权

hadoop 专栏收录该内容

8 篇文章 0 订阅

订阅专栏

1、下载hadoop2.6，编译插件

2、Eclipse安装插件

3、安装maven插件，配置jar包，实现自动将工程打包jar

4、创建mapreduce工程，运行wordcount

5、在其中遇到各种错误

(1)java.lang.NullPointerException

修复方法：

下载hadoop-common-2.2.0-bin-master，解压将其中的winutils.exe和hadoop.dll拷贝到HADOOP_HOME/bin下

(2)Exception in thread "main" java.lang.UnsatisfiedLinkError:ConfiguredFailoverProxyProvider

修复方法：

在继承的抽象类ConfiguredFailoverProxyProvider中增加抽象方法

public abstract boolean useLogicalURI();

public abstract ProxyInfo<T> getProxy();

public abstract Class<T> getInterface();

public abstract void performFailover(T currentProxy);

public abstract void close() throws IOException;；

(3)Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V

at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method)

修复方法：在hadoop2.6源码里找到NativeCrc32.java，创建与源码一样的包名，拷贝NativeCrc32.java到该包工程目录下。

(4)org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control

修改代码

http://www.imooo.com/windows/windows/1330494.htm

YarnRunner.java的390行（Apache Hadoop2.6的源码）

// Setup the command to run the AM List<String> vargs = new ArrayList<String>(8); vargs.add(Environment.JAVA_HOME.$() + "/bin/java");

改为

vargs.add("$JAVA_HOME/bin/java");

在YarnRunner.java类中，新增一个路径转换的方法

private void replaceEnvironment(Map<String, String> environment) { String tmpClassPath = environment.get("CLASSPATH"); tmpClassPath=tmpClassPath.replaceAll(";", ":"); tmpClassPath=tmpClassPath.replaceAll("%PWD%", "\\$PWD"); tmpClassPath=tmpClassPath.replaceAll("%HADOOP_MAPRED_HOME%", "\\$HADOOP_MAPRED_HOME"); tmpClassPath= tmpClassPath.replaceAll("\\\\", "/" ); environment.put("CLASSPATH",tmpClassPath); }

在YarnRunner.java的在466行添加：

replaceEnvironment(environment);

(5)Application application_1420601118686_0001 failed 2 times due to AM Container for appattempt_1420601118686_0001_000002 exited with exitCode: 1

For more detailed output, check application tracking page:http://master:8088/proxy/application_1420601118686_0001/Then, click on links to logs of each attempt.

Diagnostics: Exception from container-launch.

Container id: container_1420601118686_0001_02_000001

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

添加路径：

Mapreduce-site.xml

<name>mapreduce.application.classpath</name>

<value>

/opt/app/hadoop-2.6.0/etc/hadoop,

/opt/app/hadoop-2.6.0/share/hadoop/common/*,

/opt/app/hadoop-2.6.0/share/hadoop/common/lib/*,

/opt/app/hadoop-2.6.0/share/hadoop/hdfs/*,

/opt/app/hadoop-2.6.0/share/hadoop/hdfs/lib/*,

/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/*,

/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,

/opt/app/hadoop-2.6.0/share/hadoop/yarn/*,

/opt/app/hadoop-2.6.0/share/hadoop/yarn/lib/*

</value>

</property>

Yarn-site.xml

<name>yarn.application.classpath</name>