1、下载hadoop2.6,编译插件
2、Eclipse安装插件
3、安装maven插件,配置jar包,实现自动将工程打包jar
4、创建mapreduce工程,运行wordcount
5、在其中遇到各种错误
(1)java.lang.NullPointerException
修复方法:
下载hadoop-common-2.2.0-bin-master,解压将其中的winutils.exe和hadoop.dll拷贝到HADOOP_HOME/bin下
(2)Exception in thread "main" java.lang.UnsatisfiedLinkError:ConfiguredFailoverProxyProvider
修复方法:
在继承的抽象类ConfiguredFailoverProxyProvider中增加抽象方法
public abstract boolean useLogicalURI();
public abstract ProxyInfo<T> getProxy();
public abstract Class<T> getInterface();
public abstract void performFailover(T currentProxy);
public abstract void close() throws IOException;;
(3)Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method)
修复方法:在hadoop2.6源码里找到NativeCrc32.java,创建与源码一样的包名,拷贝NativeCrc32.java到该包工程目录下。
(4)org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control
修改代码
http://www.imooo.com/windows/windows/1330494.htm
YarnRunner.java的390行 (Apache Hadoop2.6的源码)
// Setup the command to run the AM List<String> vargs = new ArrayList<String>(8); vargs.add(Environment.JAVA_HOME.$() + "/bin/java");
改为
vargs.add("$JAVA_HOME/bin/java");
在YarnRunner.java类中,新增一个路径转换的方法
private void replaceEnvironment(Map<String, String> environment) { String tmpClassPath = environment.get("CLASSPATH"); tmpClassPath=tmpClassPath.replaceAll(";", ":"); tmpClassPath=tmpClassPath.replaceAll("%PWD%", "\\$PWD"); tmpClassPath=tmpClassPath.replaceAll("%HADOOP_MAPRED_HOME%", "\\$HADOOP_MAPRED_HOME"); tmpClassPath= tmpClassPath.replaceAll("\\\\", "/" ); environment.put("CLASSPATH",tmpClassPath); }
在YarnRunner.java的在466行添加:
replaceEnvironment(environment);
(5)Application application_1420601118686_0001 failed 2 times due to AM Container for appattempt_1420601118686_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://master:8088/proxy/application_1420601118686_0001/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1420601118686_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
添加路径:
Mapreduce-site.xml
<property>
<name>mapreduce.application.classpath</name>
<value>
/opt/app/hadoop-2.6.0/etc/hadoop,
/opt/app/hadoop-2.6.0/share/hadoop/common/*,
/opt/app/hadoop-2.6.0/share/hadoop/common/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/lib/*
</value>
</property>
Yarn-site.xml
<property>
<name>yarn.application.classpath</name>
<value>
/opt/app/hadoop-2.6.0/etc/hadoop,
/opt/app/hadoop-2.6.0/share/hadoop/common/*,
/opt/app/hadoop-2.6.0/share/hadoop/common/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/lib/*
</value>
</property>