【Hadoop】hadoop2.6.4 源码编译

编译安装Hadoop

问题

Apache官网上提供的Hadoop本地库是32位的，如果我们的Linux服务器是64位的话，就会现问题。

我们在64位服务器执行Hadoop命令时，则会报以下错误：

我们在64位服务器执行Hadoop命令时，则会报以下错误：
WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-Java classes where applicable

安装必要的软件

yum install gcc 
yum install cmake 
yum install gcc-c++
yum install autoconf automake libtool
yum install openssl-devel

安装maven
http://www-eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

tar -zxvf apache-maven-3.3.9-bin.tar.gz  //解压

vim /etc/profile //设置环境变量
#set maven path 
export MAVEN_HOME=/home/hadoop/software/maven
export PATH=$PATH:$MAVEN_HOME/bin

source /etc/profile //使profile文件立即生效

[root@Master software]# mvn -version  //出现一下内容安装成功
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /home/hadoop/software/maven
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_92/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-279.el6.x86_64", arch: "amd64", family: "unix"

安装protobuf
要求版本protobuf-2.5.0 (必须为2.5.0以上版本)

https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz

tar -zxvf protobuf-2.5.0.tar.gz

进入protobuf所在目录，然后按下面顺序操作
./configure
make 
make check
sudo make install

vim /etc/profile

#set protobuf path
export PROTOBUF_HOME=/home/hadoop/software/protobuf
export PATH=$PATH:$PROTOBUF_HOME/bin

source /etc/profile

安装ant

http://www-eu.apache.org/dist//ant/binaries/apache-ant-1.9.7-bin.tar.gz

tar -zxvf apache-ant-1.9.7-bin.tar.gz

vim /etc/profile

#set ant path
export ANT_HOME=/home/hadoop/software/ant
export PATH=$PATH:$ANT_HOME/bin

source /etc/profile

安装findbugs

http://findbugs.sourceforge.net/downloads.html

tar -zxvf findbugs-3.0.1.tar.gz

vim /etc/profile

#set findbugs path
export FINDBUGS_HOME=/home/hadoop/software/findbugs
export PATH=$PATH:$FINDBUGS_HOME/bin

source /etc/profile

编译hadoop

http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.4/hadoop-2.6.4-src.tar.gz

tar zxf hadoop-2.6.4-src.tar.gz
cd hadoop-2.6.4-src/

编辑源代码根目录下的pom.xml,找到maven-enforcer-plugin所在的部分

<plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-enforcer-plugin</artifactId>
      <version>1.3.1</version>
      <configuration>
            <rules>
               <requireMavenVersion>
                  <version>[3.0.2,)</version>
               </requireMavenVersion>
               <requireJavaVersion>
                  <version>1.7</version>
               </requireJavaVersion>
            </rules>
       </configuration>
 </plugin>

原来的值是1.6,即规定了使用jdk1.6,如果本机安装的jdk是1.7,这里相应改一下

more BUILDING.txt   //查看如何编译安装

mvn clean package -Pdist,native -DskipTests -Dtar //运行开始编译

总结

编译过程中，需要下载很多包，等待时间比较长。当看到hadoop各个项目都编译成功,即出现一系列的SUCCESS之后，即为编译成功。

编译后的项目在hadoop-2.6.0-src/hadoop-dist/target/hadoop-2.6.0中。

只需要将编译后项目中的/hadoop-dist/lib/native文件夹覆盖至之前32位的项目中即可（当然整个项目覆盖也可以）避免出现问题将整个项目覆盖掉吧，正好多安装几次也就熟练了；

编译过程中需要下载安装包，有时候可能由于网络的原因，导致安装包下载不完整，而出现编译错误

可能出现错误

错误1：

Remote host closed connection during handshake: SSL peer shut down incorrectly.......

解决方案:需要重新新多编译几次即可通过。

错误2：

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while creating archive: 
[ERROR] Exit code: 1 - /home/hadoop/hadoop-2.6.4-src/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java:27: 错误: 意外的结束标记: 
[ERROR] * 
[ERROR] ^ 
[ERROR] 
[ERROR] Command line was: /opt/jdk1.8.0_40/jre/../bin/javadoc @options @packages 
[ERROR] 
[ERROR] Refer to the generated Javadoc files in ‘/home/hadoop/hadoop-2.6.4-src/hadoop-common-project/hadoop-annotations/target’ dir. 
[ERROR] -> [Help 1] 
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 
[ERROR] Re-run Maven using the -X switch to enable full debug logging. 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles: 
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException 
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command 
[ERROR] mvn -rf :hadoop-annotations

更换JDK1.7版本，我第一次编译就是这个错误，用的jdk1.8换成1.7后成功

以上仅仅是个人出现的错误。