Hadoop 编译

1.官网：

http://hadoop.apache.org/

2.Hadoop的狭义和褒义认识：

狭义：软件(HDFS、MapReduce、Yarn)

褒义：以Hadoop为主的生态圈

3.Hadoop版本

Hadoop1.x：

   HDFS：分布式文件系统，存储  

 MapReduce：分布式计算+资源调度和作业调度

Hadoop2.x

   HDFS：存储  

 MapReduce：计算  

 Yarn：YetAnother Resource Negotiate  资源调度和作业调度

4.Maven 部署官网：  

https://mvnrepository.com/Maven：  

理解为是一个仓库，用来管理jar包

如果一个源代码工程里面有pom.xml，这个就是Maven工程需要用Maven来编译

如：Maven根据pom.xml来下载jar包管理的  

cat /opt/software/hadoop-2.8.1-src

 more pom.xml

步骤：

mkdir -p /opt/sowfware  

rz 电脑上以下软件  

.m2.tar.gz  

apache-maven-3.3.9-bin.zip  

findbugs-1.3.9.zip  

hadoop-2.8.1-src.tar.gz  

hadoop-2.8.1.tar.gz  

jdk-8u45-linux-x64.gz  

protobuf-2.5.0.tar.gz

解压Maven压缩包：    

unzip apache-maven-3.3.9-bin.zip

配置环境变量：    

vi /etc/profile        

export MAVEN_HOME=/opt/software/apache-maven-3.3.9        

export MAVEN_OPTS="-Xms256m -Xmx512m"        

export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

生效：    

source /etc/profile

查看Maven是否安装：    

mvn -version

我们CD进入家目录,把.m2.tar.gz移动到家目录里解压  

 cd /opt/software    

mv .m2.tar.gz ~    

tar zxvf .m2.tar.gz

5.Hadoop编译环境准备

5.1解压hadoop-2.8.1-src.tar.gz

5.2如果一个源代码工程里有pom.xml,该工程就是Maven工程，需要用Maven来编译

5.3查看building.txt编译需求依照需求，

依次配置JDK1.8,Maven,findbugs,protocal,CMake等，并配置好环境变量。

1）JAVA安装 rz jdk-8u45-linux-x64.gz

mkdir -p /usr/java

mv jdk-8u45-linux-x64.gz /usr/java

cd /usr/java

解压jdk-8u45-linux-x64.gz压缩包

#修改用户和用户组为root

chown -R root:root jdk1.8.0_45

#编辑环境变量

vi /etc/profileexport JAVA_HOME=/usr/java/jdk1.8.0_45

export PATH=$JAVA_HOME/bin:$PATH

#生效环境变量

source /etc/profile

2）Maven安装

mkdir -p /opt/sourcecode /opt/software

cd /opt/software/

rz apache-maven-3.3.9-bin.zip

#解压apache-maven-3.3.9-bin.zip压缩包

unzip apache-maven-3.3.9-bin.zip

#编辑环境变量

vi /etc/profileexport

export MAVEN_HOME=/opt/software/apache-maven-3.3.9

export MAVEN_OPTS="-Xms256m -Xmx512m"

export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

#生效环境变量

source /etc/profile

3）protobuf安装

rz  protobuf-2.5.0.tar.gz

#解压protobuf-2.5.0.tar.gz压缩包

tar -xzvf protobuf-2.5.0.tar.gz

#进入protobuf-2.5.0文件夹

software]# cd protobuf-2.5.0

protobuf-2.5.0]# yum install -y gcc gcc-c++ make cmake  

#已经安装cmakeprotobuf-2.5.0]

protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf

protobuf-2.5.0]# make && make install

#编辑环境变量

java]# vi /etc/profile

export PROTOC_HOME=/usr/local/protobufexport

export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

#生效环境变量

protobuf-2.5.0]# source /etc/profile

4）Findbugs安装

rz findbugs-1.3.9.zip

#解压findbugs-1.3.9.zip压缩包

software]# unzip findbugs-1.3.9.zip

software]# vi /etc/profile
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9

export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
software]# source /etc/profile

5）其他依赖
yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool

yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

6.Hadoop编译

 

#开始编译

hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar

[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-dist ---

[INFO] ------------------------------------------------------------------------[INFO] Reactor Summary:[INFO]

[INFO] Apache Hadoop Main ................................. SUCCESS [  7.770 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  4.990 s]

[INFO] Apache Hadoop Project POM .......................... SUCCESS [  4.920 s]

[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 12.936 s]

[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.809 s]

[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  5.013 s]

[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 17.617 s]

[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 23.551 s]

[INFO] Apache Hadoop Auth ................................. SUCCESS [ 26.352 s]

[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 10.305 s]

[INFO] Apache Hadoop Common ............................... SUCCESS [05:58 min]

[INFO] Apache Hadoop NFS .................................. SUCCESS [ 19.105 s]

[INFO] Apache Hadoop KMS .................................. SUCCESS [ 27.479 s]

[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.174 s]

[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:32 min]

[INFO] Apache Hadoop HDFS ................................. SUCCESS [06:42 min]

[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 20.370 s]

[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 40.720 s]

[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 21.298 s]

[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 11.818 s]

[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.148 s]

[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.192 s]

[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 41.517 s]

[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:19 min]

[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.192 s]

[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 19.421 s]

[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 42.398 s]

[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  8.925 s]

[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 16.120 s]

[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 57.415 s]

[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  3.869 s]

[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 14.325 s]

[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 11.814 s]

[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 10.027 s]

[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.276 s]

[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  8.333 s]

[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  5.473 s]

[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.160 s]

[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 13.204 s]

[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  8.106 s]

[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.514 s]

[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [01:09 min]

[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 40.479 s]

[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 10.304 s]

[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 27.335 s]

[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 19.910 s]

[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 16.657 s]

[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  4.591 s]

[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 12.346 s]

[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  5.966 s][

INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  7.940 s]

[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 15.245 s]

[INFO] Apache Hadoop Archives ............................. SUCCESS [  5.380 s]

[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  5.812 s]

[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 11.785 s]

[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  9.890 s]

[INFO] Apache Hadoop Data Join ............................ SUCCESS [  5.784 s]

[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  3.254 s]

[INFO] Apache Hadoop Extras ............................... SUCCESS [  5.495 s]

[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 10.630 s]

[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 11.234 s]

[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 14.060 s]

[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 10.535 s]

[INFO] Apache Hadoop Client ............................... SUCCESS [ 13.519 s]

[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  2.164 s]

[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 10.405 s]

[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.514 s]

[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  9.201 s]

[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.129 s]

[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:07 min]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------

[INFO] Total time: 31:41 min[INFO] Finished at: 2017-12-10T11:55:28+08:00

[INFO] Final Memory: 166M/494M

[INFO] ------------------------------------------------------------------------

7.编译好的tar包

 /opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

提醒：

1、有时候编译过程中会出现下载某个包的时间太久，这是由于连接网站的过程中会出现假死，
此时按ctrl+c，重新运行编译命令。
2、如果出现缺少了某个文件的情况，则要先清理maven(使用命令 mvn clean) 再重新编译。