Hadoop编译安装

1.准备好对应的软件

---

1.1 Hadoop-2.6.0-cdh5.7.0-src.tar.gz
1.2 jdk-7u80-linux-x64.tar.gz 必须是1.7版本的jdk，1.8的一编译就报错
1.3 apache-maven-3.3.9-bin.tar.gz 最好是3.0及以上的版本
1.4 protobuf-2.5.0.tar.gz

2.编译hadoop

首先将上述软件下载到服务器上

2.1 安装依赖

1[root@hadoop001?~]#?yum?install?-y?gcc?gcc-c++?make?cmake?svn?ncurses-devel
2[root@hadoop001?~]#?yum?install?-y?openssl?openssl-devel?svn?ncurses-devel?zlib-devel?libtool
3[root@hadoop001?~]#?yum?install?-y?snappy?snappy-devel?bzip2?bzip2-devel?lzo?lzo-devel?lzop?autoconf?automake?

2.2安装jdk

 1[root@hadoop001?~]#?mkdir?/usr/java/
 2[root@hadoop001?~]#?cd??/usr/java/
 3上传或者下载jdk文件，解压jdk压缩包的时候要注意所属组跟所属用户的相关权限
 4配置/etc/profile环境变量
 5export?JAVA_HOME=/usr/java/jdk1.7.0_80
 6export?PATH=$JAVA_HOME/bin/:$PATH
 7[root@hadoop001?java]#?source?/etc/profile
 8[root@hadoop001?jdk1.7.0_80]#?java?-version?
 9java?version?"1.7.0_80"?
10Java(TM)?SE?Runtime?Environment?(build?1.7.0_80-b15)?
11Java?HotSpot(TM)?64-Bit?Server?VM?(build?24.80-b11,?mixed?mode)

2.3安装配置maven

 1[root@hadoop001?java]#?su?-?hadoop
 2[hadoop@hadoop001?~]$?mkdir?soft?maven_repo?source
 3[hadoop@hadoop001?~]$?cd?soft/
 4[hadoop@hadoop001?~]$?tar?-zxvf?~/soft/apache-maven-3.3.9-bin.tar.gz?-C?~/app/
 5[hadoop@hadoop001?~]$?vi?~/.bash_profile
 6#添加或修改如下内容，注意MAVEN_OPTS设置了maven运行的内存，防止内存太小导致编译失败
 7export?MAVEN_HOME=/home/hadoop/app/apache-maven-3.3.9
 8export?MAVEN_OPTS="-Xms1024m?-Xmx1024m"
 9export?PATH=$MAVEN_HOME/bin:$PATH
10[hadoop@hadoop001?~]$?source?~/.bash_profile?
11[hadoop@hadoop001?~]$?which?mvn
12~/app/apache-maven-3.3.9/bin/mvn
13[hadoop@hadoop001?~]$?
14编辑maven配置文件，编辑的时候千万要小心
15/home/gzf/app/apache-maven-3.3.9/conf/settings.xml
16????<mirror>
17??????<id>nexus-aliyun</id>
18??????<mirrorOf>central</mirrorOf>
19??????<name>Nexus?aliyun</name>
20??????<url>http://maven.aliyun.com/nexus/content/groups/public</url>
21????</mirror>
22说明：修改settings.xml一定要注意，修改格式错误，一开始编译就会报错，而且有些maven的库文件阿里云的镜像是没有的，可以在https://repository.cloudera.com/artifactory/cloudera-repos/这里找到对应的文件

在这里插入图片描述

2.4安装protobuf

 1[hadoop@hadoop001?soft]$?tar?-zxvf?protobuf-2.5.0.tar.gz?-C?~/app/
 2[hadoop@hadoop001?soft]$?cd?~/app/protobuf-2.5.0
 3[hadoop@hadoop001?protobuf-2.5.0]$?./configure??--prefix=/home/hadoop/app/protobuf-2.5.0
 4[hadoop@hadoop001?protobuf-2.5.0]$?make
 5[hadoop@hadoop001?protobuf-2.5.0]$?make?install
 6[hadoop@hadoop001?protobuf-2.5.0]$?vi?~/.bash_profile?
 7export?PROTOBUF_HOME=/home/hadoop/app/protobuf-2.5.0
 8export?PATH=$PROTOBUF_HOME/bin:$PATH
 9[hadoop@hadoop001?protobuf-2.5.0]$?source?~/.bash_profile?
10[hadoop@hadoop001?protobuf-2.5.0]$?protoc?--version
11libprotoc?2.5.0

2.5编译hadoop

1[hadoop@hadoop001?~]$?tar?-zxvf?~/soft/hadoop-2.6.0-cdh5.7.0-src.tar.gz?-C?~/source/
2[hadoop@hadoop001?hadoop-2.6.0-cdh5.7.0]$?mvn?clean?package?-Pdist,native?-DskipTests?-Dtar
3-Pdist,native ??：重新编译hadoop动态库
4-DskipTests ????：跳过测试
5-Dtar ??????????：打成tar包

开始编译hadoop，时间会比较久，请耐心等待。

3.编译碰到的问题

3.1问题一

在这里插入图片描述

默认maven库的位置下载在~/.m2/repository，具体文件的具体位置可以看下载网址的路径.（上图有一个案例：下载的文件网址）
3.2问题二
下面这个报错是没有下载到文件，导致无法进行下一步，去上面的路径找对应的下载地址进行下载

1Failed?to?execute?goal?on?project?hadoop-auth:?Could?not?resolve?dependencies?for?project?org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.7.0:?Failed?to?collect?dependencies?at?org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:?Failed?to?read?artifact?descriptor?for?org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:?Could?not?transfer?artifact?org.mortbay.jetty:jetty-util:pom:6.1.26.cloudera.4?from/to?cdh.repo?(https://repository.cloudera.com/artifactory/cloudera-repos):?Remote?host?closed?connection?during?handshake:?SSL?peer?shut?down?incorrectly

3.3 问题三
未找到apache-tomcat-6.0.44.tar.gz，可以使用香港的服务器进行编译，也可以直接下载好tomcat，上传到对应的路径，以下是hadoop-2.6.0-cdh5.7.0编译对应的tomcat存放路径：

1/home/gzf/source/hadoop-2.6.0-cdh5.7.0/hadoop-common-project/hadoop-kms/downloads
2/home/gzf/source/hadoop-2.6.0-cdh5.7.0/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads/

编译成功

 1#有?BUILD?SUCCESS?信息则表示编译成功
 2[INFO]?Apache?Hadoop?Scheduler?Load?Simulator?.............?SUCCESS?[?13.592?s]
 3[INFO]?Apache?Hadoop?Tools?Dist?...........................?SUCCESS?[?12.042?s]
 4[INFO]?Apache?Hadoop?Tools?................................?SUCCESS?[??0.094?s]
 5[INFO]?Apache?Hadoop?Distribution?.........................?SUCCESS?[01:49?min]
 6[INFO]?------------------------------------------------------------------------
 7[INFO]?BUILD?SUCCESS????出现这个SUCCESS,表示编译成功
 8[INFO]?------------------------------------------------------------------------
 9[INFO]?Total?time:?37:39?min
10[INFO]?Finished?at:?2019-04-07T16:48:42+08:00
11[INFO]?Final?Memory:?200M/989M
12[INFO]?------------------------------------------------------------------------

编译成功对应的tar包位置

 1[hadoop@hadoop001?repository]$?cd?~/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target/
 2[hadoop@hadoop001?target]$?ll
 3total?563948
 4drwxrwxr-x?2?hadoop?hadoop??4096?Jul?10?15:33?antrun
 5drwxrwxr-x?3?hadoop?hadoop??4096?Jul?10?15:33?classes
 6-rw-rw-r--?1?hadoop?hadoop??1995?Jul?10?15:33?dist-layout-stitching.sh
 7-rw-rw-r--?1?hadoop?hadoop??687?Jul?10?15:33?dist-tar-stitching.sh
 8drwxrwxr-x?9?hadoop?hadoop??4096?Jul?10?15:33?hadoop-2.6.0-cdh5.7.0
 9-rw-rw-r--?1?hadoop?hadoop??191842644?Jul?10?15:33?hadoop-2.6.0-cdh5.7.0.tar.gz
10-rw-rw-r--?1?hadoop?hadoop??7313?Jul?10?15:33?hadoop-dist-2.6.0-cdh5.7.0.jar
11-rw-rw-r--?1?hadoop?hadoop??385572232?Jul?10?15:33?hadoop-dist-2.6.0-cdh5.7.0-javadoc.jar
12-rw-rw-r--?1?hadoop?hadoop??4852?Jul?10?15:33?hadoop-dist-2.6.0-cdh5.7.0-sources.jar
13-rw-rw-r--?1?hadoop?hadoop??4852?Jul?10?15:33?hadoop-dist-2.6.0-cdh5.7.0-test-sources.jar
14drwxrwxr-x?2?hadoop?hadoop??4096?Jul?10?15:33?javadoc-bundle-options
15drwxrwxr-x?2?hadoop?hadoop??4096?Jul?10?15:33?maven-archiver
16drwxrwxr-x?3?hadoop?hadoop??4096?Jul?10?15:33?maven-shared-archive-resources
17drwxrwxr-x?3?hadoop?hadoop??4096?Jul?10?15:33?test-classes
18drwxrwxr-x?2?hadoop?hadoop??4096?Jul?10?15:33?test-dir

查看编译之后是否支持各种格式的文件

 1[hadoop@hadoop001?hadoop-2.6.0-cdh5.7.0]$?bin/hadoop?checknative
 219/07/10?18:01:51?INFO?bzip2.Bzip2Factory:?Successfully?loaded?&?initialized?native-bzip2?library?system-native
 319/07/10?18:01:51?INFO?zlib.ZlibFactory:?Successfully?loaded?&?initialized?native-zlib?library
 4Native?library?checking:
 5hadoop:??true?/home/gzf/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so.1.0.0
 6zlib:????true?/lib64/libz.so.1
 7snappy:??true?/usr/lib64/libsnappy.so.1
 8lz4:?????true?revision:99
 9bzip2:???true?/lib64/libbz2.so.1
10openssl:?true?/usr/lib64/libcrypto.so