hadoop 3.3.4 编译报错记录

一，编译环境

centos7.6

在docker内编译，可能是centos的原因或者docker版本的原因，我用centos7.9安装docker23.x版本就可以完成编译。现在centos7.6，docker19.x版本会缺少一些依赖包不能一次编译过

hadoop 3.3.4 注意BUILDING.txt依赖环境的版本

二，报错信息

1.cmake-compile (cmake-compile) on project hadoop-hdfs-native-client: make failed with error

[WARNING]    #pragma message("DISABLE_CONCURRENT_WORKERS is defined so hdfs_ioservice_test will compile out the InitDefaultThreads test")
[WARNING]                                                                                                                               ^
[WARNING] /usr/bin/ld: /usr/local/lib/libprotobuf.a(common.o): relocation R_X86_64_32S against `_ZTVN6google8protobuf7ClosureE' can not be used when making a shared object; recompile with -fPIC
[WARNING] /usr/local/lib/libprotobuf.a: error adding symbols: Bad value
[WARNING] collect2: error: ld returned 1 exit status
[WARNING] make[2]: *** [native/target/usr/local/lib/libhdfspp.so.0.1.0] Error 1
[WARNING] make[1]: *** [main/native/libhdfspp/CMakeFiles/hdfspp.dir/all] Error 2
[WARNING] make[1]: *** Waiting for unfinished jobs....
[WARNING] make: *** [all] Error 2
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [ 11.575 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.454 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.854 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.520 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.082 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  0.907 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.316 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.039 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  3.917 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  1.248 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:03 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.847 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  2.946 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [  3.659 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.041 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 16.874 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 55.545 s]
[INFO] Apache Hadoop HDFS Native Client ................... FAILURE [ 40.658 s]
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS-RBF ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] Apache Hadoop YARN ................................. SKIPPED
[INFO] Apache Hadoop YARN API ............................. SKIPPED
[INFO] Apache Hadoop YARN Common .......................... SKIPPED
[INFO] Apache Hadoop YARN Server .......................... SKIPPED
[INFO] Apache Hadoop YARN Server Common ................... SKIPPED
[INFO] Apache Hadoop YARN NodeManager ..................... SKIPPED
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop YARN Services ........................ SKIPPED
[INFO] Apache Hadoop YARN Services Core ................... SKIPPED
[INFO] Apache Hadoop YARN Services API .................... SKIPPED
[INFO] Apache Hadoop YARN Application Catalog ............. SKIPPED
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SKIPPED
[INFO] Apache Hadoop YARN Application Catalog Docker Image  SKIPPED
[INFO] Apache Hadoop YARN Application MaWo ................ SKIPPED
[INFO] Apache Hadoop YARN Application MaWo Core ........... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN Registry ........................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN CSI ............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Uploader ................... SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SKIPPED
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SKIPPED
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SKIPPED
[INFO] Apache Hadoop Dynamometer Dist ..................... SKIPPED
[INFO] Apache Hadoop Dynamometer .......................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Resource Estimator Service ........... SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Image Generation Tool ................ SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Tencent COS Support .................. SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:32 min
[INFO] Finished at: 2023-04-03T10:59:30Z
[INFO] Final Memory: 227M/3280M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.3.4:cmake-compile (cmake-compile) on project hadoop-hdfs-native-client: make failed with error code 2 -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-hdfs-native-client


RPM build errors:
error: Bad exit status from /var/tmp/rpm-tmp.BnnvQI (%build)
    Bad exit status from /var/tmp/rpm-tmp.BnnvQI (%build)

> Task :hadoop-rpm FAILED

FAILURE: Build failed with an exception.

* Where:
Script '/ws/packages.gradle' line: 529

* What went wrong:
Execution failed for task ':hadoop-rpm'.
> Process 'command 'rpmbuild'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 3m 38s
5 actionable tasks: 5 executed

可能是protobuf安装的版本有问题，我是直接复制的别人编译hadoop3.1.x版本的需要yum的一些依赖包，但是观察hadoop3.3.4源码包中BUILDING.txt文件内容，他需要protocol版本3.7.1 不是2.5.0，所以还是需要卸载2.5.0的安装3.7.1版本

 解决方法

# protoc --version
libprotoc 2.5.0

# which protoc
/usr/local/bin/protoc
# rm /usr/local/bin/protoc
# rm -rf /usr/include/google/protobuf
# rm -rf /usr/local/include/google/protobuf
# rm -rf /usr/lib/libproto*
# rm -rf /usr/local/lib/libproto*

[root@ea94ade825a6 protobuf]# make check
Making check in .
make[1]: Entering directory `/opt/protobuf'
make  check-local
make[2]: Entering directory `/opt/protobuf'
Making lib/libgmock.a lib/libgmock_main.a in gmock
/bin/sh: line 0: cd: third_party/googletest/googletest: No such file or directory
make[2]: *** [check-local] Error 1
make[2]: Leaving directory `/opt/protobuf'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `/opt/protobuf'
make: *** [check-recursive] Error 1

[root@ea94ade825a6 protobuf]# ./configure
有如下提示
=== configuring in third_party/googletest (/opt/protobuf/third_party/googletest)
configure: WARNING: no configuration information is in third_party/googletest

缺少googletest的依赖，有梯子应该就直接从google下载了，但是这里没有梯子就手动从github下载

 googletest对应的版本是个未发布的中间分支

wget https://github.com/google/googletest/archive/c3bb0ee2a63279a803aaad956b9b26d74bf9e6e2.zip

解压把内容复制到 googletest 目录

 重新执行安装

# ./autogen.sh
#./configure
# make
# make check
# sudo make install

 make check 通过

安装成功 

[root@ea94ade825a6 protobuf]# protoc --version
libprotoc 3.7.1
[root@ea94ade825a6 protobuf]# 

参考连接：protobuf的安装与卸载_protobuf安装与卸载 ubuntu_tianminggenie的博客-CSDN博客

2，bower (bower install) on project hadoop-yarn-ui: Failed to run task: 'bower install --allow-root' failed

[INFO] bower ember-load-initializers#0.1.7        ECMDERR Failed to execute "git ls-remote --tags --heads https://github.com/ember-cli/ember-load-initializers.git", exit code of #128 fatal: unable to access 'https://github.com/ember-cli/ember-load-initializers.git/': Encountered end of file
[INFO] 
[INFO] Additional error details:
[INFO] fatal: unable to access 'https://github.com/ember-cli/ember-load-initializers.git/': Encountered end of file
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:

网络慢造成的

进入到源码根目录，从失败的地方开始编译，让把由于网络问题的包再下载一次

mvn compile -rf :hadoop-yarn-ui