hadoop 3(2)，2024年最新太赞了

2301_82241698

于 2024-04-21 21:34:17 发布

阅读量816

点赞数 23

分类专栏： 2024年程序员学习文章标签： hadoop 大数据分布式

本文链接：https://blog.csdn.net/2301_82241698/article/details/138046224

版权

2024年程序员学习专栏收录该内容

64 篇文章 0 订阅

订阅专栏

先自我介绍一下，小编浙江大学毕业，去过华为、字节跳动等大厂，目前阿里P7

深知大多数程序员，想要提升技能，往往是自己摸索成长，但自己不成体系的自学效果低效又漫长，而且极易碰到天花板技术停滞不前！

因此收集整理了一份《2024年最新大数据全套学习资料》，初衷也很简单，就是希望能够帮助到想自学提升又不知道该从何学起的朋友。

既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上大数据知识点，真正体系化！

由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新

如果你需要这些资料，可以添加V获取：vip204888 （备注大数据）

正文

[INFO] Apache Hadoop MiniKDC … SUCCESS [ 1.039 s]
[INFO] Apache Hadoop Auth … SUCCESS [ 3.917 s]
[INFO] Apache Hadoop Auth Examples … SUCCESS [ 1.248 s]
[INFO] Apache Hadoop Common … SUCCESS [01:03 min]
[INFO] Apache Hadoop NFS … SUCCESS [ 3.847 s]
[INFO] Apache Hadoop KMS … SUCCESS [ 2.946 s]
[INFO] Apache Hadoop Registry … SUCCESS [ 3.659 s]
[INFO] Apache Hadoop Common Project … SUCCESS [ 0.041 s]
[INFO] Apache Hadoop HDFS Client … SUCCESS [ 16.874 s]
[INFO] Apache Hadoop HDFS … SUCCESS [ 55.545 s]
[INFO] Apache Hadoop HDFS Native Client … FAILURE [ 40.658 s]
[INFO] Apache Hadoop HttpFS … SKIPPED
[INFO] Apache Hadoop HDFS-NFS … SKIPPED
[INFO] Apache Hadoop HDFS-RBF … SKIPPED
[INFO] Apache Hadoop HDFS Project … SKIPPED
[INFO] Apache Hadoop YARN … SKIPPED
[INFO] Apache Hadoop YARN API … SKIPPED
[INFO] Apache Hadoop YARN Common … SKIPPED
[INFO] Apache Hadoop YARN Server … SKIPPED
[INFO] Apache Hadoop YARN Server Common … SKIPPED
[INFO] Apache Hadoop YARN NodeManager … SKIPPED
[INFO] Apache Hadoop YARN Web Proxy … SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService … SKIPPED
[INFO] Apache Hadoop YARN Timeline Service … SKIPPED
[INFO] Apache Hadoop YARN ResourceManager … SKIPPED
[INFO] Apache Hadoop YARN Server Tests … SKIPPED
[INFO] Apache Hadoop YARN Client … SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager … SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage … SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend … SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Common … SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Client … SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Servers … SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase tests … SKIPPED
[INFO] Apache Hadoop YARN Router … SKIPPED
[INFO] Apache Hadoop YARN TimelineService DocumentStore … SKIPPED
[INFO] Apache Hadoop YARN Applications … SKIPPED
[INFO] Apache Hadoop YARN DistributedShell … SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher … SKIPPED
[INFO] Apache Hadoop MapReduce Client … SKIPPED
[INFO] Apache Hadoop MapReduce Core … SKIPPED
[INFO] Apache Hadoop MapReduce Common … SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle … SKIPPED
[INFO] Apache Hadoop MapReduce App … SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer … SKIPPED
[INFO] Apache Hadoop MapReduce JobClient … SKIPPED
[INFO] Apache Hadoop Mini-Cluster … SKIPPED
[INFO] Apache Hadoop YARN Services … SKIPPED
[INFO] Apache Hadoop YARN Services Core … SKIPPED
[INFO] Apache Hadoop YARN Services API … SKIPPED
[INFO] Apache Hadoop YARN Application Catalog … SKIPPED
[INFO] Apache Hadoop YARN Application Catalog Webapp … SKIPPED
[INFO] Apache Hadoop YARN Application Catalog Docker Image SKIPPED
[INFO] Apache Hadoop YARN Application MaWo … SKIPPED
[INFO] Apache Hadoop YARN Application MaWo Core … SKIPPED
[INFO] Apache Hadoop YARN Site … SKIPPED
[INFO] Apache Hadoop YARN Registry … SKIPPED
[INFO] Apache Hadoop YARN UI … SKIPPED
[INFO] Apache Hadoop YARN CSI … SKIPPED
[INFO] Apache Hadoop YARN Project … SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins … SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask … SKIPPED
[INFO] Apache Hadoop MapReduce Uploader … SKIPPED
[INFO] Apache Hadoop MapReduce Examples … SKIPPED
[INFO] Apache Hadoop MapReduce … SKIPPED
[INFO] Apache Hadoop MapReduce Streaming … SKIPPED
[INFO] Apache Hadoop Distributed Copy … SKIPPED
[INFO] Apache Hadoop Client Aggregator … SKIPPED
[INFO] Apache Hadoop Dynamometer Workload Simulator … SKIPPED
[INFO] Apache Hadoop Dynamometer Cluster Simulator … SKIPPED
[INFO] Apache Hadoop Dynamometer Block Listing Generator … SKIPPED
[INFO] Apache Hadoop Dynamometer Dist … SKIPPED
[INFO] Apache Hadoop Dynamometer … SKIPPED
[INFO] Apache Hadoop Archives … SKIPPED
[INFO] Apache Hadoop Archive Logs … SKIPPED
[INFO] Apache Hadoop Rumen … SKIPPED
[INFO] Apache Hadoop Gridmix … SKIPPED
[INFO] Apache Hadoop Data Join … SKIPPED
[INFO] Apache Hadoop Extras … SKIPPED
[INFO] Apache Hadoop Pipes … SKIPPED
[INFO] Apache Hadoop OpenStack support … SKIPPED
[INFO] Apache Hadoop Amazon Web Services support … SKIPPED
[INFO] Apache Hadoop Kafka Library support … SKIPPED
[INFO] Apache Hadoop Azure support … SKIPPED
[INFO] Apache Hadoop Aliyun OSS support … SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator … SKIPPED
[INFO] Apache Hadoop Resource Estimator Service … SKIPPED
[INFO] Apache Hadoop Azure Data Lake support … SKIPPED
[INFO] Apache Hadoop Image Generation Tool … SKIPPED
[INFO] Apache Hadoop Tools Dist … SKIPPED
[INFO] Apache Hadoop Tools … SKIPPED
[INFO] Apache Hadoop Client API … SKIPPED
[INFO] Apache Hadoop Client Runtime … SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants … SKIPPED
[INFO] Apache Hadoop Client Test Minicluster … SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests … SKIPPED
[INFO] Apache Hadoop Distribution … SKIPPED
[INFO] Apache Hadoop Client Modules … SKIPPED
[INFO] Apache Hadoop Cloud Storage … SKIPPED
[INFO] Apache Hadoop Tencent COS Support … SKIPPED
[INFO] Apache Hadoop Cloud Storage Project … SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:32 min
[INFO] Finished at: 2023-04-03T10:59:30Z
[INFO] Final Memory: 227M/3280M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.3.4:cmake-compile (cmake-compile) on project hadoop-hdfs-native-client: make failed with error code 2 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-hdfs-native-client

RPM build errors:
error: Bad exit status from /var/tmp/rpm-tmp.BnnvQI (%build)
Bad exit status from /var/tmp/rpm-tmp.BnnvQI (%build)

Task :hadoop-rpm FAILED

FAILURE: Build failed with an exception.

Where:
Script ‘/ws/packages.gradle’ line: 529
What went wrong:
Execution failed for task ‘:hadoop-rpm’.

Process ‘command ‘rpmbuild’’ finished with non-zero exit value 1

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org

BUILD FAILED in 3m 38s
5 actionable tasks: 5 executed


可能是protobuf安装的版本有问题，我是直接复制的别人编译hadoop3.1.x版本的需要yum的一些依赖包，但是观察hadoop3.3.4源码包中**BUILDING.txt文件内容，他需要protocol版本3.7.1 不是2.5.0，所以还是需要卸载2.5.0的安装3.7.1版本**


![](https://img-blog.csdnimg.cn/f674659cebd44c15a50923f35a714d48.png)


 解决方法

protoc --version

libprotoc 2.5.0

which protoc

/usr/local/bin/protoc

rm /usr/local/bin/protoc

rm -rf /usr/include/google/protobuf

rm -rf /usr/local/include/google/protobuf

rm -rf /usr/lib/libproto*

rm -rf /usr/local/lib/libproto*

[root@ea94ade825a6 protobuf]# make check
Making check in .
make[1]: Entering directory /opt/protobuf' make check-local make[2]: Entering directory /opt/protobuf’
Making lib/libgmock.a lib/libgmock_main.a in gmock
/bin/sh: line 0: cd: third_party/googletest/googletest: No such file or directory
make[2]: *** [check-local] Error 1
make[2]: Leaving directory /opt/protobuf' make[1]: *** [check-am] Error 2 make[1]: Leaving directory /opt/protobuf’
make: *** [check-recursive] Error 1

[root@ea94ade825a6 protobuf]# ./configure
有如下提示

网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。

需要这份系统化的资料的朋友，可以添加V获取：vip204888 （备注大数据）