Hadoop-3.3.6 编译

已于 2024-01-27 21:19:15 修改
阅读量600 收藏 7
点赞数 7
文章标签： hadoop 大数据 分布式
于 2024-01-13 12:27:00 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_29312605/article/details/135568340
版权

系统及版本

宿主环境mac + docker

Hadoop 版本 3.3.6

系统：Anolis OS 8.8（亲测效果和遇到的问题和centos8 一模一样，centos8不维护了，而且官方yum源下线了，还需要更改yum源）

jdk版本：jdk1.8.0_391 （Java Downloads | Oracle）

maven版本：3.9.6

cmake版本：3.27.1

Protocol Buffers 版本：3.7.1

snappy 版本：1.1.3

isa-l版本：2.30.0'

pmdk版本: 2.0.0

‼️ 有些库安装在mac上需要本地网络能访问外网

环境搭建

Docker准备

可以不用 Docker 直接购买 ECS 也可以（没有选择ECS主要是8G的ECS比较贵，亲测在2和2GECS上执行如下步骤，直到maven编译oom，就是说系统固定遇到问题也是同样的）

# 拉取镜像
docker pull anolisos
# 创建docker 网络, 让容器能连接网络
docker network create my-net
# 启动容器
docker run --name hadoop2 -d --privileged --network=my-net openanolis/anolisos /usr/sbin/init

进入容器

docker exec -it 6526b7415827 /bin/bash


安装依赖库

yum install wget libtirpc-devel zlib-devel lz4-devel bzip2-devel openssl-devel cyrus-sasl-devel  lzo-devel zlib-devel gcc gcc-c++ openssh-server openssh-clients

yum install make automake gcc gcc-c++ kernel-devel

yum install patch

不安装path后面编译会报错

安装cmake

wget https://github.com/Kitware/CMake/archive/refs/tags/v3.27.1.tar.gz

tar -xvf v3.27.1.tar.gz

rm -rf v3.27.1.tar.gz

cd CMake-3.27.1

./bootstrap

./configure 

gmake

gmake install


安装 Protocol Buffers

yum -y install autoconf automake libtool

wget https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.7.1.tar.gz

tar -xvf v3.7.1.tar.gz

rm -rf v3.7.1.tar.gz

cd protobuf-3.7.1/

./autogen.sh

./configure

make 

make install


安装snappy

wget https://codeload.github.com/google/snappy/tar.gz/refs/tags/1.1.3

tar -xvf 1.1.3

rm -rf 1.1.3

cd snappy-1.1.3/

./autogen.sh

./configure

make 

make install

yum install -y snappy-devel

安装isa-l

yum install -y epel-release

yum install -y libzstd-devel

yum install -y yasm

yum install -y nasm

wget https://github.com/intel/isa-l/archive/refs/tags/v2.30.0.tar.gz

tar -xvf v2.30.0.tar.gz

rm -rf v2.30.0.tar.gz

cd isa-l-2.30.0/

./autogen.sh

./configure

make

make install

安装pmdk

yum install ndctl-devel  daxctl-devel libaio-devel

yum install pandoc

yum install -y bash-completion*

yum install -y json-c*

yum install -y libudev*

yum install -y ndctl*

yum install -y autoconf* 

yum install libtool*

wget https://github.com/pmem/pmdk/archive/refs/tags/2.0.0.tar.gz

tar -xvf 2.0.0.tar.gz

rm -rf 2.0.0.tar.gz

cd pmdk-2.0.0/

make -j

make install



 

配置 java 相关

安装jdk

这个jdk下载地址或过期

在电脑浏览器访问如下官网：Java Downloads | Oracle

点击下载

复制下载地址到服务器执行wget 

wget https://download.oracle.com/otn/java/jdk/8u202-b08/1961070e4c9b4e26a04e7f5a083f551e/jdk-8u202-linux-x64.tar.gz

tar -xvf 下载的文件

下载maven

wget https://dlcdn.apache.org/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz

tar -xvf apache-maven-3.9.6-bin.tar.gz
rm -rf apache-maven-3.9.6-bin.tar.gz

修改/etc/profile

vim /etc/profile
# 在最后面增加
export JAVA_HOME=/opt/jdk1.8.0_391
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$JAVA_HOME/bin:$PATH

export MAVEN_HOME=/opt/apache-maven-3.9.6
export PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin

source /etc/profile

修改maven镜像

cd apache-maven-3.9.6/conf/

vi settings.xml

# 删除原来的镜像保护
<mirror>
  <id>maven-default-http-blocker</id>
  <mirrorOf>external:http:*</mirrorOf>
  <name>Pseudo repository to mirror external repositories initially using HTTP.</name>
  <url>http://0.0.0.0/</url>
  <blocked>true</blocked>
</mirror>

# 添加新镜像
<mirror>
  <id>alimaven-central</id>
  <mirrorOf>central</mirrorOf>
  <name>aliyun maven</name>
  <url>https://maven.aliyun.com/nexus/content/repositories/central/</url>
</mirror>
<mirror>
  <id>alimaven</id>
  <name>aliyun maven</name>
  <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
  <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
  <id>jboss-public-repository-group</id>
  <mirrorOf>central</mirrorOf>
  <name>JBoss Public Repository Group</name>
  <url>http://repository.jboss.org/nexus/content/groups/public</url>
</mirror>

编译 hadoop 源码

下载源码

wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6-src.tar.gz

tar -xvf hadoop-3.3.6-src.tar.gz & rm -rf hadoop-3.3.6-src.tar.gz

cd hadoop-3.3.6-src/

maven 编译

mvn -X package -Pdist,native,docs -DskipTests -Dtar -Dmaven.skip.test=true -Dmaven.javadoc.skip=true

报错1

[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.11.2:yarn (yarn install) on project hadoop-yarn-applications-catalog-webapp: Failed to run task: 'yarn ' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
  org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.11.2:yarn (yarn install) on project hadoop-yarn-applications-catalog-webapp: Failed to run task
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:333)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
    Caused by: org.apache.maven.plugin.MojoFailureException: Failed to run task
      at com.github.eirslett.maven.plugins.frontend.mojo.AbstractFrontendMojo.execute (AbstractFrontendMojo.java:100)
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
      at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
      at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
      at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
      at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
      at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
      at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
      at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
      at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
      at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke (Method.java:498)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
      at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
      at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
      Caused by: com.github.eirslett.maven.plugins.frontend.lib.TaskRunnerException: 'yarn ' failed.
      at com.github.eirslett.maven.plugins.frontend.lib.YarnTaskExecutor.execute (YarnTaskExecutor.java:64)
      at com.github.eirslett.maven.plugins.frontend.mojo.YarnMojo.execute (YarnMojo.java:65)
      at com.github.eirslett.maven.plugins.frontend.mojo.AbstractFrontendMojo.execute (AbstractFrontendMojo.java:94)
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
      at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
      at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
      at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
      at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
      at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
      at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
      at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
      at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
      at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke (Method.java:498)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
      at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
      at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
      Caused by: com.github.eirslett.maven.plugins.frontend.lib.ProcessExecutionException: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
      at com.github.eirslett.maven.plugins.frontend.lib.ProcessExecutor.execute (ProcessExecutor.java:82)
      at com.github.eirslett.maven.plugins.frontend.lib.ProcessExecutor.executeAndRedirectOutput (ProcessExecutor.java:64)
      at com.github.eirslett.maven.plugins.frontend.lib.YarnExecutor.executeAndRedirectOutput (YarnExecutor.java:28)
      at com.github.eirslett.maven.plugins.frontend.lib.YarnTaskExecutor.execute (YarnTaskExecutor.java:58)
      at com.github.eirslett.maven.plugins.frontend.mojo.YarnMojo.execute (YarnMojo.java:65)
      at com.github.eirslett.maven.plugins.frontend.mojo.AbstractFrontendMojo.execute (AbstractFrontendMojo.java:94)
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
      at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
      at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
      at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
      at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
      at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
      at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
      at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
      at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
      at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke (Method.java:498)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
      at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
      at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
      Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
      at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
      at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
      at com.github.eirslett.maven.plugins.frontend.lib.ProcessExecutor.execute (ProcessExecutor.java:74)
      at com.github.eirslett.maven.plugins.frontend.lib.ProcessExecutor.executeAndRedirectOutput (ProcessExecutor.java:64)
      at com.github.eirslett.maven.plugins.frontend.lib.YarnExecutor.executeAndRedirectOutput (YarnExecutor.java:28)
      at com.github.eirslett.maven.plugins.frontend.lib.YarnTaskExecutor.execute (YarnTaskExecutor.java:58)
      at com.github.eirslett.maven.plugins.frontend.mojo.YarnMojo.execute (YarnMojo.java:65)
      at com.github.eirslett.maven.plugins.frontend.mojo.AbstractFrontendMojo.execute (AbstractFrontendMojo.java:94)
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
      at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
      at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
      at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
      at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
      at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
      at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
      at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
      at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
      at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke (Method.java:498)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
      at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
      at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

修改hadoop-project/pom.xml

<properties>
...
<nodejs.version>v14.21.3</nodejs.version>
<yarnpkg.version>v1.22.19</yarnpkg.version>
...
</properties>


报错2

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
  org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:333)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
    Caused by: org.apache.maven.plugin.MojoExecutionException: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.
      at org.apache.maven.plugins.enforcer.EnforceMojo.execute (EnforceMojo.java:243)
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
      at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
      at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
      at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
      at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
      at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
      at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
      at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
      at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
      at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke (Method.java:498)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
      at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
      at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

修改文件

/opt/hadoop-3.3.6-src/hadoop-client-modules/pom.xml

注释掉

<!--<module>hadoop-client-check-test-invariants</module>-->

编译后文件目录

/opt/hadoop-3.3.6-src/hadoop-dist/target

山.那.边
关注
  • 7
    点赞
  • 踩
  • 7
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Hadoop2.6.2 编译
技术大神之路
04-09 656
1.Hadoop2.0 之后依赖 protobuf 实现RPC，因此要先安装依赖在hadoop-common-project 下 maven package -DskipTests 编译时会发现缺少依赖包protoc，并提示依赖的版本，可以修改hadoop-common-project 下 pom.xml 文件里边具体的版本（你本机安装的版本），以通过编译。 找到定义处的文件的方法： 在ha
编译Hadoop3.3.6编译
weixin_43838444的博客
08-29 336
查看BUILDING.txt，阅读Requirements 确定依赖版本根据Building on CentOS 8 提示安装。安装C环境 根据要求去githup上下载CMake 3.1。安装 Protocol Buffers v3.7.1。安装JAVA环境 根据本文安装jdk来安装。安装Maven编译环境 根据本文安装来安装。在win上解压打开一个Hadoop源码。
mac-hadoop3.3.6 源码构建以及踩坑记录
寂夜了无痕的博客
01-20 1633
mac-hadoop3.3.6 源码构建以及踩坑记录
Hadoop纠删码算法启用ISA-L加速库
sinat_28603977的博客
04-15 1986
启用ISA-L步骤如下： 安装yasm和nasm 在Hadoop集群所有节点上安装yasm和nasm。 如果没有yum, 可以直接下载对应的源码，使用make进行安装 nasm 对应的下载地址https://www.nasm.us/pub/nasm/releasebuilds/2.14.03rc2/linux/下载对应的rpm可以直接使用 yum localinstall 安装 yas...
Hadoop生态圈（七）- HDFS优化方案
程序园@大Null
01-17 2011
短路本地读取：Short Circuit Local Reads，makeHDFS Block负载平衡器：Balancer，磁盘均衡器：HDFS Disk Balancer，纠删码技术：Erasure Coding，Reed-Solomon（RS）码，Hadoop EC架构，Erasure Coding部署方式，HDFS Disk Balancer相关命令，短路本地读取安全性改进，短路本地读取配置
hadoop-3.3.6.tar.gz - hadoop 3.3.6 安装包
07-19
文件名: hadoop-3.3.6.tar.gz 这是 Hadoop 3.3.6 版本的安装包(即二进制版,不是源码)，文件格式为 tar.gz,解压后直接使用. Hadoop 是一个由 Apache 基金会所开发的分布式系统基础架构，用户可以在不了解分布式底层...
hadoop-3.3.6-src.tar.gz
最新发布
04-09
`hadoop-3.3.6-src.tar.gz`是一个源代码压缩包，包含了Hadoop项目在版本3.3.6中的所有源代码。这个版本可能包含了一些性能优化、新功能的添加以及对之前版本中bug的修复。 一、Hadoop架构概述 Hadoop由两个主要组件...
hadoop-2.7.7编译
11-23
sudo cp -r hadoop-dist/target/hadoop-2.7.7/* /usr/local/hadoop ``` 这将编译源码，并将结果安装到指定目录。 ### 7. 配置环境变量 在用户`.bashrc`或`.bash_profile`文件中添加Hadoop环境变量： ```bash export...
适用于hadoop 3.3.5 3.3.6版本的winutils
10-08
winutils文件合集，Hadoop3.3.5以及3.3.6可用 winutils.exe是在windows系统上安装hadoop时所需要的winutils文件，内附多个版本，支持 hadoop-3.3.5 hadoop-3.3.6
Hadoop下载 hadoop-3.3.3.tar.gz
07-06
Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力进 Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不...
hadoop3.2.3 编译 native库（超详细，亲测成功）
qq_36835255的博客
05-13 4660
目录为什么要编译 native 库安装工具包安装 zlib，bzip2，lz4安装 openssl安装 JDK8，maven安装 cmake安装 protobuf安装 snappy安装 zstd安装 isa-l安装 aws-DynamoDBLocal 包安装 aws-java-sdk-bundle 包编译 native 库验证踩坑总结 为什么要编译 native 库 hadoop 二进制包是提供预编译的 native 库，但是 32 位的，服务器基本是 64 位，所以需要重新编译 native 库， 安装工
大数据集群:hadoop3.3.6,spark,hbase,zookeeper
capatical的博客
11-14 994
大数据集群hadoop3.3.6，spark，hbase
Hadoop-3.3.4集群部分lib缺失问题
Brother_ning的博客
12-25 684
Failed to load libisal.so.2问题 Cannot load libcrypto.so 问题 hadoop checknative
hadoop-common: CMake failed with error code 1
寂夜了无痕的博客
01-16 551
hadoop-common: CMake failed with error code 1
Hadoop 集成Snappy 全流程记录 提供依赖组件 【MAC 躺坑记录】
留歌__36的博客
12-20 616
写在前面 之前我在本地创键Hive表的时候一般都是以textFile格式进行存储的，最近玩的一些大表确实占用空间的，就想着Parquet格式存储+Snappy压缩来实现数据的存取。殊不知， Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path 于是我检查了下本地环境： ➜ ~ hadoop checknative 20/12/20 12:00:33 WARN util.NativeCodeLoader:
hadoop 不能加载native-hadoop library问题
yanhe156的博客
04-06 4738
hadoop启动时，发现启动不了datanode，查看slave上的log，发现了这个警告： WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 然后执行 hadoop checkna...
Hadoop/Hbase的Snappy安装
lixucpf的专栏
01-15 2239
一、Snappy安装环境： gcc c++, autoconf, automake, libtool, Java 6, JAVA_HOME set, Maven 3 其中gcc一定要是4.4的版本。 二、下载Snappy 1.1.1 下载地址：http://code.google.com/p/snappy/downloads/list 三、编译Snappy ./config
MAC PRO OS 安装 Hadoop 3.3.1 [防止踩坑]
Eillfyh的博客
11-23 2279
先安装好Homebrew 苹果电脑 常规安装脚本（推荐 完全体 几分钟安装完成）： /bin/zsh -c "$(curl -fsSL https://gitee.com/cunkai/HomebrewCN/raw/master/Homebrew.sh) 苹果电脑 极速安装脚本（精简版 几秒钟安装完成）： /bin/zsh -c "$(curl -fsSL https://gitee.com/cunkai/HomebrewCN/raw/master/Homebrew.sh)" speed 其次装好JAVA
分割cd /root cp hadoop-huaweicloud-2.8.3-hw-39.jar /home/modules/hadoop- 2.8.3/share/hadoop/common/lib/ cp hadoop-huaweicloud-2.8.3-hw-39.jar /home/modules/hadoop-2.8.3/share/hadoop/tools/lib cp hadoop-huaweicloud-2.8.3-hw-39.jar /home/modules/hadoop2.8.3/share/hadoop/hƩ pfs/tomcat/webapps/webhdfs/WEB-INF/lib/ cp hadoop-huaweicloud-2.8.3-hw-39.jar /home/modules/hadoop-2.8.3/share/hadoop/hdfs/lib/
06-05
2. cp hadoop-huaweicloud-2.8.3-hw-39.jar /home/modules/hadoop-2.8.3/share/hadoop/common/lib/ 3. cp hadoop-huaweicloud-2.8.3-hw-39.jar /home/modules/hadoop-2.8.3/share/hadoop/tools/lib 4. cp hadoop-...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值