按照在hadoop2.X集群中安装压缩工具snappy(主要用于hbase)执行mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy
[INFO] Apache Hadoop Main ................................. SUCCESS [ 2.032 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 0.896 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.446 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.707 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.489 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.094 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 4.116 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 5.766 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 6.048 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.207 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [02:29 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 23.382 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:25 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.072 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [07:22 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [01:38 min]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 54.000 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 6.245 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.057 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.053 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [02:09 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 44.575 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.052 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 11.960 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 46.505 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 4.079 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 6.691 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 20.051 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 2.228 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 6.313 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.066 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 3.194 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 2.629 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.064 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 6.706 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 6.343 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.319 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 25.479 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 19.312 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 4.868 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 10.914 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 7.518 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 19.306 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 2.765 s]
[INFO] hadoop-mapreduce-client-nativetask ................. SUCCESS [01:39 min]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 8.167 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 6.858 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 9.815 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 28.586 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 2.662 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 2.878 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 6.565 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 5.282 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 3.215 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 2.709 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 3.957 s]
[INFO] Apache Hadoop Pipes ................................ FAILURE [ 8.653 s]
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Client ............................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
编译Apache Hadoop Pipes 的时候失败, 异常信息如下:
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part ...<exec dir="/application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/native" executable="make" failonerror="true">... @ 8:127 in /application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml
在hadoop2.2.0编译–错误解决办法(转)找到了一个答案,不过答案是错误的.zlib-devel
安装过,但是问题还是出现了,苦恼中。
执行命令mvn package -X -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy
查看debug日志,详细错误日志如下:
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 2
around Ant part ...<exec dir="/application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/native" executable="make" failonerror="true">... @ 8:127 in /application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 2
around Ant part ...<exec dir="/application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/native" executable="make" failonerror="true">... @ 8:127 in /application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
Caused by: /application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml:8: exec returned: 2
at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:646)
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:327)
... 22 more
[ERROR]
这篇Hadoop2.2.0源码编译文章里面遇到了这个问题,他提出的方案,也没有解决我的问题。
执行mvn validate
,校验通过
我猜想是不是hadoop cdh版本和apache hadoop有什么不同,于是使用在hadoop release里面下载hadoop 2.5.0版本的源码,进行编译,还是出现同样的问题,真是崩溃。
最终解决方案,我换了一台干净的机器环境,重新安装后,ok了。
执行命令hadoop checknative -a
安装成功。
[hadoop@dashuju174 ~]$ hadoop checknative -a
18/01/12 13:51:35 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
18/01/12 13:51:35 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /home/hadoop/application/hadoop-2.6.4/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /home/hadoop/application/hadoop-2.6.4/lib/native/libsnappy.so.1
lz4: true revision:99
bzip2: false
openssl: true /usr/lib64/libcrypto.so
18/01/12 13:51:35 INFO util.ExitUtil: Exiting with status 1