编译生成hadoop-native-Libraries的问题

最新推荐文章于 2021-01-26 22:10:41 发布
最新推荐文章于 2021-01-26 22:10:41 发布
阅读量996 收藏
点赞数
分类专栏： 岁月云——大数据杂烩 文章标签： hadoop native
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/warrah/article/details/78919737
版权

按照在hadoop2.X集群中安装压缩工具snappy（主要用于hbase）执行mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy

[INFO] Apache Hadoop Main ................................. SUCCESS [  2.032 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.896 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.446 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.707 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.489 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.094 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  4.116 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  5.766 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  6.048 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.207 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [02:29 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 23.382 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:25 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.072 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [07:22 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [01:38 min]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 54.000 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  6.245 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.057 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.053 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [02:09 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 44.575 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.052 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 11.960 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 46.505 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  4.079 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  6.691 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 20.051 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  2.228 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  6.313 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.066 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  3.194 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  2.629 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.064 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  6.706 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  6.343 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.319 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 25.479 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 19.312 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  4.868 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 10.914 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  7.518 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 19.306 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  2.765 s]
[INFO] hadoop-mapreduce-client-nativetask ................. SUCCESS [01:39 min]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  8.167 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  6.858 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  9.815 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 28.586 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  2.662 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  2.878 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  6.565 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  5.282 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  3.215 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  2.709 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  3.957 s]
[INFO] Apache Hadoop Pipes ................................ FAILURE [  8.653 s]
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Client ............................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED

编译Apache Hadoop Pipes 的时候失败， 异常信息如下：

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part ...<exec dir="/application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/native" executable="make" failonerror="true">... @ 8:127 in /application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

在hadoop2.2.0编译–错误解决办法(转)找到了一个答案，不过答案是错误的.zlib-devel安装过，但是问题还是出现了，苦恼中。
执行命令mvn package -X -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy查看debug日志,详细错误日志如下：

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 2
around Ant part ...<exec dir="/application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/native" executable="make" failonerror="true">... @ 8:127 in /application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 2
around Ant part ...<exec dir="/application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/native" executable="make" failonerror="true">... @ 8:127 in /application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml
        at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
        ... 20 more
Caused by: /application/hadoop-2.6.0-cdh5.9.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml:8: exec returned: 2
        at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:646)
        at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
        at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at org.apache.tools.ant.Target.execute(Target.java:390)
        at org.apache.tools.ant.Target.performTasks(Target.java:411)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
        at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
        at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:327)
        ... 22 more
[ERROR]

这篇Hadoop2.2.0源码编译文章里面遇到了这个问题，他提出的方案，也没有解决我的问题。
执行mvn validate,校验通过
2
我猜想是不是hadoop cdh版本和apache hadoop有什么不同，于是使用在hadoop release里面下载hadoop 2.5.0版本的源码，进行编译，还是出现同样的问题，真是崩溃。
最终解决方案，我换了一台干净的机器环境，重新安装后，ok了。
执行命令hadoop checknative -a安装成功。

[hadoop@dashuju174 ~]$ hadoop checknative -a
18/01/12 13:51:35 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
18/01/12 13:51:35 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/hadoop/application/hadoop-2.6.4/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /home/hadoop/application/hadoop-2.6.4/lib/native/libsnappy.so.1
lz4:     true revision:99
bzip2:   false 
openssl: true /usr/lib64/libcrypto.so
18/01/12 13:51:35 INFO util.ExitUtil: Exiting with status 1
warrah
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
hadoop-native-64-2.4.0.tar
10-18
hadoop-native-64-2.4.0.tar 解决运行hadoop 警告：WARN util.NativeCodeLoader: Unable to load native-hadoop
hadoop native： Failed to load/initialize native-bzip2 library system-native
breakout_alex的博客
12-06 1817
问题描述： 部署完Hadoop之后，执行hadoop checknative 对本地库做检查： ./bin/hadoop checknative 19/12/06 11:31:39 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java...
使用Flink时遇到的问题（不断更新中）
an7800666的博客
09-16 2143
1.启动不起来 查看JobManager日志： WARN org.apache.flink.runtime.webmonitor.JobManagerRetriever - Failed to retrieve leader gateway and port. akka.actor.ActorNotFound: Actor not found for: Actor...
Hive使用Snappy压缩，Parquet格式存储文件
x²＋(y－√³x²)²＝1的博客
01-26 3614
一、数据内容 95002,刘晨,女,19,IS 95017,王风娟,女,18,IS 95018,王一,女,19,IS 95013,冯伟,男,21,CS 95014,王小丽,女,19,CS 95019,邢小丽,女,19,IS 95020,赵钱,男,21,IS 二、查看 HDFS 是否支持 Snappy 格式 在 shell 命令行执行： hadoop checknative 返回： 21/01/26 20:32:52 WARN bzip2.Bzip2Factory: Failed to load/init
eclipse中新建mapreduce项目没有自动导入jar包 或者 Failed to load/initialize native-zlib library
qq_44761561的博客
03-29 1254
这里我的错误是，在Windows下解压Hadoop要到在无中文无空格的路径下，问题解决 当然如果报以下错误也可能是路径错误 Failed to load/initialize native-zlib library 实际上第一个错误困扰我很久了，我是在第二个错误中发现的，在网上找一会没有（可能是我没找到）就想换个路径，结果两个问题都解决了 ...
序列文件的读取和写入
Joy
01-04 1291
1、序列文件（二进制）的写入 /** * 写入文件 * @throws IOException */ @Test public void save() throws IOException { Configuration conf = new Configuration(); conf.set("fs.defa
Hadoop的本地库（Native Libraries）及相关问题总结
u013303361的专栏
04-23 1万+
主体转自：http://blog.sina.com.cn/s/blog_3d9e90ad0102wqrp.html （未亲测，我的getconf LONG_BIT返回32，且file libhadoop.so.1.0.0返回64，与之相反）Hadoop的本地库（Native Libraries）介绍Hadoop是使用Java语言开发的，但是有一些需求和操作并不适合使用java，所以就引入了本地库（...
CDH5.8.4-Hadoop2.6.0安常见报错及解决方法
SimpleSimpleSimples的博客
01-26 937
1.cdh2.6.0安装完成时报错 启动hadoop集群时报错，如下: 20/01/23 22:27:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 原因：导致该问题的改版本是...
WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure...
12-09 833
[root@hdp2 /root]#hadoop checknative -a 18/12/09 00:31:19 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version 18/12/09 00:31:19 INF...
hadoop native本地库问题总结
热门推荐
知猪侠的专栏
08-18 3万+
最近，打算hbase建表用snappy压缩时，碰到一些hadoop本地库的问题。其实这些问题是一直存在的，只是不影响正常使用，就没有引起重视。这次希望彻底解决以下问题： 问题一：执行start-dfs.sh时出现以下日志 xxxx: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/had
hadoop-native-lib-x64-2.7.3.tar.gz
09-11
hadoop 2.7.3 的本地接口库，snappy 使用的是1.1.3版本。使用时直接把包解压后放到$HADOOP_HOME/lib/native目录下即可。 [root@pseudo ~]# hadoop checknative -a 17/09/11 14:30:49 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version 17/09/11 14:30:49 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /taidl/hadoop/lib/native/libhadoop.so zlib: true /lib64/libz.so.1 snappy: true /taidl/hadoop/lib/native/libsnappy.so.1 lz4: true revision:99 bzip2: false openssl: true /lib64/libcrypto.so 17/09/11 14:30:50 INFO util.ExitUtil: Exiting with status 1 [root@pseudo ~]#
hadoop-2.6.0编译好的64bit的native库
03-23
在64位系统上运行Hadoop 2.6.0会出现以下提示： 用缺省的32位native库(/opt/hadoop-2.6.0/lib/native)会出现下面的错误: 14/01/27 10:52:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'. 这是因为hadoop默认编译好的native库是32bit的，所以要重新编译hadoop2.6的源代码，获取64位native库，将其复制到原来安装的hadoop的native目录中替代原来的类库，然后重新运行即可。
mac系统下编译hadoop3.2.1生成的native包
06-21
mac系统下编译hadoop3.2.1源码后生成的lib/native下的包，为了解决“WARN util.NativeCodeLoader”警告，包含snappy及isa-l，编译过程可以看https://blog.csdn.net/weixin_44570264/article/details/106846117
hadoop-native-64-2.5.0.tar
09-14
解决：Unable to load ...Apache提供的hadoop本地库是32位的，而在64位的服务器上就会有问题，因此需要自己编译64位的版本。 将准备好的64位的lib包解压到已经安装好的hadoop安装目录的lib/native 和 lib目录下：
hadoop-native-64-2.6.0.zip
07-07
Hadoop使用过程报错：WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...解决 下载过后把tar包解压在hdoop的lib/native下即可
hadoop-native-64-2.6.0.tar
09-14
解决：Unable to load ...Apache提供的hadoop本地库是32位的，而在64位的服务器上就会有问题，因此需要自己编译64位的版本。 将准备好的64位的lib包解压到已经安装好的hadoop安装目录的lib/native 和 lib目录下：
编译hadoophadoop-3.2.2-src源码
03-30
编译hadoophadoop-3.2.2-src的源码
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has cl
warrah 南极狼
01-15 3万+
今天在hbase中执行下面的命令 disable 'iw:test06' alter 'iw:test06',NAME=>'i',COMPRESSION=>'SNAPPY' count 'iw:test06' 提示下面的异常信息 2018-01-15 10:49:20,660 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0
第1.4章 elasticsearch删除字段
warrah 南极狼
10-10 2万+
es因为文档存储，所以一旦字段确立，想要更改就不是那么件容易的事情，网上找到很多的例子，有很多只是说了一半。 比如这篇文章里面的elasticsearch mapping 添加 编辑 删除字段，他说要把导出的json数据中，删除掉字段对应的数据去掉，才能导入，然后没有下文了。确实着实让人苦恼。 elasticsearch更改mapping(不停服务重建索引),这篇文章，告诉我一个思路，但是提到如何...
hadoop-native下载
最新发布
07-28
您可以通过以下步骤从CSDN下载Hadoop Native: 1. 打开浏览器，访问CSDN官网（www.csdn.net）。 2. 在搜索栏中输入“Hadoop Native”并点击搜索按钮。 3. 在搜索结果中找到适合您需求的Hadoop Native版本，点击进入...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

warrah

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值