hadoop-2.6.0-cdh5.12.1源码编译支持Snappy

标签: 源码
7人阅读 评论(0) 收藏 举报
分类:

最近学习Kylin,但是在执行到第一个Example时,提示没有Snappy支持。之前没有配置过Snappy,在经过一系列的Google、百度后,并没有成功添加配置,最后还是通过编译源码的方式才将支持搞定,现在先说一下大体流程,因为尝试了很多种方式,只能提供大体思路

下载源码

http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.12.1-src.tar.gz

环境准备

jdk1.7
protobuf 2.5.0

编译源码

mvn clean  package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib   -Dbundle.snappy

查看在编译的过程中遇到问题,大多数问题都可以百度到解决,只有下面这个异常找了很多

No versions available for com.amazonaws:DynamoDBLocal:jar:[1.11.86,2.0) with

下面的解决方案试了并没有用

https://blog.csdn.net/difffate/article/details/78767860

如果可以看到没有亚马逊这个包的错误,也就代表快成功了
PS:下面是我的解决方式
在下面下载DynamoDBLocal.jar
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html
修改hadoop-2.6.0-cdh5.12.1/hadoop-project/pom.xml

注释
<!--<dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>DynamoDBLocal</artifactId>
        <version>[1.11.86,2.0)</version>
      </dependency>-->
变更为
<dependency>
   <groupId>com.amazonaws</groupId>
   <artifactId>DynamoDBLocal</artifactId>
   <version>1.11.86</version>
   <scope>system</scope>
   <systemPath>/home/levin/download/DynamoDBLocal.jar</systemPath>
 </dependency>

并且修改
hadoop-2.6.0-cdh5.12.1/hadoop-tools/hadoop-aws/pom.xml

<dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>DynamoDBLocal</artifactId>
      <!--<scope>test</scope>-->  注释这个作用域
      <exclusions>
        <exclusion>
          <groupId>org.hamcrest</groupId>
          <artifactId>hamcrest-core</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.eclipse.jetty</groupId>
          <artifactId>jetty-http</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

在修改这些之后编译

[INFO] Executed tasks
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /home/levin/download/hadoop-2.6.0-cdh5.12.1/hadoop-dist/target/hadoop-dist-2.6.0-cdh5.12.1-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  2.775 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.545 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.610 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.424 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.381 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.657 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.740 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  3.534 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  3.630 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.029 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:05 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.236 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  7.973 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.083 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:36 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 15.541 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  3.737 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.789 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.115 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.119 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 55.179 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 14.725 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.244 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  6.425 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 16.452 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  1.987 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  3.750 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 10.428 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  0.940 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  3.255 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.067 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  1.773 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.549 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.069 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  3.271 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  4.148 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.169 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 11.265 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [  9.555 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  2.307 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  5.493 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  3.430 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  3.511 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  2.009 s]
[INFO] hadoop-mapreduce-client-nativetask ................. SUCCESS [01:03 min]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  3.900 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  3.799 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  2.431 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  6.144 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.320 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.513 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.193 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.525 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.636 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.476 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.712 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  6.417 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.381 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  8.754 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  3.018 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  3.519 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.014 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.565 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  2.515 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 10.708 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.034 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 29.076 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:08 min
[INFO] Finished at: 2018-04-16T14:53:33+08:00
[INFO] Final Memory: 264M/804M
[INFO] ------------------------------------------------------------------------

成功了
执行下面的命令

cp hadoop-2.6.0-cdh5.12.1/hadoop-dist/target/hadoop-2.6.0-cdh5.12.1/lib/native/* $HADOOP_HOME/lib/
hadoop checknative -a 

已经支持Snappy

18/04/16 14:56:37 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
18/04/16 14:56:37 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/levin/install-soft/bigdata/hadoop-2.6.0-cdh5.12.1/lib/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/x86_64-linux-gnu/libsnappy.so.1
lz4:     true revision:10301
bzip2:   false 
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
18/04/16 14:56:37 INFO util.ExitUtil: Exiting with status 1

最后编译成功的lib\native下的文件可以到下面下载直接覆盖使用

https://download.csdn.net/download/q2365921/10352534

查看评论

CMDB开发实战(支持SaltStack、SSH、Agent)

-
  • 1970年01月01日 08:00

hadoop源码编译:支持snappy压缩

hadoop源码编译:支持snappy压缩      在大数据的情况下,数据量都是特别的大,有时候呢,我们想把数据进行压缩之后在进行处理,这样呢可以大幅度的减少磁盘的IO而且还可以减少网络传输的压...
  • liyaohhh
  • liyaohhh
  • 2016-02-17 16:36:58
  • 2077

重新编译hadoop-2.7.2-src的native以支持Snappy解压压缩库

转载:http://blog.itpub.net/30089851/viewspace-2120631/ 1. 下载所需要的源码包 snappy-1.1.1.tar.gz、proto...
  • developerinit
  • developerinit
  • 2016-11-10 17:41:02
  • 849

编译hadoop支持snappy压缩

kylin在build cube时出现hadoop不支持snappy压缩的问题,原因是Hadoop的二进制安装包中没有snappy支持,需要手工重新编译。 编译过程: 1. 安装包列表 sn...
  • Gospelanswer
  • Gospelanswer
  • 2017-02-07 22:29:45
  • 1200

hadoop支持SnappyCodec的步骤

1、需求:让hadoop基础运行环境支持SnappyCodec(这个基础运行环境中运行着NameNode(也是MasterNode)和DataNode(也是ClusterNode)上的Daemon和进...
  • DSLZTX
  • DSLZTX
  • 2015-05-14 12:53:47
  • 1568

Hadoop2.6.0 Snappy 压缩安装与配置

先说点题外话,我们集群主要使用hive,Impala,spark sql等来处理数据,介于hdfs里面的数据没有压缩占用空间太大所以我们要对其进行压缩snappy比较适中 snappy 原生被hive...
  • leone911
  • leone911
  • 2016-07-08 14:28:19
  • 1568

Hadoop2.7.3 编译 支持5种压缩

导读:       按博客中的操作步骤操作后,hadoop 编译完成后可以支持5种本地压缩:zlib、snappy、lz4、bzip2、openssl笔者环境配置为:JDK1.8、Hadoop2.7...
  • HG_Harvey
  • HG_Harvey
  • 2017-08-24 19:49:19
  • 649

重新编译Hadoop 2.7.2 native以支持snappy

问题提出: 在运行kylin sample时出现以下错误: org.apache.hadoop.hive.ql.metadata.HiveException: native snappy librar...
  • wzy0623
  • wzy0623
  • 2016-04-27 17:43:02
  • 5015

hbase应用snappy的部署

因为业务需要所以对hbase的数据做整体的压缩处理。
  • chenyi8888
  • chenyi8888
  • 2013-11-05 15:55:30
  • 5572

在hadoop2.X集群中安装压缩工具snappy(主要用于hbase)

在hadoop集群中snappy是一种比较好的压缩工具,相对gzip压缩速度和解压速度有很大的优势,而且相对节省cpu资源,但压缩率不及gzip。在此不讨论孰优孰劣,各有各的用途。 在此,hadoo...
  • qq_27078095
  • qq_27078095
  • 2017-02-25 00:38:01
  • 3077
    个人资料
    持之以恒
    等级:
    访问量: 3万+
    积分: 703
    排名: 7万+
    最新评论