hadoop-2.6.0-cdh5.12.1源码编译支持Snappy

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/q2365921/article/details/79961412

最近学习Kylin,但是在执行到第一个Example时,提示没有Snappy支持。之前没有配置过Snappy,在经过一系列的Google、百度后,并没有成功添加配置,最后还是通过编译源码的方式才将支持搞定,现在先说一下大体流程,因为尝试了很多种方式,只能提供大体思路

下载源码

http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.12.1-src.tar.gz

环境准备

jdk1.7
protobuf 2.5.0

编译源码

mvn clean  package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib   -Dbundle.snappy

查看在编译的过程中遇到问题,大多数问题都可以百度到解决,只有下面这个异常找了很多

No versions available for com.amazonaws:DynamoDBLocal:jar:[1.11.86,2.0) with

下面的解决方案试了并没有用

https://blog.csdn.net/difffate/article/details/78767860

如果可以看到没有亚马逊这个包的错误,也就代表快成功了
PS:下面是我的解决方式
在下面下载DynamoDBLocal.jar
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html
修改hadoop-2.6.0-cdh5.12.1/hadoop-project/pom.xml

注释
<!--<dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>DynamoDBLocal</artifactId>
        <version>[1.11.86,2.0)</version>
      </dependency>-->
变更为
<dependency>
   <groupId>com.amazonaws</groupId>
   <artifactId>DynamoDBLocal</artifactId>
   <version>1.11.86</version>
   <scope>system</scope>
   <systemPath>/home/levin/download/DynamoDBLocal.jar</systemPath>
 </dependency>

并且修改
hadoop-2.6.0-cdh5.12.1/hadoop-tools/hadoop-aws/pom.xml

<dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>DynamoDBLocal</artifactId>
      <!--<scope>test</scope>-->  注释这个作用域
      <exclusions>
        <exclusion>
          <groupId>org.hamcrest</groupId>
          <artifactId>hamcrest-core</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.eclipse.jetty</groupId>
          <artifactId>jetty-http</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

在修改这些之后编译

[INFO] Executed tasks
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /home/levin/download/hadoop-2.6.0-cdh5.12.1/hadoop-dist/target/hadoop-dist-2.6.0-cdh5.12.1-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  2.775 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.545 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.610 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.424 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.381 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.657 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.740 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  3.534 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  3.630 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.029 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:05 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.236 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  7.973 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.083 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:36 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 15.541 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  3.737 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.789 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.115 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.119 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 55.179 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 14.725 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.244 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  6.425 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 16.452 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  1.987 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  3.750 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 10.428 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  0.940 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  3.255 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.067 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  1.773 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.549 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.069 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  3.271 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  4.148 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.169 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 11.265 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [  9.555 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  2.307 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  5.493 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  3.430 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  3.511 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  2.009 s]
[INFO] hadoop-mapreduce-client-nativetask ................. SUCCESS [01:03 min]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  3.900 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  3.799 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  2.431 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  6.144 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.320 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.513 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.193 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.525 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.636 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.476 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.712 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  6.417 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.381 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  8.754 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  3.018 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  3.519 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.014 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.565 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  2.515 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 10.708 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.034 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 29.076 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:08 min
[INFO] Finished at: 2018-04-16T14:53:33+08:00
[INFO] Final Memory: 264M/804M
[INFO] ------------------------------------------------------------------------

成功了
执行下面的命令

cp hadoop-2.6.0-cdh5.12.1/hadoop-dist/target/hadoop-2.6.0-cdh5.12.1/lib/native/* $HADOOP_HOME/lib/
hadoop checknative -a 

已经支持Snappy

18/04/16 14:56:37 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
18/04/16 14:56:37 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/levin/install-soft/bigdata/hadoop-2.6.0-cdh5.12.1/lib/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/x86_64-linux-gnu/libsnappy.so.1
lz4:     true revision:10301
bzip2:   false 
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
18/04/16 14:56:37 INFO util.ExitUtil: Exiting with status 1

最后编译成功的lib\native下的文件可以到下面下载直接覆盖使用

https://download.csdn.net/download/q2365921/10352534

阅读更多

没有更多推荐了,返回首页