hbase snappy 安装_Hadoop2.7.1+Hbase1.1.2集群环境搭建(六)snappy安装

(1)hadoop2.7.1源码编译http://zilongzilong.iteye.com/blog/2246856

(2)hadoop2.7.1安装准备http://zilongzilong.iteye.com/blog/2253544

(3)hadoop2.7.1安装http://zilongzilong.iteye.com/blog/2245547

(4)hbase安装准备http://zilongzilong.iteye.com/blog/2254451

(5)hbase安装http://zilongzilong.iteye.com/blog/2254460

(6)snappy安装                   http://zilongzilong.iteye.com/blog/2254487

(7)雅虎YCSBC测试hbase性能测试     http://zilongzilong.iteye.com/blog/2248863

(8)spring-hadoop实战                   http://zilongzilong.iteye.com/blog/2254491

lzo   snappy   gzip是hadoop支持的三种压缩方式，目前网上推荐snappy，这里讲解如何安装snappy

1.在linux上安装snappy

1)在https://github.com/google/snappy/releases这里有snappy-1.1.3.tar.gz的下载地址(https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz)，下载之。

2)执行如下命令安装

./configure

make

make install

3)默认安装在/usr/local/lib,安装成功后文件如下：

2.重新编译Hadoop2.7.1源码，主要目的是得到/opt/hadoop-2.7.1-src/hadoop-dist/target/hadoop-2.7.1/lib/native下的文件，这些文件已经基于刚才的snappy相关库进行构建。

1)进入Hadoop2.7.1源码目录 /opt/hadoop-2.7.1-src/，执行maven构建命令mvn package -Pdist,native -DskipTests -Dtar -rf :hadoop-common -Drequire.snappy -X ，大概需要15分钟，因为我之前编译过Hadoop2.7.1源码(参见http://zilongzilong.iteye.com/blogs/2246856)

2)编译完后复制/opt/hadoop-2.7.1-src/hadoop-dist/target/hadoop-2.7.1/lib/native/*到/opt/hadoop-2.7.1/lib/native/，复制/opt/hadoop-2.7.1-src/hadoop-dist/target/hadoop-2.7.1/lib/native/*到/opt/hbase-1.1.2/lib/native/Linux-amd64-64/(hbase中/opt/hbase-1.1.2/lib/native/没有Linux-amd64-64的，需要自己创建Linux-amd64-64)

3.构建hadoop-snappy源码

1)在https://github.com/electrum/hadoop-snappy中下载hadoop-snappy-master.zip(https://github.com/electrum/hadoop-snappy)源码文件hadoop-snappy-master.zip放置于/opt

2)解压unzip hadoop-snappy-master.zip -d hadoop-snappy-master-src

3)用maven进行构建

cd /opt/hadoop-snappy-master-src/hadoop-snappy-master

mvn package

4.hadooo中部署snappy

解压hadoop-snappy-0.0.1-SNAPSHOT.tar.gz文件，会生成hadoop-snappy-0.0.1-SNAPSHOT目录

拷贝这个目录下相关文件到$HADOOP_HOME/lib/native/Linux-amd64-64/*到/opt/hadoop-2.7.1/lib/native/

拷贝target目录下的hadoop-snappy-0.0.1-SNAPSHOT.jar到/opt/hadoop-2.7.1/share/hadoop/common/lib目录下

修改/opt/hadoop-2.7.1/etc/hadoop/core-site.xml,添加如下内容：

io.compression.codecs

org.apache.hadoop.io.compress.SnappyCodec

修改/opt/hadoop-2.7.1/etc/hadoop/mapred-site.xml,添加如下内容

mapred.output.compress

true

mapred.output.compression.codec

org.apache.hadoop.io.compress.SnappyCodec

mapred.compress.map.output

true

mapred.map.output.compression.codec

org.apache.hadoop.io.compress.SnappyCodec

修改后，重启hadoop集群。

5.hbase中部署snappy

复制/opt/hadoop-2.7.1/lib/native//opt/hadoop-2.7.1/lib/native/*到/opt/hbase-1.1.2/lib/native/Linux-amd64-64/下面，重启hbase。

6.在hbase中测试snappy

cd /opt/hbase-1.1.2/bin/

./hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://192.168.181.66:9000/hbase/data/default/signal/3a194dcd996fd03c0c26bf3d175caaec/info/0c7f62f10a4c4e548c5ff1c583b0bdfa snappy

上面hdfs://192.168.181.66:9000/hbase/data/default/signal/3a194dcd996fd03c0c26bf3d175caaec/info/0c7f62f10a4c4e548c5ff1c583b0bdfa是存在于我Hadoop上的HDFS文件