hbase snappy 安装_Hadoop和HBase安装snappy压缩库(2.4.1)

说明

hadoop中经常需要使用snappy这个压缩库。但这个库并未集成成hadoop安装包中，需要额外安装。

安装思路：先在集群中找一台机器，完整安装snappy。其他节点只拷贝相关库过去，并配置hadoop即可。

注意：安装Snappy之前，需要先解决Hadoop 2.4.1的libhadoop.so的位数与运行环境不一致问题(参考：Hadoop源码编译)

安装过程

选取一个节点安装

安装snappy snappy-devel库

yum install -y snappy snappy-devel

cp /usr/lib64/libsnapp* /usr/local/lib

编译snappy-hadoop库

wget https://github.com/electrum/hadoop-snappy/archive/master.zip

unzip master.zip

cd hadoop-snappy-master/

#编译

mvn package

拷贝hadoop snappy库

编译完成后,解压target目录下的hadoop-snappy-0.0.1-SNAPSHOT.tar.gz,并拷贝相关文件到hadoop

cd target

tar -xcvf hadoop-snappy-0.0.1-SNAPSHOT.tar.gz

cp hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* /opt/hadoop/lib/native/

cp hadoop-snappy-0.0.1-SNAPSHOT/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar /opt/hadoop/lib/

配置hadoop

hadoop-env.sh增加：

vi /opt/hadoop/etc/hadoop/hadoop-env.sh

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/

其他节点

将主节点的相关文件拷贝到其他节点

scp hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* data01:/opt/hadoop/lib/native/

scp hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* data02:/opt/hadoop/lib/native/

scp hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* data03:/opt/hadoop/lib/native/

scp hadoop-snappy-0.0.1-SNAPSHOT/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar data01:/opt/hadoop/lib/

scp hadoop-snappy-0.0.1-SNAPSHOT/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar data02:/opt/hadoop/lib/

scp hadoop-snappy-0.0.1-SNAPSHOT/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar data03:/opt/hadoop/lib/

scp /usr/lib64/libsnapp* data01:/usr/local/lib

scp /usr/lib64/libsnapp* data02:/usr/local/lib

scp /usr/lib64/libsnapp* data03:/usr/local/lib

其他节点配置hadoop

vi /opt/hadoop/etc/hadoop/hadoop-env.sh

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/

所有节点

在$HADOOP_HOME/etc/hadoop/core-site.xml文件中加入snappy配置:

io.compression.codecs

org.apache.hadoop.io.compress.GzipCodec,

org.apache.hadoop.io.compress.DefaultCodec,

org.apache.hadoop.io.compress.BZip2Codec,

org.apache.hadoop.io.compress.SnappyCodec

验证安装结果

hadoop checknative -a | grep snappy

HBase安装Snappy库

安装过程

在上面步骤的基础上，将snappy-hadoop编译出来的lib库拷贝到hbase目录下：

cp target/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* /opt/hbase/lib/native/Linux-amd64-64/

将Hadoop lib目录下的所有本地库也拷贝到hbase中：

cp /opt/hadoop/lib/native/* /opt/hbase/lib/native/Linux-amd64-64/

测试HBase Snappy压缩可用

hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/a.txt snappy

参考文档