hbase 使用lzo_装配HBase LZO

服务器操作系统：CentOS 6.3

一、安装LZO

首先尝试sudo apt-get -y install liblzo2-dev，如果不行则按照下面的做法。

1.检查是否已经安装

ls /usr/lib64/liblz*

2.安装LZO

2.1.从http://pkgs.org/下载

http://pkgs.org/centos-6-rhel-6/centos-rhel-x86_64/lzo-devel-2.03-3.1.el6.x86_64.rpm/download/

http://pkgs.org/centos-6-rhel-6/centos-rhel-x86_64/lzo-minilzo-2.03-3.1.el6.x86_64.rpm/download/

2.2.安装

zlib-1.2.3-29.el6.x86_64.rpm

zlib-devel-1.2.3-29.el6.x86_64.rpm

lzo-2.03-3.1.el6.x86_64.rpm

lzo-minilzo-2.03-3.1.el6.x86_64.rpm

lzo-devel-2.03-3.1.el6.x86_64.rpm

sudo rpm -ivh zlib-1.2.3-29.el6.x86_64.rpm --force

sudo rpm -ivh zlib-devel-1.2.3-29.el6.x86_64.rpm

sudo rpm -ivh lzo-*

3.检查安装是否成功

ls /usr/lib64/liblz*

二、安装Hadoop LZO

1.下载hadoop-lzo源码

git clone git://github.com/twitter/hadoop-lzo.git

2.从其他服务器上面复制文件到本地目录

scp /home/bigdata/apps/hbase/lib/hadoop-core-2.0.0-mr1-cdh4.1.2.jar 10.154.80.2:/home/bigdata/apps

scp /home/bigdata/apps/hbase/lib/hadoop-common-2.0.0-cdh4.0.1.jar 10.154.80.2:/home/bigdata/apps

3.安装hadoop-lzo

cd hadoop-lzo

CLASSPATH=/home/bigdata/apps/hadoop-common-2.0.0-cdh4.0.1.jar:/home/bigdata/apps/hadoop-core-2.0.0-mr1-cdh4.1.2.jar CFLAGS=-m64 CXXFLAGS=-m64 ant compile-native tar

4.复制构建好的库文件到相应目录

cd /home/bigdata/apps/hadoop-mr/lib

mkdir native

mkdir native/Linux-amd64-64

scp 10.154.80.2:/home/bigdata/apps/hadoop-lzo/build/hadoop-lzo-0.4.17-SNAPSHOT.jar /home/bigdata/apps/hadoop-mr/lib/

scp 10.154.80.2:/home/bigdata/apps/hadoop-lzo/build/hadoop-lzo-0.4.17-SNAPSHOT.jar /home/bigdata/apps/hbase/lib/

scp 10.154.80.2:/home/bigdata/apps/hadoop-lzo/build/native/Linux-amd64-64/lib/libgplcompression.* /home/bigdata/apps/hadoop-mr/lib/native/Linux-amd64-64/

scp 10.154.80.2:/home/bigdata/apps/hadoop-lzo/build/native/Linux-amd64-64/lib/libgplcompression.* /home/bigdata/apps/hbase/lib/native/Linux-amd64-64/

三、重启HDFS、MapReduce、HBase

四、检查安装是否成功

1.Testing Compression is enabled

hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/testfile lzo

hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/testfile lzo

2.create table

create 'mytable', {NAME => 'colfam', COMPRESSION => 'LZO'}

五、参考资料

1.OpenTSDB Setup HBase

2.OpenTSDB安装HBase

3.UsingLzoCompression

4.FAQ - hadoop-gpl-compression

5.Linux 安装lzop压缩工具