服务器操作系统:CentOS 6.3
一、安装LZO
首先尝试sudo apt-get -y install liblzo2-dev,如果不行则按照下面的做法。
1.检查是否已经安装
ls /usr/lib64/liblz*
2.安装LZO
2.1.从http://pkgs.org/下载
http://pkgs.org/centos-6-rhel-6/centos-rhel-x86_64/lzo-devel-2.03-3.1.el6.x86_64.rpm/download/
http://pkgs.org/centos-6-rhel-6/centos-rhel-x86_64/lzo-minilzo-2.03-3.1.el6.x86_64.rpm/download/
2.2.安装
zlib-1.2.3-29.el6.x86_64.rpm
zlib-devel-1.2.3-29.el6.x86_64.rpm
lzo-2.03-3.1.el6.x86_64.rpm
lzo-minilzo-2.03-3.1.el6.x86_64.rpm
lzo-devel-2.03-3.1.el6.x86_64.rpm
sudo rpm -ivh zlib-1.2.3-29.el6.x86_64.rpm --force
sudo rpm -ivh zlib-devel-1.2.3-29.el6.x86_64.rpm
sudo rpm -ivh lzo-*
3.检查安装是否成功
ls /usr/lib64/liblz*
二、安装Hadoop LZO
1.下载hadoop-lzo源码
git clone git://github.com/twitter/hadoop-lzo.git
2.从其他服务器上面复制文件到本地目录
scp /home/bigdata/apps/hbase/lib/hadoop-core-2.0.0-mr1-cdh4.1.2.jar 10.154.80.2:/home/bigdata/apps
scp /home/bigdata/apps/hbase/lib/hadoop-common-2.0.0-cdh4.0.1.jar 10.154.80.2:/home/bigdata/apps
3.安装hadoop-lzo
cd hadoop-lzo
CLASSPATH=/home/bigdata/apps/hadoop-common-2.0.0-cdh4.0.1.jar:/home/bigdata/apps/hadoop-core-2.0.0-mr1-cdh4.1.2.jar CFLAGS=-m64 CXXFLAGS=-m64 ant compile-native tar
4.复制构建好的库文件到相应目录
cd /home/bigdata/apps/hadoop-mr/lib
mkdir native
mkdir native/Linux-amd64-64
scp 10.154.80.2:/home/bigdata/apps/hadoop-lzo/build/hadoop-lzo-0.4.17-SNAPSHOT.jar /home/bigdata/apps/hadoop-mr/lib/
scp 10.154.80.2:/home/bigdata/apps/hadoop-lzo/build/hadoop-lzo-0.4.17-SNAPSHOT.jar /home/bigdata/apps/hbase/lib/
scp 10.154.80.2:/home/bigdata/apps/hadoop-lzo/build/native/Linux-amd64-64/lib/libgplcompression.* /home/bigdata/apps/hadoop-mr/lib/native/Linux-amd64-64/
scp 10.154.80.2:/home/bigdata/apps/hadoop-lzo/build/native/Linux-amd64-64/lib/libgplcompression.* /home/bigdata/apps/hbase/lib/native/Linux-amd64-64/
三、重启HDFS、MapReduce、HBase
四、检查安装是否成功
1.Testing Compression is enabled
hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/testfile lzo
hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/testfile lzo
2.create table
create 'mytable', {NAME => 'colfam', COMPRESSION => 'LZO'}
五、参考资料
1.OpenTSDB Setup HBase
2.OpenTSDB安装HBase
3.UsingLzoCompression
4.FAQ - hadoop-gpl-compression
5.Linux 安装lzop压缩工具