lzo-lzop

原创 2018年04月16日 14:58:12
编译环境生成 linux机器上执行

yum -y install lzo-devel zlib-devel gcc autoconf automake libtool

下载lzo

http://www.oberhumer.com/opensource/lzo/download/

lzo安装

tar -zxvf lzo-2.10.tar.gz -C /opt/
cd /opt/lzo-2.10
export CFLAGS=-m64
 ./configure -enable-shared -prefix=/opt/lzo/ 

make && sudo make install

安装Hadoop-LZO
wget https://github.com/twitter/hadoop-lzo/archive/master.zip
unzip master

cd /opt/hadoop-lzo-master

修改项目的pom.xml(因为我们的hadoop环境是2.6.0)

vi pom.xml
 <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <hadoop.current.version>2.6.0</hadoop.current.version>
    <hadoop.old.version>1.0.4</hadoop.old.version>

  </properties>

编译该项目

export CFLAGS=-m64
export CXXFLAGS=-m64
export C_INCLUDE_PATH=/opt/lzo/include
export LIBRARY_PATH=/opt/lzo/lib
mvn clean package -Dmaven.test.skip=true
mvn clean install -Dmaven.test.skip=true 部署到本地仓库(如果需要用这个编译)
cd target/native/Linux-amd64-64
mkdir /opt/lzo/libgplcompression

tar -cBf - -C lib . | tar -xBvf - -C /opt/lzo/libgplcompression/

将生成的文件发送到hadoop应用(所有hadoop节点都需要)

cp /opt/lzo/libgplcompression/libgplcompression* $HADOOP_HOME/lib/native/ 

cp  /opt/hadoop-lzo-master/target/hadoop-lzo-0.4.21-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/

修改hadoop的配置文件(所有hadoop节点都要改)

1、在Hadoop中的$HADOOP_HOME/etc/hadoop/hadoop-env.sh加上下面配置:
export LD_LIBRARY_PATH=/opt/lzo/lib 


2、在$HADOOP_HOME/etc/hadoop/core-site.xml加上如下配置:
<property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.GzipCodec,
           org.apache.hadoop.io.compress.DefaultCodec,
           com.hadoop.compression.lzo.LzoCodec,
           com.hadoop.compression.lzo.LzopCodec,
           org.apache.hadoop.io.compress.BZip2Codec
        </value>
</property>
<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>


3、在$HADOOP_HOME/etc/hadoop/mapred-site.xml加上如下配置


<property>
    <name>mapred.compress.map.output</name>
    <value>true</value>
</property>
 
<property>
    <name>mapred.map.output.compression.codec</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
 
<property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/opt/lzo/lib</value>
</property>

压缩文件

给lzo文件添加index

hadoop jar $HADOOP_HOMOE/share/hadoop/common/hadoop-lzo-0.4.21-SNAPSHOT.jar com.hadoop.compression.lzo.DistributedLzoIndexer /user/hadoop/test/test.lzo

生成出来的索引文件后缀为.index,并存放在lzo同一目录下。

日志文件的lzo压缩与解压(用lzop)

安装lzop

wget http://www.lzop.org/download/lzop-1.04.tar.gz
tar zxvf lzop-1.04.tar.gz -C /opt/

cd /opt/lzop-1.04/

设置下不然会报错:configure: error: LZO header files not found. Please check your installation or set the environment variable `CPPFLAGS'.
export C_INCLUDE_PATH=/opt/lzo/include
export LIBRARY_PATH=/opt/lzo/lib


./configure -enable-shared -prefix=/opt/lzop-1.04
make  && make install

lzop使用

 ./bin/lzop -f /home/hadoop/trade-service_audit_2018032223_001.log    压缩
 ./bin/lzop -dv /user/hadoop/test/trade-service_audit_2018032223_001.log.lzo 解压


收藏助手
不良信息举报
您举报文章:lzo-lzop
举报原因:
原因补充:

(最多只允许输入30个字)