Hadoop 安装LZO压缩、创建LZO索引

编译LZO

hadoop本身并不支持lzo压缩，故需要使用twitter提供的hadoop-lzo开源组件。hadoop-lzo需依赖hadoop和lzo进行编译，编译步骤如下。

Hadoop支持LZO

0. 环境准备
   maven（下载安装，配置环境变量，修改sitting.xml加阿里云镜像）
   gcc-c++
   zlib-devel
   autoconf
   automake
   libtool
   通过yum安装即可，yum -y install gcc-c++ lzo-devel zlib-devel autoconf automake libtool

1. 下载、安装并编译LZO

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz

tar -zxvf lzo-2.10.tar.gz

cd lzo-2.10

./configure -prefix=/usr/local/hadoop/lzo/

make

make install

2. 编译hadoop-lzo源码

2.1 下载hadoop-lzo的源码，下载地址：https://github.com/twitter/hadoop-lzo/archive/master.zip
2.2 解压之后，修改pom.xml
<hadoop.current.version>2.7.2</hadoop.current.version>
2.3 声明两个临时环境变量
export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
export LIBRARY_PATH=/usr/local/hadoop/lzo/lib
2.4 编译
进入hadoop-lzo-master，执行maven编译命令
mvn package -Dmaven.test.skip=true
2.5 进入target，hadoop-lzo-0.4.21-SNAPSHOT.jar 即编译成功的hadoop-lzo组件

后续操作

1. hadoop-lzo-0.4.21.jar 拷贝到 /hadoop-2.7.2/share/hadoop/common/

2. 分发到hadoop集群的各个节点中

3. core-site.xml 添加支持压缩的参数

   <property>
   <name>io.compression.codecs</name>
   <value>
   org.apache.hadoop.io.compress.GzipCodec,
   org.apache.hadoop.io.compress.DefaultCodec,
   org.apache.hadoop.io.compress.BZip2Codec,
   org.apache.hadoop.io.compress.SnappyCodec,
   com.hadoop.compression.lzo.LzoCodec,
   com.hadoop.compression.lzo.LzopCodec
   </value>
   </property>
   
   <property>
       <name>io.compression.codec.lzo.class</name>
       <value>com.hadoop.compression.lzo.LzoCodec</value>
   </property>
   
   

4. 查看本地库压缩格式是否安装成功

[root@dw-node01 hadoop-2.7.2]# hadoop checknative
20/07/20 07:03:52 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
20/07/20 07:03:52 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /opt/hadoop-2.7.2/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /opt/hadoop-2.7.2/lib/native/libsnappy.so.1
lz4:     true revision:99
bzip2:   false 
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!

使用与测试

执行job时

-D mapreduce.output.fileoutputformat.compress=true 在reduce输出阶段使用压缩

-D mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec指定压缩格式为LZO

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount -D mapreduce.output.fileoutputformat.compress=true -D mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec /wcinput /wcoutput

LZO创建索引

hadoop jar ./share/hadoop/common/hadoop-lzo-0.4.20.jar com.hadoop.compression.lzo.DistributedLzoIndexer /wcoutput

LzopCodec与LzoCodec区别

p： plus 指升级版
lzop 会在压缩文件中添加header信息，保存元压缩数据

LzoCodec 文件后缀 .lzo_defalte
LzoPCodec 文件后缀 .lzo

.lzo_defalte 作为MR输入时无法对文件进行切片操作