项目经验之HADOOP支持LZO压缩配置

1.hadoop-lzo编译

hadoop本身并不支持lzo压缩，故需要使用twitter提供的hadoop-lzo开源组件。hadoop-lzo需要依赖hadoop和lzo进行编译，编译步骤如下

Hadoop支持LZO

0. 环境准备
maven（下载安装，配置环境变量，修改sitting.xml加阿里云镜像）
gcc-c++
zlib-devel
autoconf
automake
libtool
通过yum安装即可，yum -y install gcc-c++ lzo-devel zlib-devel autoconf automake libtool

1. 下载、安装并编译LZO

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz

tar -zxvf lzo-2.10.tar.gz

cd lzo-2.10

./configure -prefix=/usr/local/hadoop/lzo/

make

make install

2. 编译hadoop-lzo源码

2.1 下载hadoop-lzo的源码，下载地址：https://github.com/twitter/hadoop-lzo/archive/master.zip
2.2 解压之后，修改pom.xml
    <hadoop.current.version>3.1.3</hadoop.current.version>
2.3 声明两个临时环境变量
     export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
     export LIBRARY_PATH=/usr/local/hadoop/lzo/lib 
2.4 编译
    进入hadoop-lzo-master，执行maven编译命令
    mvn package -Dmaven.test.skip=true
2.5 进入target，hadoop-lzo-0.4.21-SNAPSHOT.jar 即编译成功的hadoop-lzo组件

2.将编译好后的hadoop-lzo-0.4.20.jar放入hadoop-3.1.3/share/hadoop/common/

3.同步hadoop-lzo-0.4.20.jar到hadoop103、hadoop104等其他节点

xsync hadoop-lzo-0.4.20.jar

4.core-site.xml增加配置支持LZO压缩

    <property>
        <name>io.compression.codecs</name>
        <value>
            org.apache.hadoop.io.compress.GzipCodec,
            org.apache.hadoop.io.compress.DefaultCodec,
            org.apache.hadoop.io.compress.BZip2Codec,
            org.apache.hadoop.io.compress.SnappyCodec,
            com.hadoop.compression.lzo.LzoCodec,
            com.hadoop.compression.lzo.LzopCodec
        </value>
    </property>

    <property>
        <name>io.compression.codec.lzo.class</name>
        <value>com.hadoop.compression.lzo.LzoCodec</value>
    </property>

5.同步core-site.xml到集群中其他节点

xsync core-site.xml

6.启动及查看集群

start-dfs.sh
start-yarn.sh

7.测试数据准备

hdfs dfs -mkdir /input
hdfs dfs -put README.txt /input

8.测试压缩

[root@hadoop102 hadoop313]# hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec  /input /output

LZO创建索引

1.创建LZO文件的索引

LZO压缩文件的可切片特性依赖于其索引，故我们需要手动为LZO压缩文件创建索引。若无索引，则LZO文件的切片只有一个。

hadoop jar /path/to/your/hadoop-lzo.jar com.hadoop.compression.lzo.DistributedLzoIndexer big_file.lzo

2.测试

• 将bigtable.lzo（214M）上传到集群的根目录

hdfs dfs -mkdir /input
hdfs dfs -put bigtable.lzo /input

• 执行wordcount程序

hadoop jar /opt/soft/hadoop313/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount -Dmapreduce.job.inputformat.class=com.hadoop.mapreduce.LzoTextInputFormat /input /output1
由于没有创建索引，LZO文件不支持切片，所以此时214M的LZO文件的切片为1，number of splits:1

• 对上传文件创建索引

hadoop jar /opt/soft/hadoop313/share/hadoop/common/hadoop-lzo-0.4.20.jar  com.hadoop.compression.lzo.DistributedLzoIndexer /input/bigtable.lzo

• 再次执行wordcount程序

hadoop jar /opt/soft/hadoop313/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount -Dmapreduce.job.inputformat.class=com.hadoop.mapreduce.LzoTextInputFormat /input /output2
此时214M的LZO文件的切片为2，number of splits:2