Hadoop支持LZO

最新推荐文章于 2022-08-16 16:53:38 发布

bangshuangyi8424

最新推荐文章于 2022-08-16 16:53:38 发布

阅读量86

点赞数

原文链接：http://www.cnblogs.com/allthewayforward/p/11131218.html

版权

0. 环境准备

maven（下载安装，配置环境变量，修改sitting.xml加阿里云镜像）
gcc-c++
zlib-devel
autoconf
automake
libtool
通过yum安装即可

yum -y install gcc-c++ lzo-devel zlib-devel autoconf automake libtool

1. 下载、安装并编译LZO

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz
tar -zxvf lzo-2.10.tar.gz
cd lzo-2.10
./configure -prefix=/usr/local/hadoop/lzo/
make
make install

2. 编译hadoop-lzo源码

2.1 下载hadoop-lzo的源码

下载地址：https://github.com/twitter/hadoop-lzo/archive/master.zip

2.2 解压之后，修改pom.xml

<hadoop.current.version>2.7.2</hadoop.current.version>

2.3 声明两个临时环境变量

 export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
 export LIBRARY_PATH=/usr/local/hadoop/lzo/lib

进入hadoop-lzo-master，执行maven编译命令

mvn package -Dmaven.test.skip=true

2.5 进入target，将hadoop-lzo-0.4.21-SNAPSHOT.jar放到hadoop的classpath下

如${HADOOP_HOME}/share/hadoop/common

2.6 修改core-site.xml增加配置支持LZO压缩

<configuration>
        <property>
            <name>io.compression.codecs</name>
            <value>
            org.apache.hadoop.io.compress.GzipCodec,
            org.apache.hadoop.io.compress.DefaultCodec,
            org.apache.hadoop.io.compress.BZip2Codec,
            org.apache.hadoop.io.compress.SnappyCodec,
            com.hadoop.compression.lzo.LzoCodec,
            com.hadoop.compression.lzo.LzopCodec
            </value>
        </property>
        <property>
            <name>io.compression.codec.lzo.class</name>
            <value>com.hadoop.compression.lzo.LzoCodec</value>
        </property>
</configuration>

转载于:https://www.cnblogs.com/allthewayforward/p/11131218.html