hadoop-lzo安装教程请链接
https://github.com/twitter/hadoop-lzo
下载打包hadoop-lzo
https://github.com/twitter/hadoop-lzo/zipball/master
1.其中说明:首先要在本地安装lzo库,方法如下:
http://www.oberhumer.com/opensource/lzo/#download下载解压后,安装说明编译安装,建议指定安装路径如下:
tar -zxvf lzo-2.06.tar.gz -C /opt/tool/
cd /opt/tool/lzo-2.06/
mkdir /usr/local/lzo
./configure --enable-shared --prefix /usr/local/lzo
make & sudo makeinstall
2.本地库安装完成后,回头制作hadoop-lzo的jar包,即解压下载过来的hadoop-lzo-master.zip
unzip hadoop-lzo-master.zip
在目录下执行:
export CFLAGS=-m64
export CXXFLAGS=-m64
export LIBRARY_PATH=/usr/local/lzo/lib
C_INCLUDE_PATH=/usr/local/lzo/include \
LIBRARY_PATH=/usr/local/lzo/lib \
mvn clean package -Dmaven.test.skip=true
cd target/native/Linux-amd64-64
tar -cBf - -C lib . | tar -xBvf - -C ~
mv ~/libgplcompression* $HADOOP_HOME/lib/native/
3.将mvn打包的hadoop-lzo-0.4.20-SNAPSHOT.jar复制到hadoop的common目录下
cp hadoop-lzo-0.4.20-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/
4.mapreduce中间压缩的配置
在core-site.xml中添加配置
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
在mapred.size中设置