个人关于hadoop使用LZO压缩主要步骤以及带来的后续问题和解决办法

hadoop-lzo安装教程请链接

https://github.com/twitter/hadoop-lzo

下载打包hadoop-lzo

https://github.com/twitter/hadoop-lzo/zipball/master

1.其中说明:首先要在本地安装lzo库,方法如下:

http://www.oberhumer.com/opensource/lzo/#download下载解压后,安装说明编译安装,建议指定安装路径如下:

tar -zxvf lzo-2.06.tar.gz -C /opt/tool/
cd /opt/tool/lzo-2.06/
mkdir /usr/local/lzo
./configure --enable-shared --prefix /usr/local/lzo
make & sudo makeinstall

2.本地库安装完成后,回头制作hadoop-lzo的jar包,即解压下载过来的hadoop-lzo-master.zip

unzip hadoop-lzo-master.zip

在目录下执行:

export CFLAGS=-m64
export CXXFLAGS=-m64
export LIBRARY_PATH=/usr/local/lzo/lib
C_INCLUDE_PATH=/usr/local/lzo/include \
LIBRARY_PATH=/usr/local/lzo/lib \
mvn clean package -Dmaven.test.skip=true
cd target/native/Linux-amd64-64
tar -cBf - -C lib . | tar -xBvf - -C ~
mv ~/libgplcompression* $HADOOP_HOME/lib/native/

3.将mvn打包的hadoop-lzo-0.4.20-SNAPSHOT.jar复制到hadoop的common目录下

cp hadoop-lzo-0.4.20-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/

4.mapreduce中间压缩的配置

在core-site.xml中添加配置

	<property>     
		<name>io.compression.codecs</name>	
		<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
	</property>
	<property>     
		<name>io.compression.codec.lzo.class</name>     
		<value>com.hadoop.compression.lzo.LzoCodec</value>
	</property>

在mapred.size中设置