1。安装LZO
sudo apt-get install liblzo2-dev
或者下载lzo2
http://www.oberhumer.com/opensource/lzo/download/.
./configure --enable-shared
make
make install
确认安装成功
2。下载hadoop-gpl-compression源码
3.编译
4.拷贝生成的jar到指定目录
5.拷贝native library
6.修改hadoop配置文件core-site.xml
7.修改hadoop配置文件mapred-site.xml
8下载一个windows的lzop的客户端,生存一个test.lzo
9.生存索引
单机版
集群版
10.执行一个mapred程序,如wordcount
添加参数"-inputformat com.hadoop.mapred.DeprecatedLzoTextInputFormat"
sudo apt-get install liblzo2-dev
或者下载lzo2
http://www.oberhumer.com/opensource/lzo/download/.
./configure --enable-shared
make
make install
确认安装成功
% ls -l /usr/lib*/liblzo2*
-rw-r--r-- 1 root root 171056 Mar 20 2006 /usr/lib/liblzo2.a
lrwxrwxrwx 1 root root 16 Feb 17 2007 /usr/lib/liblzo2.so -> liblzo2.so.2.0.0*
lrwxrwxrwx 1 root root 16 Feb 17 2007 /usr/lib/liblzo2.so.2 -> liblzo2.so.2.0.0*
-rwxr-xr-x 1 root root 129067 Mar 20 2006 /usr/lib/liblzo2.so.2.0.0*
-rw-r--r-- 1 root root 208494 Mar 20 2006 /usr/lib64/liblzo2.a
lrwxrwxrwx 1 root root 16 Feb 17 2007 /usr/lib64/liblzo2.so -> liblzo2.so.2.0.0*
lrwxrwxrwx 1 root root 16 Feb 17 2007 /usr/lib64/liblzo2.so.2 -> liblzo2.so.2.0.0*
-rwxr-xr-x 1 root root 126572 Mar 20 2006 /usr/lib64/liblzo2.so.2.0.0*
2。下载hadoop-gpl-compression源码
http://code.google.com/a/apache-extras.org/p/hadoop-gpl-compression/downloads/list
3.编译
export JAVA_HOME=/path/to/64bit/jdk
export CFLAGS=-m64
export CXXFLAGS=-m64
ant compile-native tar
4.拷贝生成的jar到指定目录
cp build/hadoop-gpl-compression-0.1.0-dev/hadoop-gpl-compression-0.1.0-dev.jar /path/to/hadoop/dist/lib
5.拷贝native library
tar -cBf - -C build/hadoop-gpl-compression-0.1.0-dev/lib/native . | tar -xBvf - -C /path/to/hadoop/dist/lib/native
6.修改hadoop配置文件core-site.xml
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
7.修改hadoop配置文件mapred-site.xml
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
8下载一个windows的lzop的客户端,生存一个test.lzo
9.生存索引
单机版
hadoop jar /path/to/your/hadoop-lzo.jar com.hadoop.compression.lzo.LzoIndexer big_file.lzo
集群版
hadoop jar /path/to/your/hadoop-lzo.jar com.hadoop.compression.lzo.DistributedLzoIndexer big_file.lzo
10.执行一个mapred程序,如wordcount
添加参数"-inputformat com.hadoop.mapred.DeprecatedLzoTextInputFormat"