hadoop支持lzo且支持分片

1.执行命令安装一些依赖组件

[root@hadoop001 ~]# yum -y install  lzo-devel  zlib-devel  gcc autoconf automake libtool

2. 下载lzo的源码包并解压

[hadoop@hadoop001 software]$ wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz
[hadoop@hadoop001 software]$ tar -zxvf lzo-2.06.tar.gz
[hadoop@hadoop001 software]$ cd lzo-2.06
[hadoop@hadoop001 lzo-2.06]$ export CFLAGS=-m64
[hadoop@hadoop001 lzo-2.06]$ ./configure -enable-shared -prefix=/home/hadoop/lzo/
[hadoop@hadoop001 lzo-2.06]$ make && make install

3.下载twitter的hadoop-lzo源码进行编译

[hadoop@hadoop001 software]$ git clone https://github.com/twitter/hadoop-lzo.git

进入hadoop-lzo目录，修改pom中hadoop的版本为自己电脑安装的版本，并添加仓库

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <hadoop.current.version>2.6.0-cdh5.15.1</hadoop.current.version>  //修改的内容
    <hadoop.old.version>1.0.4</hadoop.old.version>
  </properties>

######################
        <repository>
            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>

然后依次执行如下命令，配置环境变量：

[hadoop@hadoop001 hadoop-lzo-master]$ export CFLAGS=-m64
[hadoop@hadoop001 hadoop-lzo-master]$ export CXXFLAGS=-m64
[hadoop@hadoop001 hadoop-lzo-master]$ export C_INCLUDE_PATH=/home/hadoop/lzo/include
[hadoop@hadoop001 hadoop-lzo-master]$ export LIBRARY_PATH=/home/hadoop/lzo/lib
[hadoop@hadoop001 hadoop-lzo-master]$ mvn clean package -Dmaven.test.skip=true
[hadoop@hadoop001 hadoop-lzo-master]$ cd target/native/Linux-amd64-64
[hadoop@hadoop001 Linux-amd64-64]$ tar -cBf - -C lib . | tar -xBvf - -C ~
[hadoop@hadoop001 ~]$ cp ~/libgplcompression* $HADOOP_HOME/lib/native
[hadoop@hadoop001 hadoop-lzo-master]$ cp target/hadoop-lzo-0.4.21-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/mapreduce/lib

4.修改hadoop的配置文件
hadoop-env.sh

export LD_LIBRARY_PATH=/home/hadoop/lzo/lib

core-site.xml

<property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.GzipCodec,
           org.apache.hadoop.io.compress.DefaultCodec,
           com.hadoop.compression.lzo.LzoCodec,
           com.hadoop.compression.lzo.LzopCodec,
           org.apache.hadoop.io.compress.BZip2Codec
        </value>
</property>
<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

mapred-site.xml

<property>
    <name>mapreduce.output.fileoutputformat.compress</name>
    <value>true</value>
</property>

<property>
    <name>mapreduce.output.fileoutputformat.compress.codec</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

<property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/home/hadoop/lzo/lib</value>
</property>

修改好配置文件之后，重启hadoop
5.测试
压缩lzop,解压缩lzop -d

[hadoop@hadoop001 data]$ lzop pk.txt