Hadoop之LZO压缩配置

一、下载地址

1、先下载lzo的jar项目
https://github.com/twitter/hadoop-lzo/archive/master.zip

2、下载后的文件名是hadoop-lzo-master，它是一个zip格式的压缩包，先进行解压，然后用maven编译。生成hadoop-lzo-0.4.20.jar。

二、配置lzo

1、将编译好后的hadoop-lzo-0.4.20.jar
放入hadoop-2.7.2/share/hadoop/common/

[root@hadoop102 common]$ pwd
/opt/module/hadoop-2.7.2/share/hadoop/common
[root@hadoop102 common]$ ls
hadoop-lzo-0.4.20.jar

2、同步hadoop-lzo-0.4.20.jar到hadoop103、hadoop104

[root@hadoop102 common]$ xsync hadoop-lzo-0.4.20.jar

3、core-site.xml增加配置支持LZO压缩

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<!--添加如下配置	-->
				<property>
				<name>io.compression.codecs</name>
				<value>
				org.apache.hadoop.io.compress.GzipCodec,
				org.apache.hadoop.io.compress.DefaultCodec,
				org.apache.hadoop.io.compress.BZip2Codec,
				org.apache.hadoop.io.compress.SnappyCodec,
				com.hadoop.compression.lzo.LzoCodec,
				com.hadoop.compression.lzo.LzopCodec
				</value>
				</property>
				<property>
				    <name>io.compression.codec.lzo.class</name>
				    <value>com.hadoop.compression.lzo.LzoCodec</value>
				</property>
</configuration>

4、同步core-site.xml到hadoop103、hadoop104

[root@hadoop102 hadoop]$ xsync core-site.xml

5、启动集群

[root@hadoop102 hadoop-2.7.2]$ sbin/start-dfs.sh
[root@hadoop103 hadoop-2.7.2]$ sbin/start-yarn.sh