hadoop配lzo压缩配置

• lzo jar包需要放到 /opt/module/hadoop/share/hadoop/common
• 分发到其他节点
• core-site.xml 中增加

<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
</property>

<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

• 分发core-site.xml 到其他节点

• 

• yarn jar /opt/module/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar \ wordcount -Dmapreduce.output.fileoutputformat.compress=true \ -Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec \ /test/baiyun/output /lzooutput

• /test/baiyun/output 中得有文件 ava.io.IOException: Unknown Job job_1622363924083_0006

• 

• /lzooutput 不能存在，否则报错FileAlreadyExistsException: Output directory hdfs://hd101:9000/lzooutput already exists

• 成功之后
  
  
  -添加索引hadoop jar /opt/module/hadoop/share/hadoop/common/hadoop-lzo-0.4.20.jar com.hadoop.compression.lzo.DistributedLzoIndexer /lzooutput