前置要求
1.安装好hadoop
2.安装好jdk和maven(maven提前配置阿里云连接,不然慢死!)
3.安装前置库
yum -y install lzo-devel zlib-devel gcc autoconf automake libtool
lzo使用流程
下载及安装
# 下载
wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz
# 解压
tar -zxvf lzo-2.10.tar.gz
cd lzo-2.10
# 配置前缀
./configure -prefix=/usr/local/hadoop/lzo/
# 安装
make && make install
编译hadoop-lzo源码
下载hadoop-lzo的源码,下载地址:https://github.com/twitter/hadoop-lzo/archive/master.zip
解压之后,修改pom.xml
<hadoop.current.version>2.6.0-cdh5.16.2</hadoop.current.version>
注意!cdh版本需要配置加下面的仓库
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
<repository>
<id>aliyun</id>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</repository>
声明两个临时环境变量
export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
export LIBRARY_PATH=/usr/local/hadoop/lzo/lib
进入hadoop-lzo-master,执行maven编译命令
mvn package -Dmaven.test.skip=true
进入target,hadoop-lzo-0.4.21-SNAPSHOT.jar 即编译成功的hadoop-lzo组件
编译好的文件改名hadoop-lzo-0.4.20.jar 放入到 $HADOOP_HOME/share/hadoop/common/下面然后进行分发
hadoop配置文件
添加下面内容
core-site.xml
<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
mapred-site.mxl
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
之后记得重启集群
测试lzo
本测试提前下载了lzop,提前将文件进行了压缩,再传到hdfs上的
yum -y install lzop
通过python3生成测试数据(测试功能为wc):
import random
filename = 'test.txt'
with open(filename, 'w') as file_object:
for i in range(1000000000):
file_object.write(str(random.randint(1, 100000))+"\t"+str(random.randint(1, 5))+"\n")
上传压缩完成的lzo数据
lzop test.txt
hadoop fs -put test.txt.lzo /wc/text.txt.lzo
无索引测试
hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar wordcount \
-Dmapreduce.map.output.compress=true \
-Dmapreduce.map.output.compress.codec=com.hadoop.compression.lzo.LzopCodec \
-Dmapreduce.output.fileoutputformat.compress=true \
-Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec \
/wc/test.txt.lzo /wc/out
20/04/10 17:39:51 INFO mapreduce.JobSubmitter: number of splits:1
有索引测试
hadoop jar hadoop-lzo-0.4.21-SNAPSHOT.jar com.hadoop.compression.lzo.LzoIndexer /wc/test.txt.lzo
hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar wordcount \
-Dmapreduce.job.inputformat.class=com.hadoop.mapreduce.LzoTextInputFormat \
-Dmapreduce.map.output.compress=true \
-Dmapreduce.map.output.compress.codec=com.hadoop.compression.lzo.LzopCodec \
-Dmapreduce.output.fileoutputformat.compress=true \
-Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec \
/wc/test.txt.lzo /wc/out
20/04/10 18:18:46 INFO input.FileInputFormat: Total input paths to process : 1
20/04/10 18:18:46 INFO mapreduce.JobSubmitter: number of splits:4