Hadoop配置LZO

前置要求

• 安装好hadoop
• 安装好jdk和maven(maven提前配置阿里云连接，不然慢死！)
• 安装前置库

  yum -y install  lzo-devel  zlib-devel  gcc autoconf automake libtool
  

lzo使用流程

• 下载及安装lzo

  1、下载
  wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz
  2、解压
  tar -zxvf lzo-2.10.tar.gz
  cd lzo-2.10
  3、配置前缀
  ./configure -prefix=/usr/local/hadoop/lzo
  4、安装
  make && make install
  

编译hadoop-lzo源码

• 下载修改pom

  1、下载hadoop-lzo的源码，下载地址：https://github.com/twitter/hadoop-lzo/archive/master.zip
  2、解压之后，修改pom.xml
  <hadoop.current.version>2.6.0-cdh5.16.2</hadoop.current.version>
  注意！cdh版本需要配置加下面的仓库
  <repository>
        <id>cloudera</id>
        <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
  </repository>
  
  <repository>
        <id>aliyun</id>
        <url>http://maven.aliyun.com/nexus/content/groups/public</url>
  </repository>
  

• 编译源码

  1、声明两个临时环境变量
  export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
  export LIBRARY_PATH=/usr/local/hadoop/lzo/lib
  
  2、进入hadoop-lzo-master，执行maven编译命令：mvn package -Dmaven.test.skip=true
  
  3、进入target，hadoop-lzo-0.4.21-SNAPSHOT.jar 即编译成功的hadoop-lzo组件
  4、编译好的文件改名hadoop-lzo-0.4.21.jar 放入到 $HADOOP_HOME/share/hadoop/common/下面然后进行分发
  

修改hadoop配置文件

• 修改配置文件

  vi core-site.xml
  
  <property>
  <name>io.compression.codecs</name>
  <value>
  org.apache.hadoop.io.compress.GzipCodec,
  org.apache.hadoop.io.compress.DefaultCodec,
  org.apache.hadoop.io.compress.BZip2Codec,
  org.apache.hadoop.io.compress.SnappyCodec,
  com.hadoop.compression.lzo.LzoCodec,
  com.hadoop.compression.lzo.LzopCodec
  </value>
  </property>
  
  <property>
      <name>io.compression.codec.lzo.class</name>
      <value>com.hadoop.compression.lzo.LzoCodec</value>
  </property>
  
  vi mapred-site.mxl
  <property>
      <name>mapreduce.map.output.compress</name>
      <value>true</value>
  </property>
  <property>
      <name>mapreduce.map.output.compress.codec</name>
      <value>com.hadoop.compression.lzo.LzoCodec</value>
  </property>
  

  注意：重启集群

测试lzo

1. 通过python3生成测试数据(测试功能为wc)

   import random
   filename = 'test.txt'
   with open(filename, 'w') as file_object:
       for i in range(1000000000):
           file_object.write(str(random.randint(1, 100000))+"\t"+str(random.randint(1, 5))+"\n")
   

2. 安装lzop，将文件进行了压缩，再传到hdfs上的

   yum -y install lzop
   lzop test.txt
   

3. 无索引测试

   hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar wordcount \
   	-Dmapreduce.map.output.compress=true \
   	-Dmapreduce.map.output.compress.codec=com.hadoop.compression.lzo.LzopCodec \
   	-Dmapreduce.output.fileoutputformat.compress=true \
   	-Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec \
   	/home/hadoop/data/test.txt.lzo /home/hadoop/data/out
   	
   20/04/10 17:39:51 INFO mapreduce.JobSubmitter: number of splits:1
   

4. 有索引测试

   hadoop jar hadoop-lzo-0.4.21-SNAPSHOT.jar com.hadoop.compression.lzo.LzoIndexer /wc/test.txt.lzo
   
   hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar wordcount \
   	-Dmapreduce.job.inputformat.class=com.hadoop.mapreduce.LzoTextInputFormat \
   	-Dmapreduce.map.output.compress=true \
   	-Dmapreduce.map.output.compress.codec=com.hadoop.compression.lzo.LzopCodec \
   	-Dmapreduce.output.fileoutputformat.compress=true \
   	-Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec \
   	/home/hadoop/data/test.txt.lzo /home/hadoop/data/out
   	
   20/04/10 18:18:46 INFO input.FileInputFormat: Total input paths to process : 1
   20/04/10 18:18:46 INFO mapreduce.JobSubmitter: number of splits:2