Hadoop集群安装lzo

Hadoop安装lzo

环境要求

java-1.8
maven-3.6.3
hadoop-3.x.x

下载安装包

到自己目录下下载即可，master.zip可能需要VPN。

[root@master source]# cd /opt/source/
[root@master source]# sudo yum -y install lzo-devel zlib-devel gcc autoconf automake libtool

[root@master source]# wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz

[root@master source]# wget https://github.com/twitter/hadoop-lzo/archive/master.zip

解压压缩包

[root@master source]# tar -zxvf lzo-2.10.tar.gz -C /opt/source
[root@master source]# unzip master.zip

修改配置

[root@basecoalmine source]# vim /opt/source/hadoop-lzo-master/pom.xml
  <hadoop.current.version>3.3.2</hadoop.current.version> #自己的hadoop版本，可用hadoop version查看

编译lzo-2.10

[root@master source]# cd /opt/source/lzo-2.10/
[root@master lzo-2.10]# bash configure -prefix=/opt/source/lzo-2.10/
[root@master lzo-2.10]# make
[root@master lzo-2.10]# make install

配置环境变量

[root@master lzo-2.10]# vim /etc/profile
export C_INCLUDE_PATH=/opt/source/lzo-2.10/include
export LIBRARY_PATH=/opt/source/lzo-2.10/lib
[root@master lzo-2.10]# source /etc/profile

编译安装hadoop-lzo

[root@master lzo-2.10]# cd /opt/source/hadoop-lzo-master/

#需要等待一段时间
[root@master hadoop-lzo-master]# mvn package -Dmaven.test.skip=true
[root@master hadoop-lzo-master]# cd /target

# 复制jar包到hadoop目录
[root@master target]# cp hadoop-lzo-0.4.21-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/

core-site.xml增加配置支持LZO压缩

<property>
    <name>io.compression.codecs</name>
    <value>
        org.apache.hadoop.io.compress.GzipCodec,
        org.apache.hadoop.io.compress.DefaultCodec,
        org.apache.hadoop.io.compress.BZip2Codec,
        org.apache.hadoop.io.compress.SnappyCodec,
        com.hadoop.compression.lzo.LzoCodec,
        com.hadoop.compression.lzo.LzopCodec
    </value>
</property>

<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

mapred-site.xml 添加如下配置

<property>
    <name>mapred.child.env </name>
    <value>LD_LIBRARY_PATH=/opt/app/lzo/complie/lib</value>
</property>
<property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
</property>
<property>
    <name>mapreduce.map.output.compress.codec</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

最后重启hadoop集群即可。

##安装maven可查看这篇文章

链接: link .