hbase 使用lzo_Hadoop3.1.2 + Hbase2.2.0 设置lzo压缩算法

Hadoop3.1.2 + Hbase2.2.0 设置lzo压缩算法:

写在前面,在配置hbase使用lzo算法时,在网上搜了很多文章,一般都是比较老的文章,一是版本低,二是一般都是使用hadoop-gpl-compression,hadoop-gpl-compression是一个比较老的依赖包,现已被hadoop-lzo替代,希望遇到hadoop和hbase配置lzo算法时,能有所帮助

安装lzo库

2.解压lzo库

tar -zxvf lzo-2.10.tar.gz

3.进入解压后的lzo目录,执行./configure  --enable-shared

cd lzo-2.10

./configure --enable-shared -prefix=/usr/local/hadoop/lzo

4.执行make进行编译,编译完成后,执行make install进行安装

make && make install

如果没有安装lzo库,在hbase中创建表时指定compression为lzo时会报错:

ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks

at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:2314)

at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2156)

at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:2048)

at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:651)

at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)

at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)

at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103)

at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2384)

at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2377)

at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2154)

... 7 more

Caused by: java.lang.RuntimeException: native-lzo library not available

at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:355)

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98)

... 10 more

5,库文件被默认安装到了/usr/local/lib,将/usr/local/lib拷贝到/usr/lib下,或者在/usr/lib下建立软连接

cd /usr/lib

ln -s /usr/local/lib/* .

6.安装lzop  wget http://www.lzop.org/download/lzop-1.04.tar.gz

tar -zxvf lzop-1.04.tar.gz

./configure -enable-shared -prefix=/usr/local/hadoop/lzop

make && make install

7.把lzop复制到/usr/bin/或建立软连接

ln -s /usr/local/hadoop/lzop/bin/lzop /usr/bin/lzop

二、安装hadoop-lzo

1.下载hadoop-lzo ,下载地址:wget https://github.com/twitter/hadoop-lzo/archive/master.zip 这是一个zip压缩包,如果想使用git下载,可以使用该链接:

2.编译hadoop-lzo源码,在编译之前如果没有安装maven需要配置maven环境,解压缩master.zip,为:hadoop-lzo-master,进入hadoop-lzo-master中,修改pom.xml中hadoop版本配置,进行maven编译

unzip master.zip

cd hadoop-lzo-master

vim pom.xml

修改hadoop.current.version为自己对应的hadoop版本,我这里是3.1.2

UTF-8

3.1.2

1.0.4

3.在hadoop-lzo-master目录中执行一下命令编译hadoop-lzo:

export CFLAGS=-m64

export CXXFLAGS=-m64

export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include #对应lzo安装的目录

export LIBRARY_PATH=/usr/local/hadoop/lzo/lib      #对应lzo安装的目录

mvn clean package -Dmaven.test.skip=true

4.打包完成后,进入target/native/Linux-amd64-64,将libgplcompression*复制到hadoop的native中,将hadoop-lzo.xxx.jar 复制到每台hadoop的common包里

cd target/native/Linux-amd64-64

tar -cBf - -C lib . | tar -xBvf - -C ~

cp ~/libgplcompression* $HADOOP_HOME/lib/native/

cp target/hadoop-lzo-0.4.18-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/

libgplcompression*文件:

其中libgplcompression.so和libgplcompression.so.0是链接文件,指向libgplcompression.so.0.0.0,

将上面生成的libgplcompression*和target/hadoop-lzo-xxx-SNAPSHOT.jar同步到集群中的所有机器对应的目录($HADOOP_HOME/lib/native/,$HADOOP_HOME/share/hadoop/common/)。

配置hadoop环境变量

1.在$HADOOP_HOME/etc/hadoop/hadoop-env.sh文件中配置:

export LD_LIBRARY_PATH=/usr/local/lib/lzo/lib

# Extra Java CLASSPATH elements. Optional.

export HADOOP_CLASSPATH=":$HADOOP_CLASSPATH:${HADOOP_HOME}/share/hadoop/common"

export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_HOME/lib/native

2.在$HADOOP_HOME/etc/hadoop/core-site.xml加上如下配置:

io.compression.codecs

org.apache.hadoop.io.compress.GzipCodec,

org.apache.hadoop.io.compress.DefaultCodec,

com.hadoop.compression.lzo.LzoCodec,

com.hadoop.compression.lzo.LzopCodec,

org.apache.hadoop.io.compress.BZip2Codec

io.compression.codec.lzo.class

com.hadoop.compression.lzo.LzoCodec

如果没有配置,在hbase中创建表时compression指定lzo时会报错:

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103)

at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2384)

at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2377)

at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2154)

... 7 more

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:128)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:114)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:353)

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98)

... 10 more

Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec

at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:124)

... 13 more

3.在$HADOOP_HOME/etc/hadoop/mapred-site.xml加上如下配置:

mapred.compress.map.output

true

mapred.map.output.compression.codec

com.hadoop.compression.lzo.LzoCodec

mapred.child.env

LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib

mapreduce.reduce.env

LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib

mapred.child.env

LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib

将上述修改的配置文件全部同步到集群的所有机器上,并重启Hadoop集群,这样就可以在Hadoop中使用lzo。

在Hbase中配置lzo

1.将hadoop-lzo-xxx.jar复制到/hbase/lib中

cp target/hadoop-lzo-0.4.18-SNAPSHOT.jar $HBASE_HOME/lib

2.在hbase/lib下创建native文件夹,在/hbase/lib/native下创建Linux-amd64-64 -> /opt/hadoop/lib/native的软连接

ln -s /opt/hadoop/lib/native Linux-amd64-64

如图:

3.在$HBASE_HOME/conf/hbase-env.sh中添加如下配置:

export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/

4.在$HBASE_HOME/conf/hbase-site.xml中添加如下配置:

hbase.regionserver.codecs

lzo

5.启动hbase一切正常

注意:关于hadoop-gpl-compression的说明:

hadoop-lzo-xxx的前身是hadoop-gpl-compression-xxx,之前是放在google code下管理,地址:http://code.google.com/p/hadoop-gpl-compression/ .但由于协议问题后来移植到github上,也就是现在的hadoop-lzo-xxx,github,链接地址:https://github.com/kevinweil/hadoop-lzo.网上介绍hadoop lzo压缩绝大部分都是基于hadoop-gpl-compression的介绍.而hadoop-gpl-compression还是09年开发的,跟现在hadoop版本已经无法再完全兼容,会发生一些问题。因此也趟了一些坑。希望能给一些朋友一点帮助。

在使用hadoop-gpl-compression-xxx.jar时,hbase启动会报如下错:

2019-09-03 11:36:22,771 INFO [main] lzo.GPLNativeCodeLoader: Loaded native gpl library

2019-09-03 11:36:22,866 WARN [main] lzo.LzoCompressor: java.lang.NoSuchFieldError: lzoCompressLevelFunc

2019-09-03 11:36:22,866 ERROR [main] lzo.LzoCodec: Failed to load/initialize native-lzo library

2019-09-03 11:36:23,169 WARN [main] util.CompressionTest: Can't instantiate codec: lzo

org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103)

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:69)

at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:834)

at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:565)

at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:506)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180)

at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)

at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)

at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)

at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)

Caused by: java.lang.RuntimeException: native-lzo library not available

at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:355)

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98)

... 14 more

2019-09-03 11:36:23,183 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer

java.io.IOException: Compression codec lzo not supported, aborting RS construction

at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:835)

at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:565)

at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:506)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180)

at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)

at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)

at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)

at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)

2019-09-03 11:36:23,184 ERROR [main] master.HMasterCommandLine: Master exiting

java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.

at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3187)

at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)

at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)

at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)

at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)

Caused by: java.io.IOException: Compression codec lzo not supported, aborting RS construction

at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:835)

at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:565)

at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:506)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180)

... 5 more

当删除hadoop-gpl-compression-xxx.jar时,替换为hadoop-lzo.xxx.jar后,再启动hbase,一切正常:

2019-09-03 14:57:43,755 INFO [main] lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries

2019-09-03 14:57:43,758 INFO [main] lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 5dbdddb8cfb544e58b4e0b9664b9d1b66657faf5]

2019-09-03 14:57:43,983 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available

2019-09-03 14:57:44,088 INFO [main] compress.CodecPool: Got brand-new compressor [.lzo_deflate]

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值