hbase 使用lzo_Hadoop3.1.2 + Hbase2.2.0 设置lzo压缩算法

最新推荐文章于 2021-02-15 22:30:51 发布

weixin_39590472

最新推荐文章于 2021-02-15 22:30:51 发布

阅读量306

点赞数

文章标签： hbase 使用lzo

本文链接：https://blog.csdn.net/weixin_39590472/article/details/111517335

版权

Hadoop3.1.2 + Hbase2.2.0 设置lzo压缩算法：

写在前面，在配置hbase使用lzo算法时，在网上搜了很多文章，一般都是比较老的文章，一是版本低，二是一般都是使用hadoop-gpl-compression，hadoop-gpl-compression是一个比较老的依赖包，现已被hadoop-lzo替代，希望遇到hadoop和hbase配置lzo算法时，能有所帮助

安装lzo库

2.解压lzo库

tar -zxvf lzo-2.10.tar.gz

3.进入解压后的lzo目录，执行./configure --enable-shared

cd lzo-2.10

./configure --enable-shared -prefix=/usr/local/hadoop/lzo

4.执行make进行编译，编译完成后，执行make install进行安装

make && make install

如果没有安装lzo库，在hbase中创建表时指定compression为lzo时会报错：

ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks

at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:2314)

at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2156)

at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:2048)

at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:651)

at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)

at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)

at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103)

at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2384)

at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2377)

at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2154)

... 7 more

Caused by: java.lang.RuntimeException: native-lzo library not available

at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:355)

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98)

... 10 more

5,库文件被默认安装到了/usr/local/lib，将/usr/local/lib拷贝到/usr/lib下，或者在/usr/lib下建立软连接

cd /usr/lib

ln -s /usr/local/lib/* .

6.安装lzop wget http://www.lzop.org/download/lzop-1.04.tar.gz

tar -zxvf lzop-1.04.tar.gz

./configure -enable-shared -prefix=/usr/local/hadoop/lzop

make && make install

7.把lzop复制到/usr/bin/或建立软连接

ln -s /usr/local/hadoop/lzop/bin/lzop /usr/bin/lzop

二、安装hadoop-lzo

1.下载hadoop-lzo ，下载地址：wget https://github.com/twitter/hadoop-lzo/archive/master.zip 这是一个zip压缩包，如果想使用git下载，可以使用该链接：

2.编译hadoop-lzo源码，在编译之前如果没有安装maven需要配置maven环境，解压缩master.zip，为：hadoop-lzo-master，进入hadoop-lzo-master中，修改pom.xml中hadoop版本配置，进行maven编译

unzip master.zip

cd hadoop-lzo-master

vim pom.xml

修改hadoop.current.version为自己对应的hadoop版本，我这里是3.1.2

UTF-8

3.1.2

1.0.4

3.在hadoop-lzo-master目录中执行一下命令编译hadoop-lzo:

export CFLAGS=-m64

export CXXFLAGS=-m64

export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include #对应lzo安装的目录

export LIBRARY_PATH=/usr/local/hadoop/lzo/lib　　　　　　#对应lzo安装的目录

mvn clean package -Dmaven.test.skip=true

4.打包完成后，进入target/native/Linux-amd64-64，将libgplcompression*复制到hadoop的native中，将hadoop-lzo.xxx.jar 复制到每台hadoop的common包里

cd target/native/Linux-amd64-64

tar -cBf - -C lib . | tar -xBvf - -C ~

cp ~/libgplcompression* $HADOOP_HOME/lib/native/

cp target/hadoop-lzo-0.4.18-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/

libgplcompression*文件：

其中libgplcompression.so和libgplcompression.so.0是链接文件，指向libgplcompression.so.0.0.0，

将上面生成的libgplcompression*和target/hadoop-lzo-xxx-SNAPSHOT.jar同步到集群中的所有机器对应的目录($HADOOP_HOME/lib/native/，$HADOOP_HOME/share/hadoop/common/)。

配置hadoop环境变量

1.在$HADOOP_HOME/etc/hadoop/hadoop-env.sh文件中配置：

export LD_LIBRARY_PATH=/usr/local/lib/lzo/lib

# Extra Java CLASSPATH elements. Optional.

export HADOOP_CLASSPATH=":$HADOOP_CLASSPATH:${HADOOP_HOME}/share/hadoop/common"

export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_HOME/lib/native

2.在$HADOOP_HOME/etc/hadoop/core-site.xml加上如下配置：

io.compression.codecs

org.apache.hadoop.io.compress.GzipCodec,

org.apache.hadoop.io.compress.DefaultCodec,

com.hadoop.compression.lzo.LzoCodec,

com.hadoop.compression.lzo.LzopCodec,

org.apache.hadoop.io.compress.BZip2Codec

io.compression.codec.lzo.class

com.hadoop.compression.lzo.LzoCodec

如果没有配置，在hbase中创建表时compression指定lzo时会报错：

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103)

at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2384)

at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2377)

at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2154)

... 7 more

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:128)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:114)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:353)

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98)

... 10 more

Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec

at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:124)

... 13 more

3.在$HADOOP_HOME/etc/hadoop/mapred-site.xml加上如下配置：

mapred.compress.map.output

true

mapred.map.output.compression.codec

com.hadoop.compression.lzo.LzoCodec

mapred.child.env

LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib

mapreduce.reduce.env

LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib

mapred.child.env

LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib

将上述修改的配置文件全部同步到集群的所有机器上，并重启Hadoop集群，这样就可以在Hadoop中使用lzo。

在Hbase中配置lzo

1.将hadoop-lzo-xxx.jar复制到/hbase/lib中

cp target/hadoop-lzo-0.4.18-SNAPSHOT.jar $HBASE_HOME/lib

2.在hbase/lib下创建native文件夹，在/hbase/lib/native下创建Linux-amd64-64 -> /opt/hadoop/lib/native的软连接

ln -s /opt/hadoop/lib/native Linux-amd64-64

如图：

3.在$HBASE_HOME/conf/hbase-env.sh中添加如下配置：

export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/

4.在$HBASE_HOME/conf/hbase-site.xml中添加如下配置：

hbase.regionserver.codecs

lzo

5.启动hbase一切正常

注意：关于hadoop-gpl-compression的说明：

hadoop-lzo-xxx的前身是hadoop-gpl-compression-xxx,之前是放在google code下管理,地址:http://code.google.com/p/hadoop-gpl-compression/ .但由于协议问题后来移植到github上,也就是现在的hadoop-lzo-xxx,github,链接地址:https://github.com/kevinweil/hadoop-lzo.网上介绍hadoop lzo压缩绝大部分都是基于hadoop-gpl-compression的介绍.而hadoop-gpl-compression还是09年开发的,跟现在hadoop版本已经无法再完全兼容,会发生一些问题。因此也趟了一些坑。希望能给一些朋友一点帮助。

在使用hadoop-gpl-compression-xxx.jar时，hbase启动会报如下错：

2019-09-03 11:36:22,771 INFO [main] lzo.GPLNativeCodeLoader: Loaded native gpl library

2019-09-03 11:36:22,866 WARN [main] lzo.LzoCompressor: java.lang.NoSuchFieldError: lzoCompressLevelFunc

2019-09-03 11:36:22,866 ERROR [main] lzo.LzoCodec: Failed to load/initialize native-lzo library

2019-09-03 11:36:23,169 WARN [main] util.CompressionTest: Can't instantiate codec: lzo

org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103)

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:69)

at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:834)

at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:565)

at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:506)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180)

at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)

at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)

at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)

at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)

Caused by: java.lang.RuntimeException: native-lzo library not available

at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)

at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:355)

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98)

... 14 more

2019-09-03 11:36:23,183 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer

java.io.IOException: Compression codec lzo not supported, aborting RS construction

at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:835)