解决Hbase KeyValue size too large或 Cell with size 25000046 exceeds limit of 10485760 bytes at的异常。
HBase默认一个cell(单元格)只能存10M,有一些需求一个cell存大于10M的数据,比如将BitSet、BitMap对象存在HBase上。BitSet或BitMap存一亿个整数只需要11.9M,如存Integer需要381M,总之就是能节省很多空间。下一篇文章会讲BitSet、BitMap的算法这里就不多讲了,今天主要讲HBase一个cell存储超过10M以上的文件(20M、50M、100M)。HBase数据是存在HDFS上,存一些大文件是没有问题的,性能可能会差一点,但可以满足我们的需求,别的数据库暂时好像不满足这个需求。
先来看一下报错的几种情况:
#情况一 这种情况是什么都没有配置的或程序中没有设置hbase.client.keyvalue.maxsize
java.lang.IllegalArgumentException: KeyValue size too large
at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:952)
at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:940)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:545)
at
#情况二 这种情况是程序或Hbase只设置了hbase.client.keyvalue.maxsize
Exception in thread "main"org.apache.hadoop.hbase.DoNotRetryIOException:
org.apache.hadoop.hbase.DoNotRetryIOException: Cell with size 25000046 exceeds limit of 10485760 bytes at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:944)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2792)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42000)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
1、首先配置HBase的hbase-site.xml文件
<!--两个的值应该保持一致,老版HBase1.2.4只用client就可以-->
<!--如果两个值为0或更小,则禁用检查,个人认为是不推荐的,如果一个单元格存1G数据,那性能就会大打折扣-->
<!--我目前的需求不会大于50M,值为:52428800-->
<property>
<name>hbase.client.keyvalue.maxsize</name>
<value>52428800</value>
</property>
<property>
<name>hbase.server.keyvalue.maxsize</name>
<value>52428800</value>
</property>
2、创建Hbase链接的配置
//java scala 通用
val conf = HBaseConfiguration.create()
conf.set(HConstants.ZOOKEEPER_QUORUM, "CentOS");
conf.set(HConstants.ZOOKEEPER_CLIENT_PORT,"2181");
// Hbase每个单元格数据最大50M,如需更大需修改HBase配置
conf.set("hbase.client.keyvalue.maxsize","52428800");
// 一般程序中配置client就可以,不行在配置server参数
//conf.set("hbase.server.keyvalue.maxsize","52428800");
Connection conn = ConnectionFactory.createConnection(conf);
接下就可以操作了,有什么疑问可以给我留言
官网摘要
-
hbase.client.keyvalue.maxsize
DescriptionSpecifies the combined maximum allowed size of a KeyValue instance. This is to set an upper boundary for a single entry saved in a storage file. Since they cannot be split it helps avoiding that a region cannot be split any further because the data is too large. It seems wise to set this to a fraction of the maximum region size. Setting it to zero or less disables the check.Default
10485760
-
hbase.server.keyvalue.maxsize
DescriptionMaximum allowed size of an individual cell, inclusive of value and all key components. A value of 0 or less disables the check. The default value is 10MB. This is a safety setting to protect the server from OOM situations. Default
10485760
官网地址:https://hbase.apache.org/book.html#important_configurations
HBase的具体使用,可以参考这片文章:https://blog.csdn.net/z1987865446/article/details/109061984