hbase集群写入的TPS慢分析

异常日志:

04 Jul 2019 08:31:01,333 INFO [htable-pool206-t448] (org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit:1193) - #1, table=TTrace_20

190701, attempt=17/35 failed=348ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above mem

store limit, regionName=TTrace_20190701,c|,1561906829940.f7862508344d68a23966f52a4453497f., server=t-bj-hdp-14,16020,1561741819967, memstoreSize=1612330976, blockingMemStoreSize=1610612736

at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:3750)

at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2924)

at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)

at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:715)

at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:677)

at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2054)

at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)

at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

at java.lang.Thread.run(Thread.java:745)

on t-bj-hdp-14,16020,1561741819967, tracking started null, retrying after=20027ms, replay=348ops

 

hbase相关参数:

hbase.hregion.memstore.flush.size             
每次update之后会检查此region memstore是否达到这个大小，达到之后就会请求flush。默认值为64MB
hbase.hregion.memstore.block.multiplier
当单个region 的memstore size达到hbase.hregion.memstore.flush.size*hbase.hregion.memstore.block.multiplier时，本region的update会被block。
hbase.regionserver.global.memstore.upperLimit
当RS memstore超过这个比率，RS的所有update会被block。
具体见reclaimMemStoreMemory()，这几乎是个跟gc一样的stop word上限。
hbase.regionserver.global.memstore.lowerLimit
当用memstore高于这个值时，即使没有region要求flush，MemstoreFlusher也会挑一个region去flush然后sleep一会，这个参数影响不大。
见地址:https://blog.csdn.net/kalaamong/article/details/7324894

分析:

hbase集群中相关配置

hbase.hregion.memstore.flush.size : 256MB

hbase.hregion.memstore.block.multiplier : 6

hbase.hregion.memstore.flush.size*hbase.hregion.memstore.block.multiplier=256MB*6 : 1.536G

当单个region 的memstore size达到hbase.hregion.memstore.flush.size*hbase.hregion.memstore.block.multiplier时，本region的update会被block。

hbase.regionserver.global.memstore.upperLimit

当RS memstore超过这个比率，RS的所有update会被block

hbase.regionserver.global.memstore.lowerLimit

当用memstore高于这个值时，即使没有region要求flush，MemstoreFlusher也会挑一个region去flush然后sleep一会，这个参数影响不大。

memstoreSize=1612330976, blockingMemStoreSize=1610612736

1612330976/1024/1024/1024=1.5016G

1610612736/1024/1024/1024=1.5G