一、问题发现
近期,公司新上线了一个基于hbase的应用,在该应用中会涉及到大量hbase的scan行为。运行一阵子后,该应用间歇性的会出现scan非常缓慢的现象。公司还有大量其他基于hbase的应用,每次都只有新应用有这种现象。
唯一与以往应用的区别是新应用中有reversedScan的使用,难道是reversescan引起的?
增加对应用的jstack监控,捕捉到慢scan发生时应用的jstack,发现了一些眉目。jstack中大量线程处于block状态。
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1319)
- waiting to lock <0x00000000876a58d8> (a java.lang.Object)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1177)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:294)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:130)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:55)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:201)
at org.apache.hadoop.hbase.client.ReversedClientScanner.nextScanner(ReversedClientScanner.java:124)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135)
at org.apache.hadoop.hbase.client.ReversedClientScanner.<init>(ReversedClientScanner.java:62)
at org.apache.hadoop.hbase.client.HTable.getScanner(