单元主服务器解决性能瓶颈的方法,HBase

12.4.2.4  jstack

jstack 是一个最重要(除了看Log)的java工具，可以看到具体的Java进程的在做什么。可以先用Jps看到进程的Id,然后就可以用jstack。他会按线程的创建顺序显示线程的列表，还有这个线程在做什么。下面是例子：

这个主线程是一个RegionServer正在等master返回什么信息。

"regionserver60020" prio=10 tid=0x0000000040ab4000 nid=0x45cf waiting on condition [0x00007f16b6a96000..0x00007f16b6a96a70]

java.lang.Thread.State: TIMED_WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x00007f16cd5c2f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)

at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:647)

at java.lang.Thread.run(Thread.java:619)

The MemStore flusher thread that is currently flushing to a file:

"regionserver60020.cacheFlusher" daemon prio=10 tid=0x0000000040f4e000 nid=0x45eb in Object.wait() [0x00007f16b5b86000..0x00007f16b5b87af0]

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:485)

at org.apache.hadoop.ipc.Client.call(Client.java:803)

- locked <0x00007f16cb14b3a8> (a org.apache.hadoop.ipc.Client$Call)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:221)

at $Proxy1.complete(Unknown Source)

at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)

at $Proxy1.complete(Unknown Source)

at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3390)

- locked <0x00007f16cb14b470> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)

at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3304)

at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)

at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)

at org.apache.hadoop.hbase.io.hfile.HFile$Writer.close(HFile.java:650)

at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:853)

at org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:467)

- locked <0x00007f16d00e6f08> (a java.lang.Object)

at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:427)

at org.apache.hadoop.hbase.regionserver.Store.access$100(Store.java:80)

at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:1359)

at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:907)

at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:834)

at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:786)

at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:250)

at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:224)

at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)

一个处理线程是在等一些东西(例如put, delete, scan...):

"IPC Server handler 16 on 60020" daemon prio=10 tid=0x00007f16b011d800 nid=0x4a5e waiting on condition [0x00007f16afefd000..0x00007f16afefd9f0]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x00007f16cd3f8dd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)

有一个线程正在忙，在递增一个counter(这个阶段是正在创建一个scanner来读最新的值):

"IPC Server handler 66 on 60020" daemon prio=10 tid=0x00007f16b006e800 nid=0x4a90 runnable [0x00007f16acb77000..0x00007f16acb77cf0]

java.lang.Thread.State: RUNNABLE

at org.apache.hadoop.hbase.regionserver.KeyValueHeap.(KeyValueHeap.java:56)

at org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:79)

at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)

at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.(HRegion.java:2209)

at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)

at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)

at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)

at org.apache.hadoop.hbase.regionserver.HRegion.getLastIncrement(HRegion.java:2875)

at org.apache.hadoop.hbase.regionserver.HRegion.incrementColumnValue(HRegion.java:2978)

at org.apache.hadoop.hbase.regionserver.HRegionServer.incrementColumnValue(HRegionServer.java:2433)

at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)

at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)

还有一个线程在从HDFS获取数据。

"IPC Client (47) connection to sv4borg9/10.4.24.40:9000 from hadoop" daemon prio=10 tid=0x00007f16a02d0000 nid=0x4fa3 runnable [0x00007f16b517d000..0x00007f16b517dbf0]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

- locked <0x00007f17d5b68c00> (a sun.nio.ch.Util$1)

- locked <0x00007f17d5b68be8> (a java.util.Collections$UnmodifiableSet)

- locked <0x00007f1877959b50> (a sun.nio.ch.EPollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)

at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)

at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)

at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)

at java.io.FilterInputStream.read(FilterInputStream.java:116)

at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:304)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

at java.io.BufferedInputStream.read(BufferedInputStream.java:237)

- locked <0x00007f1808539178> (a java.io.BufferedInputStream)

at java.io.DataInputStream.readInt(DataInputStream.java:370)

at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:569)

at org.apache.hadoop.ipc.Client$Connection.run(Client.java:477)

这里是一个RegionServer死了，master正在试着恢复。

"LeaseChecker" daemon prio=10 tid=0x00000000407ef800 nid=0x76cd waiting on condition [0x00007f6d0eae2000..0x00007f6d0eae2a70]

--

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:485)

at org.apache.hadoop.ipc.Client.call(Client.java:726)

- locked <0x00007f6d1cd28f80> (a org.apache.hadoop.ipc.Client$Call)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)

at $Proxy1.recoverBlock(Unknown Source)

at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2636)

at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:2832)

at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:529)

at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:186)

at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:530)

at org.apache.hadoop.hbase.util.FSUtils.recoverFileLease(FSUtils.java:619)

at org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1322)

at org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1210)

at org.apache.hadoop.hbase.master.HMaster.splitLogAfterStartup(HMaster.java:648)

at org.apache.hadoop.hbase.master.HMaster.joinCluster(HMaster.java:572)

at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:503)