Hbase性能测试及优化过程记

测试场景描述

6台X86主机，Hadoop部署：3台作为管理节点，3台作为数据节点，HMaster和Hadoop管理节点复用，Region和Hadoop数据节点复用；主机位千兆网络

客户端同时并发8个进程，每个进程put 20000000条数据，每条数据1K大小；

测试过程

优化前

8个进程处理耗时如下：

3806

3815

3813

3815

3806

3810

3814

3809

平均耗时：time:3811

客户端网络IO

RegionServer网络IO

日志分析及优化

分析RegionServer日志发现如下信息：

wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 3 regions(s): 336ddc5f32aa8e52064eeca793e47cba, 8d286647c97d7904bd0e6e43a463fbce, 056808c2f903c63a12c7d521747d1ad2

</property>
<name>hbase.regionserver.maxlogs</name>
<value>64</value>
</property>

2017-03-15 09:39:41,506 INFO [regionserver/hkweb37/137.0.0.47:14720-shortCompactions-1489485665547]regionserver.HStore: Completed compaction of 5 (all) file(s) in c1 ofh_test,23,1489483853929.630e23ca40ebb01f2bb89bca9674b0dd. intodd438aff4a064e5f8f303a531b42662d(size=1.2 G), total size for store is 1.2 G.This selection was in queue for 0sec, and took 35sec to execute.

2017-03-15 09:39:41,506 INFO [regionserver/hkweb37/137.0.0.47:14720-shortCompactions-1489485665547]regionserver.CompactSplitThread: Completed compaction: Request =regionName=h_test,23,1489483853929.630e23ca40ebb01f2bb89bca9674b0dd.,storeName=c1, fileCount=5, fileSize=1.2 G, priority=7, time=1423787495784043;duration=35sec

2017-03-1509:39:43,237 INFO [regionserver/hkweb37/137.0.0.47:14720.logRoller] wal.FSHLog: Rolled WAL/hbase/WALs/hkweb37,14720,1489485656123/hkweb37%2C14720%2C1489485656123.default.1489541939284with entries=17194, filesize=243.36 MB; new WAL/hbase/WALs/hkweb37,14720,1489485656123/hkweb37%2C14720%2C1489485656123.default.1489541983188

<property>
<name>hbase.hstore.compaction.min</name>
<value>10</value>
</property>
<property>
<name>hbase.hstore.compaction.max</name>
<value>20</value>
</property>
<property>
<name>hbase.regionserver.thread.compaction.large</name>
<value>4</value>
</property>
<property>
<name>hbase.regionserver.thread.compaction.small</name>
<value>4</value>
</property>

优化后

8个进程处理耗时如下：

3097

3107

3104

3106

3097

3107

3106

3110

平均耗时：time:3104.25

客户端网络IO

RegionServer网络IO

测试结论

优化后网络IO占用更均衡，当Memstore Flush时客户端有明显的性能下降，主要是由于Hdfs做副本复制导致数据节点网络IO占满。