测试场景描述
6台X86主机,Hadoop部署:3台作为管理节点,3台作为数据节点,HMaster和Hadoop管理节点复用,Region和Hadoop数据节点复用;主机位千兆网络
客户端同时并发8个进程,每个进程put 20000000条数据,每条数据1K大小;
测试过程
优化前
3806
3815
3813
3815
3806
3810
3814
3809
平均耗时:time:3811日志分析及优化
wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 3 regions(s): 336ddc5f32aa8e52064eeca793e47cba, 8d286647c97d7904bd0e6e43a463fbce, 056808c2f903c63a12c7d521747d1ad2
</property>
<name>hbase.regionserver.maxlogs</name>
<value>64</value>
</property>
2017-03-15 09:39:41,506 INFO [regionserver/hkweb37/137.0.0.47:14720-shortCompactions-1489485665547]regionserver.HStore: Completed compaction of 5 (all) file(s) in c1 ofh_test,23,1489483853929.630e23ca40ebb01f2bb89bca9674b0dd. intodd438aff4a064e5f8f303a531b42662d(size=1.2 G), total size for store is 1.2 G.This selection was in queue for 0sec, and took 35sec to execute.
2017-03-15 09:39:41,506 INFO [regionserver/hkweb37/137.0.0.47:14720-shortCompactions-1489485665547]regionserver.CompactSplitThread: Completed compaction: Request =regionName=h_test,23,1489483853929.630e23ca40ebb01f2bb89bca9674b0dd.,storeName=c1, fileCount=5, fileSize=1.2 G, priority=7, time=1423787495784043;duration=35sec
2017-03-1509:39:43,237 INFO [regionserver/hkweb37/137.0.0.47:14720.logRoller] wal.FSHLog: Rolled WAL/hbase/WALs/hkweb37,14720,1489485656123/hkweb37%2C14720%2C1489485656123.default.1489541939284with entries=17194, filesize=243.36 MB; new WAL/hbase/WALs/hkweb37,14720,1489485656123/hkweb37%2C14720%2C1489485656123.default.1489541983188<property>
<name>hbase.hstore.compaction.min</name>
<value>10</value>
</property>
<property>
<name>hbase.hstore.compaction.max</name>
<value>20</value>
</property>
<property>
<name>hbase.regionserver.thread.compaction.large</name>
<value>4</value>
</property>
<property>
<name>hbase.regionserver.thread.compaction.small</name>
<value>4</value>
</property>
优化后
3097
3107
3104
3106
3097
3107
3106
3110
平均耗时:time:3104.25