1. 下载YCSB并解压缩
① 下载对应版本的YCSB
不同版本的HBase对应不同的YCSB,本机安装的是hbase-1.4.6版本,所以下载的是ycsb-hbase14-binding-0.15.0.tar.gz
。
YCSB的下载地址为:YCSB下载
② 解压缩YCSB
$ sudo tar -zxvf ycsb-hbase14-binding-0.15.0.tar.gz
$ sudo chmod -R 0777 ycsb-hbase14-binding-0.15.0 #设置文件权限
$ sudo cp -rf ycsb-hbase14-binding-0.15.0 /home/cephlee
将ycsb-hbase14-binding-0.15.0
改名为ycsb-hbase14。
2. 配置YCSB
① 使用gedit
命令设置YCSB系统环境变量
$ sudo gedit ~/.bashrc
添加以下内容:
export YCSB_HOME=/home/cephlee/ycsb-hbase14
export PATH=$PATH:$YCSB_HOME/bin
使用source
命令使系统环境变量生效:
$ source ~/.bashrc
输入以下命令,查看系统环境变量是否生效:
$ ycsb
显示信息如下(部分信息截图):
② 添加hbase-site.xml
在ycsb-hbase14中创建conf文件夹,将hbase/conf
中的hbase-site.xml
复制到ycsb-hbase14/conf
中。
3. 学习YCSB的压力测试场景
在ycsb-hbase14/workloads
目录中,包含一些压力测试的场景文件。YCSB自带了6种压力测试场景,文件和相应场景的对应关系如下:
名称 | 特点 | Request distribution | scan length distribution |
---|---|---|---|
workloada | 读写均衡型,50% Read,50% Write | zipfian,随机选择记录,存在热记录 | 无 |
workloadb | 读多写少型,95% Read,5% Write | zipfian,随机选择记录,存在热记录 | 无 |
workloadc | 只读型,100% Read | zipfian,随机选择记录,存在热记录 | 无 |
workloadd | 95% Read,5% Insert | latest,近期写入的记录为热记录 | 无 |
workloade | 95% Scan, 5% Insert | zipfian,随机选择记录,存在热记录 | uniform,等概论随机选择记录 |
workloadf | 50% Read,50% Read-modify-write | zipfian,随机选择记录,存在热记录 | 无 |
4. 使用YCSB进行HBase压力测试
① 在HBase中创建表
在hbase shell
模式下,输入以下命令创建创建usertable表,列簇family 。这个是YCSB测试的前置条件,测试脚本中不会自动创建:
$ create 'usertable','family'
如下信息表示建表成功:
hbase(main):006:0> create 'usertable','family'
0 row(s) in 4.3920 seconds
=> Hbase::Table - usertable
本人在创建usertable表时,提示:
hbase(main):001:0> create 'usertable','family'
ERROR: Table already exists: usertable!
但是通过list
命令查看table,没有发现usertable
,解决办法:
参考链接:Hbase ERROR: Table already exists ,问题得到解决。使用到的命令如下:
$ hbase zkcli #进入zookeeper client模式
$ ls /hbase/table #查看zombie table
$ rmr /hbase/table/usertable #删除zombie table,其中usertable表示要删除的TABLE_NAME
$ stop-hbase.sh # 重启Hbase
$ start-hbase.sh
② 加载数据
切换到$YCSB_HOME
目录,执行以下命令,加载10000条记录:
$ ycsb load hbase14 -P workloads/workloada -cp /home/ycsb-hbase14/conf -p table=usertable -p columnfamily=family -p recordcount=10000
参数解释如下:
load:
表示这是一次加载数据操作
hbase14::
指明了所用的数据库类型,本人使用的是hbase1.4.6
,所以为hbase14
;
-P:
指明了所用的配置文件的路径
-cp:
指定了hbase的配置文件所在的路径,由于最开始已经将hbase/conf
中的hbase-site.xml
复制到ycsb-hbase14/conf
中,所以这里写
-p:
可以显示修改YCSB内置的默认配置,例如这里配置recordcount=10000来覆盖之前说过的workloada中默认的recordcount=1000;
数据加载结果:
$ ycsb load hbase14 -P workloads/workloada -cp /home/ycsb-hbase14/conf -p table=usertable -p columnfamily=family -p recordcount=10000
/usr/local/jdk18/bin/java -cp /home/ycsb-hbase14/conf:/home/cephlee/ycsb-hbase14/hbase14-binding/conf:/home/cephlee/ycsb-hbase14/conf:/home/cephlee/ycsb-hbase14/lib/htrace-core-3.1.0-incubating.jar:/home/cephlee/ycsb-hbase14/lib/log4j-1.2.17.jar:/home/cephlee/ycsb-hbase14/lib/hbase14-binding-0.15.0.jar:/home/cephlee/ycsb-hbase14/lib/slf4j-log4j12-1.7.7.jar:/home/cephlee/ycsb-hbase14/lib/core-0.15.0.jar:/home/cephlee/ycsb-hbase14/lib/jackson-mapper-asl-1.9.4.jar:/home/cephlee/ycsb-hbase14/lib/slf4j-api-1.7.25.jar:/home/cephlee/ycsb-hbase14/lib/commons-logging-1.2.jar:/home/cephlee/ycsb-hbase14/lib/HdrHistogram-2.1.4.jar:/home/cephlee/ycsb-hbase14/lib/hbase-shaded-client-1.4.2.jar:/home/cephlee/ycsb-hbase14/lib/htrace-core4-4.1.0-incubating.jar:/home/cephlee/ycsb-hbase14/lib/findbugs-annotations-1.3.9-1.jar:/home/cephlee/ycsb-hbase14/lib/hbase10-binding-0.15.0.jar:/home/cephlee/ycsb-hbase14/lib/jackson-core-asl-1.9.4.jar com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.hbase14.HBaseClient14 -P workloads/workloada -p table=usertable -p columnfamily=family -p recordcount=10000 -load
Command line: -db com.yahoo.ycsb.db.hbase14.HBaseClient14 -P workloads/workloada -p table=usertable -p columnfamily=family -p recordcount=10000 -load
YCSB Client 0.15.0
Loading workload...
(might take a few minutes for large data sets)
log4j:WARN No appenders could be found for logger (org.apache.htrace.core.Tracer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Starting test.
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
[OVERALL], RunTime(ms), 91443
[OVERALL], Throughput(ops/sec), 109.35774198134357
[TOTAL_GCS_PS_Scavenge], Count, 25
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 284
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.31057598722701574
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 119
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.13013571295779885
[TOTAL_GCs], Count, 26
[TOTAL_GC_TIME], Time(ms), 403
[TOTAL_GC_TIME_%], Time(%), 0.4407117001848146
[CLEANUP], Operations, 2
[CLEANUP], AverageLatency(us), 61883.5
[CLEANUP], MinLatency(us), 23
[CLEANUP], MaxLatency(us), 123775
[CLEANUP], 95thPercentileLatency(us), 123775
[CLEANUP], 99thPercentileLatency(us), 123775
[INSERT], Operations, 10000
[INSERT], AverageLatency(us), 7909.6476
[INSERT], MinLatency(us), 4372
[INSERT], MaxLatency(us), 2238463
[INSERT], 95thPercentileLatency(us), 12111
[INSERT], 99thPercentileLatency(us), 17711
[INSERT], Return=OK, 10000
③ 执行压力测试
在$YCSB_HOME
目录下,执行以下命令,执行10000条记录的压力测试:
$ ycsb run hbase14 -P workloads/workloada -cp /home/ycsb-hbase14/conf -p table=usertable -p columnfamily=family -p operationcount=10000
参数解读:
-p operationcount=10000:
覆盖了workloada中operationcount=1000
表示本次测试操作记录数为10000
数据运行结果:
$ ycsb run hbase14 -P workloads/workloada -cp /home/ycsb-hbase14/conf -p table=usertable -p columnfamily=family -p operationcount=10000
/usr/local/jdk18/bin/java -cp /home/ycsb-hbase14/conf:/home/cephlee/ycsb-hbase14/hbase14-binding/conf:/home/cephlee/ycsb-hbase14/conf:/home/cephlee/ycsb-hbase14/lib/htrace-core-3.1.0-incubating.jar:/home/cephlee/ycsb-hbase14/lib/log4j-1.2.17.jar:/home/cephlee/ycsb-hbase14/lib/hbase14-binding-0.15.0.jar:/home/cephlee/ycsb-hbase14/lib/slf4j-log4j12-1.7.7.jar:/home/cephlee/ycsb-hbase14/lib/core-0.15.0.jar:/home/cephlee/ycsb-hbase14/lib/jackson-mapper-asl-1.9.4.jar:/home/cephlee/ycsb-hbase14/lib/slf4j-api-1.7.25.jar:/home/cephlee/ycsb-hbase14/lib/commons-logging-1.2.jar:/home/cephlee/ycsb-hbase14/lib/HdrHistogram-2.1.4.jar:/home/cephlee/ycsb-hbase14/lib/hbase-shaded-client-1.4.2.jar:/home/cephlee/ycsb-hbase14/lib/htrace-core4-4.1.0-incubating.jar:/home/cephlee/ycsb-hbase14/lib/findbugs-annotations-1.3.9-1.jar:/home/cephlee/ycsb-hbase14/lib/hbase10-binding-0.15.0.jar:/home/cephlee/ycsb-hbase14/lib/jackson-core-asl-1.9.4.jar com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.hbase14.HBaseClient14 -P workloads/workloada -p table=usertable -p columnfamily=family -t
Command line: -db com.yahoo.ycsb.db.hbase14.HBaseClient14 -P workloads/workloada -p table=usertable -p columnfamily=family -t
YCSB Client 0.15.0
Loading workload...
log4j:WARN No appenders could be found for logger (org.apache.htrace.core.Tracer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Starting test.
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
[OVERALL], RunTime(ms), 12446
[OVERALL], Throughput(ops/sec), 80.34709946970915
[TOTAL_GCS_PS_Scavenge], Count, 5
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 108
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8677486742728587
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 89
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.7150891852804114
[TOTAL_GCs], Count, 6
[TOTAL_GC_TIME], Time(ms), 197
[TOTAL_GC_TIME_%], Time(%), 1.58283785955327
[READ], Operations, 518
[READ], AverageLatency(us), 5644.959459459459
[READ], MinLatency(us), 3222
[READ], MaxLatency(us), 482815
[READ], 95thPercentileLatency(us), 7099
[READ], 99thPercentileLatency(us), 13935
[READ], Return=OK, 518
[CLEANUP], Operations, 2
[CLEANUP], AverageLatency(us), 56741.5
[CLEANUP], MinLatency(us), 43
[CLEANUP], MaxLatency(us), 113471
[CLEANUP], 95thPercentileLatency(us), 113471
[CLEANUP], 99thPercentileLatency(us), 113471
[UPDATE], Operations, 482
[UPDATE], AverageLatency(us), 9502.92531120332
[UPDATE], MinLatency(us), 5748
[UPDATE], MaxLatency(us), 269055
[UPDATE], 95thPercentileLatency(us), 15191
[UPDATE], 99thPercentileLatency(us), 34143
[UPDATE], Return=OK, 482