HBase使用YCSB进行性能测试

最新推荐文章于 2024-01-25 01:55:23 发布

沧泽

最新推荐文章于 2024-01-25 01:55:23 发布

阅读量547

点赞数 1

文章标签： hbase docker 数据库

本文链接：https://blog.csdn.net/weixin_44171618/article/details/130021597

版权

HBase初步学习与性能测试_hbase性能测试_LLLSoul的博客-CSDN博客

一、连接到docker中的hbase

1.docker ps查看运行的docker

在这里插入图片描述

2. 连接到docker容器中

docker进入当前正在运行容器的两种方式（exec和attach的区别）_docker exec进入容器_星丶空LTL的博客-CSDN博客

2.1> 开启一个新的终端

docker exec -it b30eae75b255 /bin/bash

可以使用exit命令退出

3.执行hbase shell 进入hbase终端

1.设置pre-splitting策略
hbase(main):001:0> n_splits=30 # 设置预分区数为30个

2.list查看已有的表格
hbase(main):003:0> list
TABLE
TESTDATA_WS11_001_UNIT01_SS001
TESTDATA_WS11_001_UNIT01_SS001_PP
TESTDATA_Z3PMz7rp
TESTDATA_data_import_tes
TESTDATA_data_preprocess_res
TESTDATA_hbase_press_test
TESTDATA_hbase_press_test2
TESTDATA_hbase_press_test3

3.创建表
hbase(main):004:0> create 'hbase_test','family',{SPLITS => (1..n_splits).map{|i| "user#{1000+i*(9999-1000)/n_splits}"}}

4.加载workload：

bin/ycsb load hbase12 -P workloads/workloada -cp /opt/module/hbase-1.4.13/conf/ -p table=usertable -p columnfamily=family

bin/ycsb load hbase12 -P workloads/workloada -cp /hbase-2.1.3/conf/ -p table=hbase_test -p columnfamily=family
-P: 指明所用的配置文件的路径
-p: 显示修改YCSB内置的默认配置

报错1：

env: can't execute 'python': No such file or directory

解决：创建python环境

(1条消息) Docker Alpine容器中安装Python和pip3（附Dockerfile模板）_python:3-alpine 怎么下载第三方库_Amour_柒的博客-CSDN博客

docker exec -it b30eae75b255 apk add --update --no-cache curl jq py3-configobj py3-pip py3-setuptools python3 python3-dev

进入docker，执行命令，创建软连接

bash-4.4# ln -s /usr/bin/python3 /usr/bin/python

报错2：

bash-4.4# bin/ycsb load hbase12 -P workloads/workloada -cp /hbase-2.1.3/conf/ -p table=hbase_test -p columnfamily=family
  File "bin/ycsb", line 228
    except subprocess.CalledProcessError, err:

解决：

bin/ycsb.sh load hbase12 -P workloads/workloada -cp /hbase-2.1.3/conf/ -p table=hbase_test -p columnfamily=family

报错3：

unknown -cp

#####换一种方式加载####

sh bin/ycsb.sh load hbase12 -P /ycsb-hbase12-binding-0.17.0/workloads/workloada -threads 30 -p table=hbase_test -p columnfamily=family -p recordcount=100000 -s > logs/load.log

load 表示这是一次加载数据操作；
hbase12指明了所用的数据库类型；
-P指明了所用的配置文件的路径；
-p 可以显示修改YCSB内置的默认配置，例如这里配置recordcount=10000来覆盖之前说过的workloada中默认的recordcount=1000；
-s一次加载数据或执行测试的过程实践很长，YCSB客户端需要定时(默认10s)发送状态信息；> logs/load.log表示结果日志写入位置（若没有logs文件夹就自己新建一个，也可以不加此参数，则运行结果直接打印到屏幕上）

加载成功，查看load.log文件

在这里插入图片描述

5.执行workload

bin/ycsb.sh run hbase12 -P workloads/workloada -cp /hbase-2.1.3/conf/ -p table=hbase_test -p columnfamily=family

###上面命令不能使用###

执行压力测试

sh ycsb.sh run hbase12 -P /opt/YCSB/ycsb-hbase12-binding-0.14.0/workloads/workloada -threads 10 -p operationcount=1000000 -p table=usertable -p columnfamily=family -p measurementtype=timeseries -p timeseries.granularity=2000 -s > logs/transaction-workloadAA.log

sh bin/ycsb.sh run hbase12 -P /ycsb-hbase12-binding-0.17.0/workloads/workloada -threads 10 -p operationcount=1000000 -p table=hbase_test -p columnfamily=family -p measurementtype=timeseries -p timeseries.granularity=2000 -s > logs/transaction-workloadAA.log

结果：

bash-4.4# cat logs/transaction-workloadAA.log
/usr/bin/java  -classpath /ycsb-hbase12-binding-0.17.0/conf:/ycsb-hbase12-binding-0.17.0/lib/HdrHistogram-2.1.4.jar:/ycsb-hbase12-binding-0.17.0/lib/commons-logging-1.2.jar:/ycsb-hbase12-binding-0.17.0/lib/core-0.17.0.jar:/ycsb-hbase12-binding-0.17.0/lib/findbugs-annotations-1.3.9-1.jar:/ycsb-hbase12-binding-0.17.0/lib/hbase-shaded-client-1.2.5.jar:/ycsb-hbase12-binding-0.17.0/lib/hbase10-binding-0.17.0.jar:/ycsb-hbase12-binding-0.17.0/lib/hbase12-binding-0.17.0.jar:/ycsb-hbase12-binding-0.17.0/lib/htrace-core-3.1.0-incubating.jar:/ycsb-hbase12-binding-0.17.0/lib/htrace-core4-4.1.0-incubating.jar:/ycsb-hbase12-binding-0.17.0/lib/jackson-core-asl-1.9.4.jar:/ycsb-hbase12-binding-0.17.0/lib/jackson-mapper-asl-1.9.4.jar:/ycsb-hbase12-binding-0.17.0/lib/log4j-1.2.17.jar:/ycsb-hbase12-binding-0.17.0/lib/slf4j-api-1.7.25.jar:/ycsb-hbase12-binding-0.17.0/lib/slf4j-log4j12-1.6.1.jar site.ycsb.Client -t -db site.ycsb.db.hbase12.HBaseClient12 -P /ycsb-hbase12-binding-0.17.0/workloads/workloada -threads 10 -p operationcount=1000000 -p table=hbase_test -p columnfamily=family -p measurementtype=timeseries -p timeseries.granularity=2000 -s
2023-04-07 13:29:15:035 0 sec: 0 operations; est completion in 0 second
2023-04-07 13:29:24:993 10 sec: 160507 operations; 16050.7 current ops/sec; est completion in 53 seconds [READ AverageLatency(us)=457.22] [UPDATE AverageLatency(us)=685.58]
2023-04-07 13:29:34:992 20 sec: 361240 operations; 20073.3 current ops/sec; est completion in 36 seconds [READ AverageLatency(us)=441.55] [UPDATE AverageLatency(us)=549.2]
2023-04-07 13:29:44:992 30 sec: 559483 operations; 19824.3 current ops/sec; est completion in 24 seconds [READ AverageLatency(us)=450.31] [UPDATE AverageLatency(us)=553.07]
2023-04-07 13:29:54:992 40 sec: 753578 operations; 19409.5 current ops/sec; est completion in 14 seconds [READ AverageLatency(us)=463.03] [UPDATE AverageLatency(us)=561.81]
2023-04-07 13:30:04:992 50 sec: 947656 operations; 19407.8 current ops/sec; est completion in 3 second [READ AverageLatency(us)=467.62] [UPDATE AverageLatency(us)=557.41]
2023-04-07 13:30:07:933 52 sec: 1000000 operations; 17804.08 current ops/sec; [READ AverageLatency(us)=483.93] [CLEANUP AverageLatency(us)=2465.5] [UPDATE AverageLatency(us)=558.32]
[OVERALL], RunTime(ms), 52940
[OVERALL], Throughput(ops/sec)加载操作的吞吐量(ops/sec), 18889.308651303363
[TOTAL_GCS_PS_Scavenge], Count, 110
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 416
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.7857952398942198
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0
[TOTAL_GCs], Count, 110
[TOTAL_GC_TIME], Time(ms), 416
[TOTAL_GC_TIME_%], Time(%), 0.7857952398942198
[READ], Operations, 499323
[READ], AverageLatency(us), 457.2533850834029
[READ], MinLatency(us), 147
[READ], MaxLatency(us), 52710
[READ], Return=OK, 499323
[READ], 0, 701.6545917131087
[READ], 2000, 441.11767493287
[READ], 4000, 422.18283417479535
[READ], 6000, 424.84448356807513
[READ], 8000, 434.0281038429692
[READ], 10000, 425.3071624680892
[READ], 12000, 453.91895876760384
[READ], 14000, 449.01038841410366
[READ], 16000, 449.88070247299163
[READ], 18000, 430.4783983652817
[READ], 20000, 454.8975220068183
[READ], 22000, 456.8564662273476
[READ], 24000, 452.22350605721266
[READ], 26000, 439.91921514899445
[READ], 28000, 449.2450283727826
[READ], 30000, 459.69042863727145
[READ], 32000, 470.3479426626792
[READ], 34000, 463.2190549563431
[READ], 36000, 467.61667540639746
[READ], 38000, 460.9345548387097
[READ], 40000, 469.539915857269
[READ], 42000, 458.74156511229734
[READ], 44000, 475.11790416008427
[READ], 46000, 467.39115328769964
[READ], 48000, 476.9374197194677
[READ], 50000, 486.88857631419273
[READ], 52000, 309.02830188679246
[CLEANUP], Operations, 20
[CLEANUP], AverageLatency(us), 2465.5
[CLEANUP], MinLatency(us), 2
[CLEANUP], MaxLatency(us), 43692
[CLEANUP], 0, 2465.5
[UPDATE], Operations, 500677
[UPDATE], AverageLatency(us)#每次update操作的平均延时(微妙), 576.3625471112114 
[UPDATE], MinLatency(us), 175
[UPDATE], MaxLatency(us), 107102
[UPDATE], Return=OK, 500677
[UPDATE], 0, 1747.6644176136363
[UPDATE], 2000, 605.0278335269885
[UPDATE], 4000, 551.4616250611846
[UPDATE], 6000, 542.5884725664577
[UPDATE], 8000, 541.7539248978788
[UPDATE], 10000, 533.2333767926988
[UPDATE], 12000, 558.4006817950544
[UPDATE], 14000, 559.179505229284
[UPDATE], 16000, 560.1013053777532
[UPDATE], 18000, 538.7887103862494
[UPDATE], 20000, 551.7934158864391
[UPDATE], 22000, 558.2820344129555
[UPDATE], 24000, 563.6328004918537
[UPDATE], 26000, 535.9347740667977
[UPDATE], 28000, 559.1775070479259
[UPDATE], 30000, 546.3841530602024
[UPDATE], 32000, 564.4708991939705
[UPDATE], 34000, 552.6139565637557
[UPDATE], 36000, 578.2039934671513
[UPDATE], 38000, 558.4095077957811
[UPDATE], 40000, 558.5023652817771
[UPDATE], 42000, 550.3681604736627
[UPDATE], 44000, 571.1756841774143
[UPDATE], 46000, 562.8433859694538
[UPDATE], 48000, 545.4066069774622
[UPDATE], 50000, 567.6255558682803
[UPDATE], 52000, 429.4032258064516

此输出显示更新和读取操作的单独时序，每 2000 毫秒报告一次数据。某个时间点报告的数据仅为前 2000 毫秒的延迟的平均值。

各属性含义：

属性	含义
[OVERALL], RunTime(ms)	数据加载所用时间(毫秒)
[OVERALL], Throughput(ops/sec)	加载操作的平均吞吐量(ops/sec)（操作/秒）
[INSERT], Operations	执行insert操作的总数
[INSERT], AverageLatency(us)	每次insert操作的平均延时(微秒)
[INSERT], MinLatency(us)	每次insert操作的最小延时(微秒)
[INSERT], MaxLatency(us)	每次insert操作的最大延时(微秒)
[INSERT], 95thPercentileLatency(us)	95%的insert操作延时在多少微秒以内
[INSERT], 99thPercentileLatency(us)	99%的insert操作延时在多少微秒以内
[INSERT], Return=OK	操作成功返回数
[INSERT], Return=0,491	insert返回代码为0的操作数个数