一.YCSB介绍
英文全称：Yahoo! Cloud Serving Benchmark (YCSB) 。是Yahoo公司的一个用来对云服务进行基础测试的工具。目标是促进新一代云数据服务系统的性能比较。为四个广泛使用的系统：Cassandra,、HBase、PNUTS和一个简单的片式MySQL执行，订了套核心基准测试和结果报告。
YCSB的特点：可扩展的，除了很容易对新系统进行基准测试，支持新定义的简单工作量。本文主要介绍使用ycsb来测试hbase。
YCSB测试基本流程如下：
1.客户端服务器部署YCSB程序，向集群中load数据
2.按照预先定义的场景修改负载文件workload
3.使用run方法执行测试，向集群写入读取数据
4.获得吞吐量和平均延迟两个指标
YCSB的6个操作文件：
YCSB的workloads文件夹下面有6个文件，a/b/c/d/e/f，每个文件里面的配置参数不同，执行的时候得到的结果自然不同。
[root@wxb-1 workloads]# cat workloada
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you
# may not use this file except in compliance with the License. You
# may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License. See accompanying
# LICENSE file.
# Yahoo! Cloud System Benchmark
# Workload A: Update heavy workload
# Application example: Session store recording recent actions
#
# Read/update ratio: 50/50
# Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
# Request distribution: zipfian
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0
requestdistribution=zipfian[root@wxb-1 workloads]# cat workloadb
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you
# may not use this file except in compliance with the License. You
# may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License. See accompanying
# LICENSE file.
# Yahoo! Cloud System Benchmark
# Workload B: Read mostly workload
# Application example: photo tagging; add a tag is an update, but most operations are to read tags
#
# Read/update ratio: 95/5
# Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
# Request distribution: zipfian
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.95
updateproportion=0.05
scanproportion=0
insertproportion=0
requestdistribution=zipfian[root@wxb-1 workloads]# cat workloadc
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you
# may not use this file except in compliance with the License. You
# may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License. See accompanying
# LICENSE file.
# Yahoo! Cloud System Benchmark
# Workload C: Read only
# Application example: user profile cache, where profiles are constructed elsewhere (e.g., Hadoop)
#
# Read/update ratio: 100/0
# Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
# Request distribution: zipfian
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=1
updateproportion=0
scanproportion=0
insertproportion=0
requestdistribution=zipfian[root@wxb-1 workloads]# cat workloadd
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you
# may not use this file except in compliance with the License. You
# may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License. See accompanying
# LICENSE file.
# Yahoo! Cloud System Benchmark
# Workload D: Read latest workload
# Application example: user status updates; people want to read the latest
#
# Read/update/insert ratio: 95/0/5
# Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
# Request distribution: latest
# The insert order for this is hashed, not ordered. The "latest" items may be
# scattered around the keyspace if they are keyed by userid.timestamp. A workload
# which orders items purely by time, and demands the latest, is very different than
# workload here (which we believe is more typical of how people build systems.)
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.95
updateproportion=0
scanproportion=0
insertproportion=0.05
requestdistribution=latest[root@wxb-1 workloads]# cat workloade
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you
# may not use this file except in compliance with the License. You
# may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License. See accompanying
# LICENSE file.
# Yahoo! Cloud System Benchmark
# Workload E: Short ranges
# Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id)
#
# Scan/insert ratio: 95/5
# Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
# Request distribution: zipfian
# The insert order is hashed, not ordered. Although the scans are ordered, it does not necessarily
# follow that the data is inserted in order. For example, posts for thread 342 may not be inserted contiguously, but
# instead interspersed with posts from lots of other threads. The way the YCSB client works is that it will pick a start
# key, and then request a number of records; this works fine even for hashed insertion.
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0
updateproportion=0
scanproportion=0.95
insertproportion=0.05
requestdistribution=zipfian
maxscanlength=100
scanlengthdistribution=uniform[root@wxb-1 workloads]# cat workloadf
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you
# may not use this file except in compliance with the License. You
# may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License. See accompanying
# LICENSE file.
# Yahoo! Cloud System Benchmark
# Workload F: Read-modify-write workload
# Application example: user database, where user records are read and modified by the user or to record user activity.
#
# Read/read-modify-write ratio: 50/50
# Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
# Request distribution: zipfian
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.5
updateproportion=0
scanproportion=0
insertproportion=0
readmodifywriteproportion=0.5
requestdistribution=zipfian
workload
a-update heavy read50%
update50% zipfian
b-read heavy read95%
update5% zipfian
c-read heavy read100% zipfian
d-read latest read95%
insert5% latest
e-short ranges scan95%
insert5% zipfian/uniform
a-update heavy read50%
update50% zipfian
b-read heavy read95%
update5% zipfian
c-read heavy read100% zipfian
d-read latest read95%
insert5% latest
e-short ranges scan95%
insert5% zipfian/uniform
YCSB支持支持各种不同的数据分布方式
1. Uniform：等概论随机选择记录
2. Zipfian：随机选择记录，存在热记录
3. Latest：近期写入的记录为热记录
1. Uniform：等概论随机选择记录
2. Zipfian：随机选择记录，存在热记录
3. Latest：近期写入的记录为热记录
二．安装YCSB
编译YCSB
：https://codeload.github.com/brianfrankcooper/YCSB/zip/master
Unzip YCSB-master.zip
[root@wxb-1 ycsb]# cd YCSB-master
[root@wxb-1 hbase10]# cat pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2012 - 2016 YCSB contributors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you
may not use this file except in compliance with the License. You
may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License. See accompanying
LICENSE file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.yahoo.ycsb</groupId>
<artifactId>binding-parent</artifactId>
<version>0.11.0-SNAPSHOT</version>
<relativePath>../binding-parent/</relativePath>
</parent>
<artifactId>hbase10-binding</artifactId>
<name>HBase 1.0 DB Binding</name>
<dependencies>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.2.2</version>
</dependency>
<dependency>
<groupId>com.yahoo.ycsb</groupId>
<artifactId>core</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.5.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.5.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-testing-util</artifactId>
<version>1.2.2</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>下载的ycsb的pom文件中少了部分依赖，对照上边的配置文件进行修改。
编译
mvn -pl com.yahoo.ycsb:hbase10-binding -am clean package
这里只进行对hbase部分的编译将编译后的tar包解压
tar -zxf ycsb-hbase10-binding-0.11.0-SNAPSHOT.tar.gz
三．YCSB测试hbase
建表
Create table ‘usertable’,’family’
加载数据
ycsb load hbase10 -P ../workloads/workloada -cp /hbase/conf -p table=usertable -p columnfamily=family -p recordcount=100000 -threads 10
执行事物：主要有read 和 update操作
bin/ycsb run hbase10 -P workloads/workloada -cp /hbase/conf -p measurementtype=timeseries -p columnfamily=family -p timeseries.granularity=2000 threads 10
在执行测试操作的时候，-cp来指定本机Hbase的配置文件，-p来指定一些参数，-threads来指定线程数。
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
