YCSB-HBase介绍与使用

原创 2016年08月30日 14:46:44

一.YCSB介绍

英文全称:Yahoo! Cloud Serving Benchmark (YCSB) Yahoo公司的一个用来对云服务进行基础测试的工具。目标是促进新一代云数据服务系统的性能比较。为四个广泛使用的系统:Cassandra,HBasePNUTS和一个简单的片式MySQL执行,订了套核心基准测试和结果报告

YCSB的特点:可扩展的,除了很容易对新系统进行基准测试,支持新定义的简单工作量。本文主要介绍使用ycsb来测试hbase

YCSB测试基本流程如下:

1.客户端服务器部署YCSB程序,向集群中load数据

2.按照预先定义的场景修改负载文件workload

3.使用run方法执行测试,向集群写入读取数据

4.获得吞吐量和平均延迟两个指标

YCSB的6个操作文件:

YCSB的workloads文件夹下面有6个文件,a/b/c/d/e/f,每个文件里面的配置参数不同,执行的时候得到的结果自然不同。

[root@wxb-1 workloads]# cat workloada 
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   


# Yahoo! Cloud System Benchmark
# Workload A: Update heavy workload
#   Application example: Session store recording recent actions
#                        
#   Read/update ratio: 50/50
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0

requestdistribution=zipfian

 
[root@wxb-1 workloads]# cat workloadb
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload B: Read mostly workload
#   Application example: photo tagging; add a tag is an update, but most operations are to read tags
#                        
#   Read/update ratio: 95/5
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.95
updateproportion=0.05
scanproportion=0
insertproportion=0

requestdistribution=zipfian

[root@wxb-1 workloads]# cat workloadc
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload C: Read only
#   Application example: user profile cache, where profiles are constructed elsewhere (e.g., Hadoop)
#                        
#   Read/update ratio: 100/0
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=1
updateproportion=0
scanproportion=0
insertproportion=0

requestdistribution=zipfian

[root@wxb-1 workloads]# cat workloadd
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload D: Read latest workload
#   Application example: user status updates; people want to read the latest
#                        
#   Read/update/insert ratio: 95/0/5
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: latest

# The insert order for this is hashed, not ordered. The "latest" items may be 
# scattered around the keyspace if they are keyed by userid.timestamp. A workload
# which orders items purely by time, and demands the latest, is very different than 
# workload here (which we believe is more typical of how people build systems.)

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.95
updateproportion=0
scanproportion=0
insertproportion=0.05

requestdistribution=latest

[root@wxb-1 workloads]# cat workloade
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload E: Short ranges
#   Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id)
#                        
#   Scan/insert ratio: 95/5
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

# The insert order is hashed, not ordered. Although the scans are ordered, it does not necessarily
# follow that the data is inserted in order. For example, posts for thread 342 may not be inserted contiguously, but
# instead interspersed with posts from lots of other threads. The way the YCSB client works is that it will pick a start
# key, and then request a number of records; this works fine even for hashed insertion.

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0
updateproportion=0
scanproportion=0.95
insertproportion=0.05

requestdistribution=zipfian

maxscanlength=100

scanlengthdistribution=uniform

[root@wxb-1 workloads]# cat workloadf 
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload F: Read-modify-write workload
#   Application example: user database, where user records are read and modified by the user or to record user activity.
#                        
#   Read/read-modify-write ratio: 50/50
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5
updateproportion=0
scanproportion=0
insertproportion=0
readmodifywriteproportion=0.5

requestdistribution=zipfian

上边就是workloads下面的6个文件,里面的配置不多,都是来描述读写修改的比例,所以可以看出,YCSB可以进行自己想要的任何比例的读写修改等操作。
workload
a-update heavy read50%
                             update50%    zipfian
b-read heavy read95%
                          update5%       zipfian
c-read heavy read100%       zipfian

d-read latest read95%      
                        insert5%      latest
e-short ranges scan95%
                           insert5%     zipfian/uniform

YCSB支持支持各种不同的数据分布方式


1. Uniform:等概论随机选择记录

2. Zipfian:随机选择记录,存在热记录

3. Latest:近期写入的记录为热记录

二.安装YCSB

编译YCSB

https://codeload.github.com/brianfrankcooper/YCSB/zip/master

Unzip YCSB-master.zip

[root@wxb-1 ycsb]# cd YCSB-master

[root@wxb-1 hbase10]# cat pom.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2012 - 2016 YCSB contributors. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you
may not use this file except in compliance with the License. You
may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License. See accompanying
LICENSE file.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>com.yahoo.ycsb</groupId>
    <artifactId>binding-parent</artifactId>
    <version>0.11.0-SNAPSHOT</version>
    <relativePath>../binding-parent/</relativePath>
  </parent>

  <artifactId>hbase10-binding</artifactId>
  <name>HBase 1.0 DB Binding</name>

  <dependencies>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>
      <version>1.2.2</version>
    </dependency>
    <dependency>
      <groupId>com.yahoo.ycsb</groupId>
      <artifactId>core</artifactId>
      <version>${project.version}</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>2.5.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.5.1</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-testing-util</artifactId>
      <version>1.2.2</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

下载的ycsb的pom文件中少了部分依赖,对照上边的配置文件进行修改。

编译

mvn -pl com.yahoo.ycsb:hbase10-binding -am clean package

这里只进行对hbase部分的编译

将编译后的tar包解压

tar -zxf ycsb-hbase10-binding-0.11.0-SNAPSHOT.tar.gz

三.YCSB测试hbase

建表

Create table usertable,family 

加载数据

ycsb load hbase10 -P ../workloads/workloada -cp /hbase/conf -p table=usertable -p columnfamily=family -p recordcount=100000 -threads 10

执行事物主要有read 和 update操作

bin/ycsb run hbase10 -P workloads/workloada -cp /hbase/conf  -p measurementtype=timeseries -p columnfamily=family -p timeseries.granularity=2000  threads 10

在执行测试操作的时候,-cp来指定本机Hbase的配置文件,-p来指定一些参数,-threads来指定线程数。

版权声明:本文为博主原创文章,未经博主允许不得转载。

YCSB配置和测试HBase

本文主要介绍如何使用YCSB来测试HBase,以及如何解决No FileSystem for scheme: hdfs错误和Error accessing HBase table错误。 YC...
  • hustsselbj
  • hustsselbj
  • 2015年06月17日 23:56
  • 3419

YCSB基础知识及HBase性能测试

YCSB是雅虎开源的NoSQL测试工具,本篇以对HBase的测试为例介绍其运行流程。一、准备工作下载安装: YCSB提供编译后的安装文件,只要下载安装包解压就好。 踩过的坑: (1) 目前最新的版...
  • zhangchen2449
  • zhangchen2449
  • 2016年11月11日 10:58
  • 1429

利用ycsb测试hbase性能

java 、maven、ycsb 的安装及配置见这篇博客: http://blog.csdn.net/hs794502825/article/details/17309845 本篇博客主要介绍...
  • hs794502825
  • hs794502825
  • 2013年12月16日 13:52
  • 8361

使用YCSB对HBase进行压测

(一)YCSB介绍: YCSB,全称为“Yahoo!Cloud Serving Benchmark”。是雅虎开发的用来对云服务进行基础测试的工具,其内部涵盖了常见的NoSQL数据库产品,如Cassan...
  • bryce123phy
  • bryce123phy
  • 2016年04月26日 21:22
  • 3282

YCSB--HBase性能测试工具的安装和使用

YCSB 一、背景概念 英文全称:Yahoo!CloudServing Benchmark(YCSB)。是Yahoo公司的一个用来对云服务进行基础测试的工具。目标是促进新一代云数据服务系统的性能比...
  • zx8167107
  • zx8167107
  • 2017年12月08日 16:50
  • 179

YCSB测试HBase的使用方法

n_splits = 20 create 'usertable','family',{SPLITS => (1..n_splits).map {|i| "user#{1000+i*(9999-100...
  • ArivnXu
  • ArivnXu
  • 2017年07月14日 15:08
  • 533

YCSB测试hbase

下载YCSB软件 wget https://github.com/downloads/brianfrankcooper/YCSB/ycsb-0.1.4.tar.gz   解压  tar xf Big...
  • haoxiaoyan
  • haoxiaoyan
  • 2017年02月21日 16:23
  • 859

使用YCSB对HBASE进行测试

YCSB介绍 YCSB(Yahoo! Cloud Serving Benchmark)是雅虎开源的一款通用的性能测试工具。 通过这个工具我们可以对各类NoSQL产品进行相关...
  • libing13810124573
  • libing13810124573
  • 2016年02月27日 13:43
  • 1149

HBase0.98.8 YCSB测试环境搭建

YCSB的全称是Yahoo! Cloud Serving Benchmark,这是一个用于测试cloud serving/NoSQL/Key-Value Store的benchmark。本文编译了HB...
  • u011291159
  • u011291159
  • 2014年12月16日 12:13
  • 2046

YCSB性能测试工具使用

在网上查In-Memory NoSQL性能测试的资料时,偶然间发现了这个性能测试工具YCSB,全称为“Yahoo! Cloud Serving Benchmark”。它内置了对常见NoSQL数据库和数...
  • dc_726
  • dc_726
  • 2015年02月28日 22:46
  • 18310
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:YCSB-HBase介绍与使用
举报原因:
原因补充:

(最多只允许输入30个字)