YCSB-HBase介绍与使用

原创 2016年08月30日 14:46:44

一.YCSB介绍

英文全称:Yahoo! Cloud Serving Benchmark (YCSB) Yahoo公司的一个用来对云服务进行基础测试的工具。目标是促进新一代云数据服务系统的性能比较。为四个广泛使用的系统:Cassandra,HBasePNUTS和一个简单的片式MySQL执行,订了套核心基准测试和结果报告

YCSB的特点:可扩展的,除了很容易对新系统进行基准测试,支持新定义的简单工作量。本文主要介绍使用ycsb来测试hbase

YCSB测试基本流程如下:

1.客户端服务器部署YCSB程序,向集群中load数据

2.按照预先定义的场景修改负载文件workload

3.使用run方法执行测试,向集群写入读取数据

4.获得吞吐量和平均延迟两个指标

YCSB的6个操作文件:

YCSB的workloads文件夹下面有6个文件,a/b/c/d/e/f,每个文件里面的配置参数不同,执行的时候得到的结果自然不同。

[root@wxb-1 workloads]# cat workloada 
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   


# Yahoo! Cloud System Benchmark
# Workload A: Update heavy workload
#   Application example: Session store recording recent actions
#                        
#   Read/update ratio: 50/50
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0

requestdistribution=zipfian

 
[root@wxb-1 workloads]# cat workloadb
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload B: Read mostly workload
#   Application example: photo tagging; add a tag is an update, but most operations are to read tags
#                        
#   Read/update ratio: 95/5
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.95
updateproportion=0.05
scanproportion=0
insertproportion=0

requestdistribution=zipfian

[root@wxb-1 workloads]# cat workloadc
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload C: Read only
#   Application example: user profile cache, where profiles are constructed elsewhere (e.g., Hadoop)
#                        
#   Read/update ratio: 100/0
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=1
updateproportion=0
scanproportion=0
insertproportion=0

requestdistribution=zipfian

[root@wxb-1 workloads]# cat workloadd
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload D: Read latest workload
#   Application example: user status updates; people want to read the latest
#                        
#   Read/update/insert ratio: 95/0/5
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: latest

# The insert order for this is hashed, not ordered. The "latest" items may be 
# scattered around the keyspace if they are keyed by userid.timestamp. A workload
# which orders items purely by time, and demands the latest, is very different than 
# workload here (which we believe is more typical of how people build systems.)

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.95
updateproportion=0
scanproportion=0
insertproportion=0.05

requestdistribution=latest

[root@wxb-1 workloads]# cat workloade
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload E: Short ranges
#   Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id)
#                        
#   Scan/insert ratio: 95/5
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

# The insert order is hashed, not ordered. Although the scans are ordered, it does not necessarily
# follow that the data is inserted in order. For example, posts for thread 342 may not be inserted contiguously, but
# instead interspersed with posts from lots of other threads. The way the YCSB client works is that it will pick a start
# key, and then request a number of records; this works fine even for hashed insertion.

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0
updateproportion=0
scanproportion=0.95
insertproportion=0.05

requestdistribution=zipfian

maxscanlength=100

scanlengthdistribution=uniform

[root@wxb-1 workloads]# cat workloadf 
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.                                                                                                                             
#                                                                                                                                                                                 
# Licensed under the Apache License, Version 2.0 (the "License"); you                                                                                                             
# may not use this file except in compliance with the License. You                                                                                                                
# may obtain a copy of the License at                                                                                                                                             
#                                                                                                                                                                                 
# http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                      
#                                                                                                                                                                                 
# Unless required by applicable law or agreed to in writing, software                                                                                                             
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                                               
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or                                                                                                                 
# implied. See the License for the specific language governing                                                                                                                    
# permissions and limitations under the License. See accompanying                                                                                                                 
# LICENSE file.                                                                                                                                                                   

# Yahoo! Cloud System Benchmark
# Workload F: Read-modify-write workload
#   Application example: user database, where user records are read and modified by the user or to record user activity.
#                        
#   Read/read-modify-write ratio: 50/50
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5
updateproportion=0
scanproportion=0
insertproportion=0
readmodifywriteproportion=0.5

requestdistribution=zipfian

上边就是workloads下面的6个文件,里面的配置不多,都是来描述读写修改的比例,所以可以看出,YCSB可以进行自己想要的任何比例的读写修改等操作。
workload
a-update heavy read50%
                             update50%    zipfian
b-read heavy read95%
                          update5%       zipfian
c-read heavy read100%       zipfian

d-read latest read95%      
                        insert5%      latest
e-short ranges scan95%
                           insert5%     zipfian/uniform

YCSB支持支持各种不同的数据分布方式


1. Uniform:等概论随机选择记录

2. Zipfian:随机选择记录,存在热记录

3. Latest:近期写入的记录为热记录

二.安装YCSB

编译YCSB

https://codeload.github.com/brianfrankcooper/YCSB/zip/master

Unzip YCSB-master.zip

[root@wxb-1 ycsb]# cd YCSB-master

[root@wxb-1 hbase10]# cat pom.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2012 - 2016 YCSB contributors. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you
may not use this file except in compliance with the License. You
may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License. See accompanying
LICENSE file.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>com.yahoo.ycsb</groupId>
    <artifactId>binding-parent</artifactId>
    <version>0.11.0-SNAPSHOT</version>
    <relativePath>../binding-parent/</relativePath>
  </parent>

  <artifactId>hbase10-binding</artifactId>
  <name>HBase 1.0 DB Binding</name>

  <dependencies>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>
      <version>1.2.2</version>
    </dependency>
    <dependency>
      <groupId>com.yahoo.ycsb</groupId>
      <artifactId>core</artifactId>
      <version>${project.version}</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>2.5.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.5.1</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-testing-util</artifactId>
      <version>1.2.2</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

下载的ycsb的pom文件中少了部分依赖,对照上边的配置文件进行修改。

编译

mvn -pl com.yahoo.ycsb:hbase10-binding -am clean package

这里只进行对hbase部分的编译

将编译后的tar包解压

tar -zxf ycsb-hbase10-binding-0.11.0-SNAPSHOT.tar.gz

三.YCSB测试hbase

建表

Create table usertable,family 

加载数据

ycsb load hbase10 -P ../workloads/workloada -cp /hbase/conf -p table=usertable -p columnfamily=family -p recordcount=100000 -threads 10

执行事物主要有read 和 update操作

bin/ycsb run hbase10 -P workloads/workloada -cp /hbase/conf  -p measurementtype=timeseries -p columnfamily=family -p timeseries.granularity=2000  threads 10

在执行测试操作的时候,-cp来指定本机Hbase的配置文件,-p来指定一些参数,-threads来指定线程数。

版权声明:本文为博主原创文章,未经博主允许不得转载。

相关文章推荐

YCSB配置和测试HBase

本文主要介绍如何使用YCSB来测试HBase,以及如何解决No FileSystem for scheme: hdfs错误和Error accessing HBase table错误。 YC...

使用YCSB对HBase进行压测

(一)YCSB介绍: YCSB,全称为“Yahoo!Cloud Serving Benchmark”。是雅虎开发的用来对云服务进行基础测试的工具,其内部涵盖了常见的NoSQL数据库产品,如Cassan...

YCSB测试HBase的使用方法

n_splits = 20 create 'usertable','family',{SPLITS => (1..n_splits).map {|i| "user#{1000+i*(9999-100...
  • ArivnXu
  • ArivnXu
  • 2017年07月14日 15:08
  • 393

YCSB对HBase性能测试

我认真的在想,关在牢里面的人,不一定比放在外面的人坏。 -- 三毛     这篇文章放草稿箱快两个多月了,今天突然被告知,要换方向,看了近三个月的HBase就这样别了,很突然。   刚开始装上HB...

YCSB测试hbase

下载YCSB软件 wget https://github.com/downloads/brianfrankcooper/YCSB/ycsb-0.1.4.tar.gz   解压  tar xf Big...

利用ycsb测试hbase性能

java 、maven、ycsb 的安装及配置见这篇博客: http://blog.csdn.net/hs794502825/article/details/17309845 本篇博客主要介绍...

YCSB基础知识及HBase性能测试

YCSB是雅虎开源的NoSQL测试工具,本篇以对HBase的测试为例介绍其运行流程。一、准备工作下载安装: YCSB提供编译后的安装文件,只要下载安装包解压就好。 踩过的坑: (1) 目前最新的版...

使用YCSB对HBASE进行测试

YCSB介绍 YCSB(Yahoo! Cloud Serving Benchmark)是雅虎开源的一款通用的性能测试工具。 通过这个工具我们可以对各类NoSQL产品进行相关...

使用YCSB压测Cassandra 3.7

使用YCSB压测Cassandra 3.7 1、在https://github.com/brianfrankcooper/YCSB/获取源代码(Git或直接下载,我用Git下载到82%时候不动了,于是...

YCSB官方文档-核心属性

原文链接 1. 核心YCSB属性 所有工作量文件可以指定以下属性: workload:要使用的工作量类(例如com.yahoo.ycsb.workloads.CoreWorklo...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:YCSB-HBase介绍与使用
举报原因:
原因补充:

(最多只允许输入30个字)