YCSB负载工具测试leveldb

最新推荐文章于 2022-05-12 15:44:56 发布

机智帅气的葫芦娃

最新推荐文章于 2022-05-12 15:44:56 发布

阅读量2.5k

点赞数 1

分类专栏：数据库 ycsb 文章标签： mapkeeper leveldb ycsb

本文链接：https://blog.csdn.net/clever_wr/article/details/88924165

版权

ycsb 同时被 2 个专栏收录

10 篇文章 1 订阅

订阅专栏

数据库

4 篇文章 0 订阅

订阅专栏

网上对有关YCSB怎么连接leveldb的讲解简直是太少了。我弄了好久才弄好分享下自己的经验，也为之后自己不用再查一遍。

首先下载需要的一些包：（我的系统版本是centos 7）

1.首先安装thrift：

wget http://mirrors.tuna.tsinghua.edu.cn/apache/thrift/0.12.0/thrift-0.12.0.tar.gz
tar zxvf thrift-0.12.0.tar.gz
yum install -y automake libtool flex bison pkgconfig 
yum install -y gcc-c++ libevent-devel zlib-devel python-devel ruby-devel openssl-devel

2.安装boost：（仅限于boost_1_67_0版本，因为老版本安装方式好像和这个有差别，我试过make一直报错）

wget https://sourceforge.net/projects/boost/files/boost/1.67.0/boost_1_67_0.tar.bz2
tar jxvf boost_1_67_0.tar.bz2
yum install gcc gcc-c++ bzip2 bzip2-devel bzip2-libs python-devel zlib-devel -y
sudo ./bootstrap.sh --prefix=/usr/local/include/boost
sudo ./b2 install

进入boost_1_67_0目录下的tools/build目录，执行：

sudo ./bootstrap.sh
sudo ./b2 install --prefix=/usr/local/boost
cp -rf /usr/local/include/boost/lib/* /usr/lib64 #一定要把库拷贝到/usr/lib64中，否则会报找不到头文件的错误

嘤嘤嘤，没执行cp -rf /usr/local/include/boost/lib/* /usr/lib64的操作，导致一直报以下错误了好几天，软连接也不行。。。

主要是报以下错误，现在终于弄好了，走了好多弯路。

processor/ProcessorTest.o:/usr/include/boost/test/test_tools.hpp:523: more undefined references to `boost::test_tools::tt_detail::check_impl(boost::test_tools::predicate_result const&, boost::unit_test::lazy_ostream const&, boost::unit_test::basic_cstring<char const>, unsigned long, boost::test_tools::tt_detail::tool_level, boost::test_tools::tt_detail::check_type, unsigned long, ...)' follow
g++: error: /usr/local/lib/libboost_unit_test_framework.a: No such file or directory
make[5]: *** [processor_test] Error 1
make[5]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0/lib/cpp/test'
make[4]: *** [all] Error 2
make[4]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0/lib/cpp/test'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0/lib/cpp'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0/lib'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0'
make: *** [all] Error 2

设置环境变量：

vim /etc/profile之后在文件中添加
    export BOOST_INCLUDE = /usr/local/include/boost/include
    export BOOST_LIB =/usr/local/include/boost/lib
source /etc/profile

如果显示这个报错：fatal error: boost/tokenizer.hpp: No such file or directory，尝试使用

yum install boost-devel

到目前为止，boost已经安装完成。判断是否boost安装成功。创建一个cpp文件。

#include <boost/thread/thread.hpp> //包含boost头文件
#include <iostream>
#include <cstdlib>
using namespace std;

volatile bool isRuning = true;

void func1()
{
    static int cnt1 = 0;
    while(isRuning)
    {
        cout << "func1:" << cnt1++ << endl;
        sleep(1);
    }
}

void func2()
{
    static int cnt2 = 0;
    while(isRuning)
    {
        cout << "\tfunc2:" << cnt2++ << endl;
        sleep(2);
    }
}

int main()
{
    boost::thread thread1(&func1);
    boost::thread thread2(&func2);

    system("read");
    isRuning = false;

    thread2.join();
    thread1.join();
    cout << "exit" << endl;
    return 0;
}

然后执行，如果不报错则表示成功：

g++ test.cpp -g -o test -lboost_thread -lboost_system

下面回到thrift路径进行以下操作：

./bootstrap.sh
./configure --with-boost=/usr/local/include/boost
make -j20 && make install
thrift -version #验证是否成功
Thrift version 0.12.0 #显示结果表示已经安装成功

在进行make &&install 指令的时候可能会出现一些错误：

错误1：如果make error和composer有关可能是没有安装php，需要先安装pip，在这里需要安装的是pip7否则会报以下错误：

Problem 1
phpunit/phpunit 5.7.5 requires php ^5.6 || ^7.0 -> your PHP version (5.4.16) does not satisfy that requirement.

解决方案：

yum remove php*
rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
yum install php70w
curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer
chmod +x /usr/local/bin/composer
然后在进行在有composer.ini的文件中composer install

错误二：可能是没有安装php.xml的问题：我是使用的centos 7操作系统，如果用的是ubuntu请使用apt-get下载。

Problem 2:
Installation request for phpunit/phpunit ~4.8.36 -> satisfiable by phpunit/phpunit[4.8.36].

解决方案：

[root@localhost ~]# yum search php70w    查询可供安装的包
[root@localhost ~]# yum -y install php70w-xml.x86_64
[root@localhost ~]# systemctl restart httpd.service  
[root@localhost laravel_demo]# composer install #在含有composer.ini的路径下执行

3.snappy下载：

git clone https://github.com/google/snappy.git
mkdir build
cd build && cmake ../ && make
sudo cp build/libsnappy.a /usr/local/lib/

在安装snappy的时候进行cmake时有可能会显示cmake版本太老。需要进行以下操作，然后在进行cmake操作即可。

cd /home/wangrui
wget https://cmake.org/files/v3.10/cmake-3.10.2-Linux-x86_64.tar.gz
yum remove cmake
tar zxvf cmake-3.10.2-Linux-x86_64.tar.gz
cd cmake-3.10.2-Linux-x86_64
vi /etc/profile
在环境变量中添加：
export CMAKE_HOME=/home/wangrui/cmake-3.10.2-Linux-x86_64
export PATH=$PATH:$CMAKE_HOME/bin


source /etc/profile

4.levelDB安装：

git clone https://github.com/google/leveldb.git
cd leveldb
mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Release .. && cmake --build

然后再build目录下：

make
cd ..
cp -r include/leveldb /usr/local/include/
sudo cp build/libleveldb.a /usr/local/lib/

测试leveldb安装是否成功，创建demo.cc：

#include <cassert>
#include <iostream>
#include <string>
#include <leveldb/db.h>

int main() {
  leveldb::DB* db;
  leveldb::Options options;
  options.create_if_missing = true;
  leveldb::Status status = leveldb::DB::Open(options, "/tmp/testdb", &db);
  assert(status.ok());

  std::string key = "apple";
  std::string value = "A";
  std::string get;

  leveldb::Status s = db->Put(leveldb::WriteOptions(), key, value);

  if (s.ok()) s = db->Get(leveldb::ReadOptions(), key, &get);
  if (s.ok()) std::cout << "读取到的与(key=" << key << ")对应的(value=" << get << ")" << std::endl;
  else std::cout << "读取失败!" << std::endl;

  delete db;

  return 0;
}

命令行中输入：在makefile文件中添加-std=c++11这一行，如果版本g++不一致的话。

g++ -o demo demo.cc -pthread -lleveldb -std=c++11 -lsnappy
./demo
显示以下内容表示level安装成功：
读取到的与(key=apple)对应的(value=A)

5.下载libevent：

wget http://monkey.org/~provos/libevent-2.0.12-stable.tar.gz
tar xfvz libevent-2.0.12-stable.tar.gz
cd libevent-2.0.12-stable
./configure --prefix=/usr/local
make
sudo make install

下载mapkeeper：

git clone https://github.com/sears/mapkeeper.git
export MKROOT=/home/wangrui/downloads/mapkeeper
cd $MKROOT/thrift
make

返回到mapkeeper目录下

cd leveldb
make

make可能会报很多错误：

错误1，如果出现nullptr没有define的错误，错误显示如下：

/usr/local/include/leveldb/options.h:161:30: error: ‘nullptr’ was not declared in this scope
   const Snapshot* snapshot = nullptr;

改正1，应该在makefile文件中添加-std=c++11来解决这个问题。

错误2，如果出现LevelDbServer出现57行错误的话:

Error: LevelDbServer error: conversion from ‘boost::filesystem::path’ to non-scalar type ‘std::string {aka std::basic_string<char>}’ requested

改正2，修改第57行:

std::string mapName = boost::filesystem::path(itr->path().filename()).string();

错误3:

/usr/local/lib/libleveldb.a(table_builder.cc.o): In function `leveldb::TableBuilder::WriteBlock(leveldb::BlockBuilder*, leveldb::BlockHandle*)':
table_builder.cc:(.text+0x83d): undefined reference to `snappy::MaxCompressedLength(unsigned long)'
table_builder.cc:(.text+0x8a2): undefined reference to `snappy::RawCompress(char const*, unsigned long, char*, unsigned long*)'
/usr/local/lib/libleveldb.a(format.cc.o): In function `leveldb::ReadBlock(leveldb::RandomAccessFile*, leveldb::ReadOptions const&, leveldb::BlockHandle const
&, leveldb::BlockContents*)':format.cc:(.text+0x44e): undefined reference to `snappy::GetUncompressedLength(char const*, unsigned long, unsigned long*)'
format.cc:(.text+0x5bf): undefined reference to `snappy::RawUncompress(char const*, unsigned long, char*)'
collect2: error: ld returned 1 exit status

改正3，添加/usr/local/lib/libleveldb.a /usr/local/lib/libsnappy.a对应依赖，主要是因为libleveldb.a misses Snappy when being linked which would be probably in libsnappy.a in the same directory.，同时添加-std=c++11消除版本问题:

include ../Makefile.config
EXECUTABLE = mapkeeper_leveldb
all : thrift
        g++ -Wall -o $(EXECUTABLE) *cpp -I $(THRIFT_DIR)/include/thrift -I $(THRIFT_DIR)/include \
        /usr/local/lib/libleveldb.a /usr/local/lib/libsnappy.a -lboost_thread -lboost_filesystem -lthrift -lleveldb -I ../thrift/gen-cpp \
        -L $(THRIFT_DIR)/lib \
        -L ../thrift/gen-cpp -lmapkeeper \
           -Wl,-rpath,\$$ORIGIN/../thrift/gen-cpp                       \
           -Wl,-rpath,$(THRIFT_DIR)/lib \
        -std=c++11

thrift:
        make -C ../thrift

run:
        ./$(EXECUTABLE) --sync

clean :
        - rm -rf $(THRIFT_SRC) $(EXECUTABLE) *.o

wipe:
        - rm -rf data/*

错误4:如果报错是leveldbserver.cpp的255行和259行，修改相应行数的内容，修改如下：

std::shared_ptr<LevelDbServer> handler(new LevelDbServer("data"));
std::shared_ptr<TProcessor> processor(new MapKeeperProcessor(handler));
std::shared_ptr<TServerTransport> serverTransport(new TServerSocket(port));
std::shared_ptr<TTransportFactory> transportFactory(new TFramedTransportFactory());
std::shared_ptr<TProtocolFactory> protocolFactory(new TBinaryProtocolFactory());

错误5:

/usr/bin/ld: /tmp/cch6BDEI.o: undefined reference to symbol 'pthread_condattr_setclock@@GLIBC_2.3.3'
/usr/lib64/libpthread.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [all] Error 1

解决修改makefile文件，修改后文件内容如下：

include ../Makefile.config
EXECUTABLE = mapkeeper_leveldb
all : thrift
        g++ -Wall -o $(EXECUTABLE) *cpp -I $(THRIFT_DIR)/include/thrift -I $(THRIFT_DIR)/include \
        /usr/local/lib/libleveldb.a /usr/local/lib/libsnappy.a -lpthread -lboost_thread -lboost_system -lboost_filesystem -lthrift -lleveldb -I ../thrift/gen-cpp  \
        -L $(THRIFT_DIR)/lib \
        -L ../thrift/gen-cpp -lmapkeeper \
        -Wl,-rpath,\$$ORIGIN/../thrift/gen-cpp                       \
        -Wl,-rpath,$(THRIFT_DIR)/lib \
        -std=c++11
thrift:
        make -C ../thrift

run:
        ./$(EXECUTABLE) --sync

clean :
        -rm -rf $(THRIFT_SRC) $(EXECUTABLE) *.o

wipe:
        -rm -rf data/*

运行mapkeeper，成功运行表示成功：

./mapkeeper_leveldb 1 -d ~/mapkeeper/level/data

然后是下载ycsb加到mapkeeper/ycsb中（我试过好多版本都会make出错，我也很无奈是在没办法了，就选用特定这个0.1.4版本吧，能运行成功）：如果想找这个版本可以到我的博客中进行下载。

wget https://github.com/downloads/brianfrankcooper/YCSB/ycsb-0.1.4.tar.gz
tar xfvz ycsb-0.1.4
cd ycsb-0.1.4
bin/ycsb load mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090"
bin/ycsb run mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090"

显示类似(截取一段)：

[UPDATE], 296, 0
[UPDATE], 297, 0
[UPDATE], 298, 0
[UPDATE], 299, 0
[UPDATE], 300, 0
[UPDATE], 301, 0
[UPDATE], 302, 0
[UPDATE], 303, 0
[UPDATE], 304, 0
[UPDATE], 305, 0
[UPDATE], 306, 0
[UPDATE], 307, 0
[UPDATE], 308, 0
[UPDATE], 309, 0
[UPDATE], 310, 0
[UPDATE], 311, 0
[UPDATE], 312, 0
[UPDATE], 313, 0
[UPDATE], 314, 0
[UPDATE], 315, 0
[UPDATE], 316, 0
[UPDATE], 317, 0

表示成功。

在最后的最后，我重启了服务器发现/home文件夹下什么都没有了。哭了。。。

最后发现是重启之后没有挂载硬盘，执行下面命令就好了。

mount /dev/nvme0n1 /home/hsk/kvstore

批量测试leveldb在ycsb工作负载下运行情况，编写run.sh。（一定要在每一次load和run一个工作负载文件之后删除数据库路径“rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*”下的所有文件）否则会报错。同时通过ulimit查看open file文件设置的上限，并把上限通过ulimit -n 65535指令调到64MB大小，否则会报如下错误。

insert not ok! IO error: While open a file for random read: /home/hsk/kvstore/software/mapkeeper/rocksdb/data/usertable/011619.sst: Too many open files
status: IO error: lock : /home/hsk/kvstore/software/mapkeeper/rocksdb/data/usertable/LOCK: No locks available

运行脚本如下，放在ycsb所在路径下：

bin/ycsb load mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/1_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/1_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloadb -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/2_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloadb -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/2_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloadc -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/3_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloadc -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/3_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloadd -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/4_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloadd -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/4_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloade -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/5_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloade -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/5_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloadf -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/6_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloadf -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/6_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*

机智帅气的葫芦娃

关注

1
点赞
踩
7

收藏

觉得还不错? 一键收藏
0
评论
YCSB负载工具测试leveldb

网上对有关YCSB怎么连接leveldb的讲解简直是太少了。我弄了好久才弄好分享下自己的经验，也为之后自己不用再查一遍。首先下载需要的一些包：（我的系统版本是centos 7）1.首先安装thrift：wget http://mirrors.tuna.tsinghua.edu.cn/apache/thrift/0.12.0/thrift-0.12.0.tar.gztar zxvf ...
复制链接

扫一扫