cosbench 部署配置

作者： Yang Honggang <rtlinux@163.com>

下载

  // 不能直接使用 git clone 的版本，应该使用 release 下的
  // ref: http://cosbench.1094679.n5.nabble.com/cosbench-3-3-0-on-ubuntu-13-10-td114.html
  wget https://github.com/intel-cloud/cosbench/releases/download/v0.4.2.c4/0.4.2.c4.zip

解压

# unzip 0.4.2.c4.zip

启动

  # yum install nmap-ncat
  # ln -s 0.4.2.c4 cos
  # cd cos
  # unset http_proxy

  [root@yhg-1 cos]# sh start-all.sh 
  Launching osgi framwork ... 
  Successfully launched osgi framework!
  Booting cosbench driver ... 
  .
  Starting    cosbench-log_0.4.2    [OK]
  .
  Starting    cosbench-tomcat_0.4.2    [OK]
  Starting    cosbench-config_0.4.2    [OK]
  Starting    cosbench-http_0.4.2    [OK]
  Starting    cosbench-cdmi-util_0.4.2    [OK]
  Starting    cosbench-core_0.4.2    [OK]
  Starting    cosbench-core-web_0.4.2    [OK]
  Starting    cosbench-api_0.4.2    [OK]
  Starting    cosbench-mock_0.4.2    [OK]
  Starting    cosbench-ampli_0.4.2    [OK]
  Starting    cosbench-swift_0.4.2    [OK]
  Starting    cosbench-keystone_0.4.2    [OK]
  Starting    cosbench-httpauth_0.4.2    [OK]
  Starting    cosbench-s3_0.4.2    [OK]
  Starting    cosbench-librados_0.4.2    [OK]
  Starting    cosbench-scality_0.4.2    [OK]
  Starting    cosbench-cdmi-swift_0.4.2    [OK]
  Starting    cosbench-cdmi-base_0.4.2    [OK]
  Starting    cosbench-driver_0.4.2    [OK]
  Starting    cosbench-driver-web_0.4.2    [OK]
  Successfully started cosbench driver!
  Listening on port 0.0.0.0/0.0.0.0:18089 ... 
  Persistence bundle starting...
  Persistence bundle started.
  ----------------------------------------------
  !!! Service will listen on web port: 18088 !!!
  ----------------------------------------------
  
  ======================================================
  
  Launching osgi framwork ... 
  Successfully launched osgi framework!
  Booting cosbench controller ... 
  Ncat: Connection refused.
  ..
  Starting    cosbench-log_0.4.2    [OK]
  .
  Starting    cosbench-tomcat_0.4.2    [OK]
  Starting    cosbench-config_0.4.2    [OK]
  Starting    cosbench-core_0.4.2    [OK]
  Starting    cosbench-core-web_0.4.2    [OK]
  Starting    cosbench-controller_0.4.2    [OK]
  Starting    cosbench-controller-web_0.4.2    [OK]
  Successfully started cosbench controller!
  Listening on port 0.0.0.0/0.0.0.0:19089 ... 
  Persistence bundle starting...
  Persistence bundle started.
  ----------------------------------------------
  !!! Service will listen on web port: 19088 !!!
  ----------------------------------------------

检查 controller和driver是否正启动

  [root@yhg-1 cos]# netstat -an | grep LISTEN| grep 19088
  tcp6       0      0 :::19088                :::*                    LISTEN     
  [root@yhg-1 cos]# netstat -an | grep LISTEN| grep 18088
  tcp6       0      0 :::18088                :::*                    LISTEN 

浏览器中查看 controller

http://192.168.12.79:19088/controller/index.html

测试任务

  [root@yhg-1 cos]# sh cli.sh submit conf/workload-config.xml 
  Accepted with ID: w1
  [root@yhg-1 cos]# sh cli.sh info
  Drivers:
  driver1 http://127.0.0.1:18088/driver
  Total: 1 drivers
  
  Active Workloads:
  w1      Fri Sep 29 10:23:15 UTC 2017    PROCESSING      s1-init
  Total: 1 active workloads
  
  [root@yhg-1 cos]# sh cli.sh cancel w1
  w1      Fri Sep 29 10:23:15 UTC 2017    CANCELLED
  
  [root@yhg-1 cos]# sh cli.sh info
  Drivers:
  driver1 http://127.0.0.1:18088/driver
  Total: 1 drivers
  
  Active Workloads:
  Total: 0 active workloads

workload例子（针对ceph rgw的）

<?xml version="1.0" encoding="UTF-8" ?>
<workload name="100000000-8k-shard-16" description="yhg benchmark for s3">

  <storage type="s3" config="accesskey=8WKYZ8E6HYV2R62VLSMG;secretkey=mBwmRiJ6zHCygmJboNHWqW3VCc9Aq5oDbmtcQQYo;proxyhost=;proxyport=;endpoint=http://192.168.10.8:99;path_style_access=true" />

  <workflow>

    <workstage name="init">
      <work type="init" workers="1" config="cprefix=lvbu;containers=r(1,2)" />
    </workstage>

    <workstage name="write">
      <work name="write" workers="256" totalOps="200000000">
        <operation type="write" ratio="100" config="cprefix=lvbu;containers=u(1,2);objects=u(1,100000000);sizes=c(8)KB" />
      </work>
    </workstage>

    <workstage name="cleanup">
      <work type="cleanup" workers="128" config="cprefix=lvbu;containers=r(1,2);objects=r(1,100000000)" />
    </workstage>

    <workstage name="dispose">
      <work type="dispose" workers="1" config="cprefix=lvbu;containers=r(1,2)" />
    </workstage>

  </workflow>

</workload>

再添加一个driver节点

  [root@xt9 cos]# ./start-driver.sh 
  Launching osgi framwork ... 
  Successfully launched osgi framework!
  Booting cosbench driver ... 
  Starting    cosbench-log_0.4.2    [OK]
  .
  Starting    cosbench-tomcat_0.4.2    [OK]
  Starting    cosbench-config_0.4.2    [OK]
  Starting    cosbench-http_0.4.2    [OK]
  Starting    cosbench-cdmi-util_0.4.2    [OK]
  Starting    cosbench-core_0.4.2    [OK]
  Starting    cosbench-core-web_0.4.2    [OK]
  Starting    cosbench-api_0.4.2    [OK]
  Starting    cosbench-mock_0.4.2    [OK]
  Starting    cosbench-ampli_0.4.2    [OK]
  Starting    cosbench-swift_0.4.2    [OK]
  Starting    cosbench-keystone_0.4.2    [OK]
  Starting    cosbench-httpauth_0.4.2    [OK]
  Starting    cosbench-s3_0.4.2    [OK]
  Starting    cosbench-librados_0.4.2    [OK]
  Starting    cosbench-scality_0.4.2    [OK]
  Starting    cosbench-cdmi-swift_0.4.2    [OK]
  Starting    cosbench-cdmi-base_0.4.2    [OK]
  Starting    cosbench-driver_0.4.2    [OK]
  Starting    cosbench-driver-web_0.4.2    [OK]
  Successfully started cosbench driver!
  Listening on port 0.0.0.0/0.0.0.0:18089 ... 
  Persistence bundle starting...
  Persistence bundle started.
  ----------------------------------------------
  !!! Service will listen on web port: 18088 !!!
  ----------------------------------------------

修改 conf/controller.conf

  [controller]
  drivers = 2
  log_level = INFO
  log_file = log/system.log
  archive_dir = archive
  
  [driver1]
  name = driver1
  url = http://127.0.0.1:18088/driver
  
  [driver2]
  name = driver2
  url = http://xt9:18088/driver

重启 controller

  # ./stop-all.sh
  # ./start-all.sh

查看所有 driver

  [root@xt8 cos]# sh cli.sh info
  Drivers:
  driver1 http://127.0.0.1:18088/driver
  driver2 http://xt9:18088/driver
  Total: 2 drivers
  
  Active Workloads:
  Total: 0 active workloads

常见问题记录

Unable to verify integrity of data download

  // master 分支中可能已经fix了该问题
  // ref: https://github.com/intel-cloud/cosbench/issues/346
  // ref: http://www.cnblogs.com/bodhitree/p/6140469.html
  2017-10-09 10:58:29,544 [ERROR] [AbstractOperator] - worker 59 fail to perform operation s3testqwer2/myobjects8
  com.amazonaws.AmazonClientException: Unable to verify integrity of data download.  Client calculated content hash didn't match hash calculated by Amazon S3.  The data may be corrupt.
          at com.amazonaws.services.s3.internal.DigestValidationInputStream.validateMD5Digest(DigestValidationInputStream.java:79)
          at com.amazonaws.services.s3.internal.DigestValidationInputStream.read(DigestValidationInputStream.java:61)
          at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155)
          at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:147)
          at com.intel.cosbench.driver.operator.Reader.copyLarge(Reader.java:120)
          at com.intel.cosbench.driver.operator.Reader.doRead(Reader.java:92)
          at com.intel.cosbench.driver.operator.Reader.operate(Reader.java:69)
          at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)
          at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)
          at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)
          at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)
          at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)
          at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)

解决方案

    stop-all.sh 停止 cosbench 进程(controller, drivers)
    编辑 cosbench-start.sh 文件,找到java启动命令行参数, 增加"-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true" 关闭S3的MD5校验功能
    start-all.sh 启动所有的cosbench processes.

同一个节点添加多个driver(3个)

将 start-driver.sh 拷贝2两份，分别命名为 start-driver2.sh、start-driver3.sh

对于 start-driver2.sh 只需要修改 base_port=18077

对于start-driver3.sh 只需要修改 base_port=18066

修改 start-all.sh 如下

# cat start-all.sh 
#!/bin/bash
#
#Copyright 2013 Intel Corporation, All Rights Reserved.
#
#Licensed under the Apache License, Version 2.0 (the "License");
#you may not use this file except in compliance with the License.
#You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
#Unless required by applicable law or agreed to in writing, software
#distributed under the License is distributed on an "AS IS" BASIS,
#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#See the License for the specific language governing permissions and
#limitations under the License.
#

bash start-driver.sh
bash start-driver2.sh
bash start-driver3.sh
echo
echo "======================================================"
echo
bash start-controller.sh

修改controller.conf如下

# cat conf/controller.conf 
[controller]
drivers = 3
log_level = INFO
log_file = log/system.log
archive_dir = archive


[driver1]
name = driver1
url = http://127.0.0.1:18088/driver

[driver2]
name = driver2
url = http://127.0.0.1:18077/driver

[driver3]
name = driver3
url = http://127.0.0.1:18066/driver

参考

[1] http://www.cnblogs.com/landhu/p/5843282.html

[2] COSBenchUserGuide.pdf

[3] http://cosbench.1094679.n5.nabble.com/