hibench 之cdh5-hadoop集群性能测试

1.下载源码

https://github.com/intel-hadoop/HiBench

2.全部编译（两个半小时）

mvn -Dspark=2.1 -Dscala=2.11 clean package

3.修改hadoop配置文件 conf/hadoop.conf

  我的参考配置：

  

# Hadoop home
hibench.hadoop.home /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop

# The path of hadoop executable
hibench.hadoop.executable /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/bin/hadoop

# Hadoop configraution directory
hibench.hadoop.configure.dir /etc/hadoop/conf

# The root HDFS path to store HiBench data
hibench.hdfs.master hdfs://spark-4:8020

# Hadoop release provider. Supported value: apache, cdh5, hdp
hibench.hadoop.release cdh5

4.修改spark配置文件 conf/spark.conf

我的参考配置：

# Spark home
hibench.spark.home /opt/cloudera/parcels/SPARK2/lib/spark2

# Spark master
# standalone mode: spark://xxx:7077
# YARN mode: yarn-client
hibench.spark.master yarn-client

# executor number and cores when running on Yarn
hibench.yarn.executor.num 1
hibench.yarn.executor.cores 2

# executor and driver memory in standalone & YARN mode
spark.executor.memory 512m
spark.driver.memory 512m

5.编辑conf/benchmarks.lst 选择你要测试的模块以及功能，如以wordcount为例子

6.编辑conf/frameworks.lst 选择你要测试的环境如hadoop或spark

7.准备数据

   bin/workloads/micro/wordcount/prepare/prepare.sh

8.运行测试的项目
   bin/workloads/micro/wordcount/spark/run.sh

9 查看结果

a、在HiBench/report/hibench.report中查看 workload name, execution duration, data size, throughput per cluster, throughput per node等信息

 

   b、在HiBench/report目录下还有cpu、内存、io的使用报告以及使用图

10、注意点

   a、集群每台机器必须安装 

   yum install bc 计算 才会出来hibench.report文件

   b、数据规模也是可以设置的：修改conf/hibench.conf里面的hibench.scale.profile项

 

转载于:https://www.cnblogs.com/damoyuan/p/8034368.html