1.下载HiBench
官网地址:https://github.com/Intel-bigdata/HiBench/tree/HiBench-7.1
2.上传服务器
过程略
3.解压文件
由于压缩包是.zip文件,无法使用tar解压,需要下载unzip
yum install unzip
unzip xxx.zip
4.修改配置文件
- 进入到conf目录
cp hadoop.conf.template hadoop.conf
- 修改 hadoop.conf(根据自己节点情况配置)
# Hadoop home
hibench.hadoop.home /usr/local/hadoop-3.1.1
# The path of hadoop executable
hibench.hadoop.executable ${hibench.hadoop.home}/bin/hadoop
# Hadoop configraution directory
hibench.hadoop.configure.dir ${hibench.hadoop.home}/etc/hadoop
# The root HDFS path to store HiBench data
hibench.hdfs.master hdfs://192.168.152.109:9820/hibench
# Hadoop release provider. Supported value: apache, cdh5, hdp
hibench.hadoop.release apache
~
- 进入conf/workloads/micro/,wordcount.conf文件配置的是生成的数据量大小
这里我自定义了一个1G大小的数据集
#datagen
#hibench.wordcount.tiny.datasize 32000
#hibench.wordcount.small.datasize 320000000
#hibench.wordcount.large.datasize 3200000000
#hibench.wordcount.huge.datasize 32000000000
#hibench.wordcount.gigantic.datasize 320000000000
#hibench.wordcount.bigdata.datasize 1600000000000
hibench.wordcount.large.datasize 1073741824
hibench.workload.datasize ${hibench.wordcount.${hibench.scale.profile}.datasize}
# export for shell script
hibench.workload.input ${hibench.hdfs.data.dir}/Wordcount/Input
hibench.workload.output ${hibench.hdfs.data.dir}/Wordcount/Output
- 配置conf/hibench.conf
注意标记处类型要与上面wordcount.conf定义的大小名称相同!
7. 执行测试脚本
在bin/run_all.sh 该脚本为测试所有的测试基准模块(将运行所有在conf/benchmarks.lst和conf/frameworks.lst中的workloads);
这里还是以wordcount为例,
①生成测试数据 bin/workloads/micro/wordcount/prepare/prepare.sh
②运行wordcount测试例子 bin/workloads/micro/wordcount/hadoop/run.sh
③生成的测试数据在conf/hadoop.conf中hibench.hdfs.master项配置,我的是在/user/hibench/HiBench目录下
执行命令①
生成成功!