HiBench生成基准数据集【WordCount为例】

1.下载HiBench
官网地址：https://github.com/Intel-bigdata/HiBench/tree/HiBench-7.1

2.上传服务器
过程略
3.解压文件
由于压缩包是.zip文件，无法使用tar解压，需要下载unzip

yum install unzip

unzip xxx.zip

4.修改配置文件

1. 进入到conf目录

cp hadoop.conf.template hadoop.conf

2. 修改 hadoop.conf（根据自己节点情况配置）

# Hadoop home
hibench.hadoop.home     /usr/local/hadoop-3.1.1

# The path of hadoop executable
hibench.hadoop.executable     ${hibench.hadoop.home}/bin/hadoop

# Hadoop configraution directory
hibench.hadoop.configure.dir  ${hibench.hadoop.home}/etc/hadoop

# The root HDFS path to store HiBench data
hibench.hdfs.master       hdfs://192.168.152.109:9820/hibench


# Hadoop release provider. Supported value: apache, cdh5, hdp
hibench.hadoop.release    apache
~                                 

4. 进入conf/workloads/micro/,wordcount.conf文件配置的是生成的数据量大小

这里我自定义了一个1G大小的数据集

#datagen
#hibench.wordcount.tiny.datasize                 32000
#hibench.wordcount.small.datasize                320000000
#hibench.wordcount.large.datasize                3200000000
#hibench.wordcount.huge.datasize                 32000000000
#hibench.wordcount.gigantic.datasize             320000000000
#hibench.wordcount.bigdata.datasize              1600000000000

hibench.wordcount.large.datasize                1073741824


hibench.workload.datasize               ${hibench.wordcount.${hibench.scale.profile}.datasize}

# export for shell script
hibench.workload.input                  ${hibench.hdfs.data.dir}/Wordcount/Input
hibench.workload.output                 ${hibench.hdfs.data.dir}/Wordcount/Output

6. 配置conf/hibench.conf

注意标记处类型要与上面wordcount.conf定义的大小名称相同！

7. 执行测试脚本
在bin/run_all.sh 该脚本为测试所有的测试基准模块(将运行所有在conf/benchmarks.lst和conf/frameworks.lst中的workloads)；

这里还是以wordcount为例，

①生成测试数据 bin/workloads/micro/wordcount/prepare/prepare.sh

②运行wordcount测试例子 bin/workloads/micro/wordcount/hadoop/run.sh

③生成的测试数据在conf/hadoop.conf中hibench.hdfs.master项配置，我的是在/user/hibench/HiBench目录下

执行命令①

生成成功！