Spark 性能测试

运行SparkBench 测试

尝试在单个服务器上进行SparkBench的测试。

测试平台

单台服务器测试

前置条件

了解并安装了Spark软件、SparkBench软件。

• 开源项目官网

• 安装文档

测试方法

采用SparkBench开源测试方案，对比目前的集群

生成随机数据

<Workload>/bin/gen_data.sh

运行

<Workload>/bin/run.sh

测试结果

Apptype (测试方法)	start_ts (开始时间)	duration (运行时间)	size (输入数据大小MB)	throughput (输出数据大小MB)
ConnectedComponent-gen	2018-08-05-17:26:36	16.850000	88.996429	5.281687
ConnectedComponent	2018-08-05-17:27:10	37.094000	88.996429	2.399213
DecisionTree-gen	2018-08-05-17:28:30	10.579000	0.060088	0.005679
DecisionTree	2018-08-05-17:28:54	13.434000	0.060088	0.004472
KMeans	2018-08-05-17:29:28	9.968000	0.372898	0.037409
LinearRegression-gendata	2018-08-05-17:31:36	10.804000	3.924393	0.363235
LinearRegression	2018-08-05-17:32:01	18.483000	3.924393	0.212324
LogisticRegression-gendata	2018-08-05-17:32:35	10.861000	7.598749	0.699636
LogisticRegression	2018-08-05-17:33:00	17.159000	7.598749	0.442843
PageRank-gen	2018-08-05-17:35:56	10.345000	0.147607	0.014268
PageRank	2018-08-05-17:36:20	14.422000	0.147607	0.010234
PCA-gen	2018-08-05-17:36:53	11.193000	18.873979	1.686230
PCA	2018-08-05-17:37:14	23.754000	18.873979	0.794560
PregelOperation-gen	2018-08-05-17:37:51	12.956000	0.831987	0.064216
PregelOperation	2018-08-05-17:38:17	19.383000	0.831987	0.042923
ShortestPaths-gen	2018-08-05-17:38:50	26.032000	0.279758	0.010746
ShortestPaths	2018-08-05-17:39:30	18.323000	0.279758	0.015268
sql	2018-08-05-17:40:05	2.170000	0.056137	0.025869
sql_rddRelation	2018-08-05-17:40:19	25.076000	0.056137	0.002238
StronglyConnectedComponent-gen	2018-08-05-17:41:30	13.650000	0.032388	0.002372
StronglyConnectedComponent	2018-08-05-17:41:55	73.910000	0.032388	0.000438
SVDPlusPlus-gen	2018-08-05-17:43:23	15.839000	81.302511	5.133058
SVDPlusPlus	2018-08-05-17:43:51	97.667000	81.302511	0.832446
SVM-gen	2018-08-05-17:46:38	12.318000	3.924085	0.318565
SVM	2018-08-05-17:47:14	25.643000	3.924085	0.153027
Terasort-gen	2018-08-05-17:48:37	12.516000	0.095367	0.007619
Terasort	2018-08-05-17:49:02	12.101000	0.095367	0.007880
TriangleCount-gen	2018-08-05-17:49:26	15.967000	12.807078	0.802096
TriangleCount	2018-08-05-17:49:59	85.970000	12.807078	0.148971

以上为得到的测试结果，仅供参考！