一次Giraph自定义的benchmark(KMeansBenchmark)实现过程

最新推荐文章于 2021-01-15 14:46:49 发布
最新推荐文章于 2021-01-15 14:46:49 发布
阅读量563 收藏
点赞数
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/cloudeagle_bupt/article/details/54633324
版权

这里Kmeans的用例来自 https://github.com/tmalaska/Giraph.KMeans.Example/tree/master/src,  需要做修改部分源码


1)CmmmandLine类(commons-cli-1.2.jar)中两个链表，分别存储识别的和未识别的参数，根据"-"来进行区分。

   2)在
   在giraph-env中替换GiraphRunner为自己定义的benchmark,需要implementes Tool 且extends GiraphBenchmark.
   对于自定义的参数，如-vif等，需要在解析之前加进getBenchmarkOptions()中，这些所有参数的选项名称(如-w,-vof等)，如果不加，在后面
   CommandLineParser parser = new PosixParser();
    CommandLine cmd = parser.parse(options, args);
解析参数时，会被视为未定义参数报错。


  执行命令为:  giraph ../giraph-core-1.1.0.jar  org.apache.giraph.benchmark.kmeans.KMeansComputation -clu 5 -dim 2 -mIr 5  -vip /test/youTube.txt -op /output  -w 1 
  注意(../giraph-core-1.1.0.jar不能省略),否则会报:17/01/20 17:17:42 WARN fs.FSInputChecker: Problem opening checksum file: file:/opt/hadoop-1.2.1/logs/history/job_201611032251_0013_1478242352396_liuqiang2_Giraph%3A+org.apache.giraph.benchmark.PageRankComput.  Ignoring exception: java.io.EOFException
  原因不明,猜测可能是由于执行的jar包没有传过去导致分布式系统和本地文件校验和不同出错。
   
  giraph ../giraph-core-1.1.0.jar  org.apache.giraph.benchmark.kmeans.KMeansComputation -clu 5 -dim 2 -mIr 5  -vip /test/dataset.txt -op /output  -w 1 -v true


  注意下: -v参数指 verbose, 即是否打印细节: 

   另外, KMeansBenchmark 代码定义如下:

package org.apache.giraph.benchmark.kmeans;

import java.io.IOException;
import java.util.Set;

import org.apache.commons.cli.CommandLine;
import org.apache.giraph.benchmark.BenchmarkOption;
import org.apache.giraph.benchmark.GiraphBenchmark;
import org.apache.giraph.conf.GiraphConfiguration;
import org.apache.giraph.io.formats.GiraphFileInputFormat;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.ToolRunner;

import com.google.common.collect.Sets;

public class KMeansBenchmark extends GiraphBenchmark {

	/**
	 * @param args
	 */
	public static void main(String[] args) throws Exception {
		System.exit(ToolRunner.run(new KMeansBenchmark(), args));
	}

	/**
	 * 所有的参数选项声明
	 */
	@Override
	public Set<BenchmarkOption> getBenchmarkOptions() {
		 return Sets.newHashSet(new BenchmarkOption("vip", "vertexInputPath", true, "Vertex input path"),
				 new BenchmarkOption("op",  "outputPath", true, "Output path"),
				 new BenchmarkOption("clu",  "clusterNumber", true, "number of clusters"),
				 new BenchmarkOption("dim",  "dimension", true, "number of dimensions"),
				 new BenchmarkOption("mIr",  "maxIterations", true, "max Iterations"));
	}

	@Override
	protected void prepareConfiguration(GiraphConfiguration conf,
			CommandLine cmd) {
		try {
			conf.setComputationClass(KMeansComputation.class);
			conf.setVertexInputFormatClass(KMeansTextVertexInputFormat.class);
			conf.setVertexOutputFormatClass(KMeansTextVertexOutputFormat.class);
			conf.setWorkerContextClass(KMeansNodeWorkerContext.class);
			conf.setMasterComputeClass(KMeansVertixMasterCompute.class);
			int workers = Integer.parseInt(BenchmarkOption.WORKERS.getOptionValue(cmd));
			conf.setWorkerConfiguration(workers, workers, 100.0f);
			conf.set(Const.NUMBER_OF_CLUSTERS, cmd.getOptionValue("clu"));
			conf.set(Const.NUMBER_OF_DIMENSIONS, cmd.getOptionValue("dim"));
			conf.set(Const.MAX_ITERATIONS, cmd.getOptionValue("mIr"));
			
			int workers2 = conf.getMinWorkers() ;
			if (cmd.hasOption("vip")) {
				if (FileSystem.get(new Configuration()).listStatus(
						new Path(cmd.getOptionValue("vip"))) == null) {
					throw new IllegalArgumentException(
							"Invalid vertex input path (-vip): "
									+ cmd.getOptionValue("vip"));
				}
				GiraphFileInputFormat.addVertexInputPath(conf,
						new Path(cmd.getOptionValue("vip")));
			}
		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}


另外,  GiraphBenchmark  代码也需要进行一定修改:

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.giraph.benchmark;

import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.CommandLineParser;
import org.apache.commons.cli.HelpFormatter;
import org.apache.commons.cli.Options;
import org.apache.commons.cli.PosixParser;
import org.apache.giraph.conf.GiraphConfiguration;
import org.apache.giraph.job.GiraphJob;
import org.apache.giraph.utils.LogVersions;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.log4j.Logger;

import java.io.IOException;
import java.util.Set;

/**
 * Abstract class which benchmarks should extend.
 */
public abstract class GiraphBenchmark implements Tool {
  /** Class logger */
  public static final Logger LOG = Logger.getLogger(GiraphBenchmark.class);
  /** Configuration */
  private Configuration conf;

  @Override
  public void setConf(Configuration conf) {
    this.conf = conf;
  }

  @Override
  public Configuration getConf() {
    return conf;
  }

  @Override
  public int run(String[] args) throws Exception {
    Set<BenchmarkOption> giraphOptions = getBenchmarkOptions();
    giraphOptions.add(BenchmarkOption.HELP);
    giraphOptions.add(BenchmarkOption.VERBOSE);
    giraphOptions.add(BenchmarkOption.WORKERS);
    Options options = new Options();
    for (BenchmarkOption giraphOption : giraphOptions) {
      giraphOption.addToOptions(options);
    }

    HelpFormatter formatter = new HelpFormatter();
    if (args.length == 0) {
      formatter.printHelp(getClass().getName(), options, true);
      return 0;
    }
    CommandLineParser parser = new PosixParser();
    CommandLine cmd = parser.parse(options, args);
    for (BenchmarkOption giraphOption : giraphOptions) {
      if (!giraphOption.checkOption(cmd, LOG)) {
        return -1;
      }
    }
    if (BenchmarkOption.HELP.optionTurnedOn(cmd)) {
      formatter.printHelp(getClass().getName(), options, true);
      return 0;
    }

    GiraphJob job = new GiraphJob(getConf(), getClass().getName());
    int workers = Integer.parseInt(BenchmarkOption.WORKERS.getOptionValue(cmd));

    prepareHadoopMRJob(job, cmd);
    GiraphConfiguration giraphConf = job.getConfiguration();
    giraphConf.addWorkerObserverClass(LogVersions.class);
    giraphConf.addMasterObserverClass(LogVersions.class);

    giraphConf.setWorkerConfiguration(workers, workers, 100.0f);
    prepareConfiguration(giraphConf, cmd);

    boolean isVerbose = false;
    if (BenchmarkOption.VERBOSE.optionTurnedOn(cmd)) {
      isVerbose = true;
    }
    if (job.run(isVerbose)) {
      return 0;
    } else {
      return -1;
    }
  }

  private void prepareHadoopMRJob(GiraphJob job, CommandLine cmd) {
	  if (cmd.hasOption("op")) {
			FileOutputFormat.setOutputPath(job.getInternalJob(),
					new Path(cmd.getOptionValue("op")));
	  }
  }

/**
   * Get the options to use in this benchmark.
   * BenchmarkOption.VERBOSE, BenchmarkOption.HELP and BenchmarkOption.WORKERS
   * will be added automatically, so you don't have to specify those.
   *
   * @return Options to use in this benchmark
   */
  public abstract Set<BenchmarkOption> getBenchmarkOptions();

  /**
   * Process options from CommandLine and prepare configuration for running
   * the job.
   * BenchmarkOption.VERBOSE, BenchmarkOption.HELP and BenchmarkOption.WORKERS
   * will be processed automatically so you don't have to process them.
   *
   * @param conf Configuration
   * @param cmd Command line
   */
  protected abstract void prepareConfiguration(GiraphConfiguration conf,
      CommandLine cmd);
}


结果:

17/01/20 21:18:09 INFO mapred.JobClient: Running job: job_201701201708_0025
17/01/20 21:18:10 INFO mapred.JobClient:  map 100% reduce 0%
17/01/20 21:18:17 INFO mapred.JobClient: Job complete: job_201701201708_0025
17/01/20 21:18:17 INFO mapred.JobClient: Counters: 48
17/01/20 21:18:17 INFO mapred.JobClient:   Zookeeper halt node
17/01/20 21:18:17 INFO mapred.JobClient:     /_hadoopBsp/job_201701201708_0025/_haltComputation=0
17/01/20 21:18:17 INFO mapred.JobClient:   Zookeeper base path
17/01/20 21:18:17 INFO mapred.JobClient:     /_hadoopBsp/job_201701201708_0025=0
17/01/20 21:18:17 INFO mapred.JobClient:   Job Counters 
17/01/20 21:18:17 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=45288
17/01/20 21:18:17 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
17/01/20 21:18:17 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
17/01/20 21:18:17 INFO mapred.JobClient:     Launched map tasks=2
17/01/20 21:18:17 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
17/01/20 21:18:17 INFO mapred.JobClient:   Giraph Timers
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 8 KMeansComputation (ms)=73
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 9 KMeansComputation (ms)=75
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 7 KMeansComputation (ms)=77
17/01/20 21:18:17 INFO mapred.JobClient:     Input superstep (ms)=423
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 2 KMeansComputation (ms)=78
17/01/20 21:18:17 INFO mapred.JobClient:     Shutdown (ms)=8953
17/01/20 21:18:17 INFO mapred.JobClient:     Initialize (ms)=11614
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 0 KMeansComputation (ms)=91
17/01/20 21:18:17 INFO mapred.JobClient:     Setup (ms)=111
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 3 KMeansComputation (ms)=83
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 10 KMeansComputation (ms)=71
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 1 KMeansComputation (ms)=112
17/01/20 21:18:17 INFO mapred.JobClient:     Total (ms)=10379
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 4 KMeansComputation (ms)=82
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 6 KMeansComputation (ms)=74
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep 5 KMeansComputation (ms)=73
17/01/20 21:18:17 INFO mapred.JobClient:   Zookeeper server:port
17/01/20 21:18:17 INFO mapred.JobClient:     c02b03:22181=0
17/01/20 21:18:17 INFO mapred.JobClient:   Giraph Stats
17/01/20 21:18:17 INFO mapred.JobClient:     Aggregate edges=0
17/01/20 21:18:17 INFO mapred.JobClient:     Sent message bytes=0
17/01/20 21:18:17 INFO mapred.JobClient:     Superstep=11
17/01/20 21:18:17 INFO mapred.JobClient:     Last checkpointed superstep=0
17/01/20 21:18:17 INFO mapred.JobClient:     Current workers=1
17/01/20 21:18:17 INFO mapred.JobClient:     Aggregate sent messages=0
17/01/20 21:18:17 INFO mapred.JobClient:     Current master task partition=0
17/01/20 21:18:17 INFO mapred.JobClient:     Sent messages=0
17/01/20 21:18:17 INFO mapred.JobClient:     Aggregate finished vertices=24
17/01/20 21:18:17 INFO mapred.JobClient:     Aggregate sent message message bytes=0
17/01/20 21:18:17 INFO mapred.JobClient:     Aggregate vertices=24
17/01/20 21:18:17 INFO mapred.JobClient:   File Output Format Counters 
17/01/20 21:18:17 INFO mapred.JobClient:     Bytes Written=0
17/01/20 21:18:17 INFO mapred.JobClient:   FileSystemCounters
17/01/20 21:18:17 INFO mapred.JobClient:     HDFS_BYTES_READ=311
17/01/20 21:18:17 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=268140
17/01/20 21:18:17 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=296
17/01/20 21:18:17 INFO mapred.JobClient:   File Input Format Counters 
17/01/20 21:18:17 INFO mapred.JobClient:     Bytes Read=0
17/01/20 21:18:17 INFO mapred.JobClient:   Map-Reduce Framework
17/01/20 21:18:17 INFO mapred.JobClient:     Map input records=2
17/01/20 21:18:17 INFO mapred.JobClient:     Physical memory (bytes) snapshot=791097344
17/01/20 21:18:17 INFO mapred.JobClient:     Spilled Records=0
17/01/20 21:18:17 INFO mapred.JobClient:     CPU time spent (ms)=9390
17/01/20 21:18:17 INFO mapred.JobClient:     Total committed heap usage (bytes)=1331167232
17/01/20 21:18:17 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=3453415424
17/01/20 21:18:17 INFO mapred.JobClient:     Map output records=0
17/01/20 21:18:17 INFO mapred.JobClient:     SPLIT_RAW_BYTES=88


注意到10次迭代后  Aggregate sent message message bytes=0 为0，可见kmeans确实是计算密集型计算。


假设有7个任务，每个任务首轮根据中心点数组计算各自分区的点的分类，划分完毕后各任务重新生成新的中心点数据加入aggregator并汇总至master, master生成新的中心点数组发送至各子节点进行下一轮计算。中间需要发送的消息仅仅是更新后的中心点数据信息，通信量非常小.



飞火流云
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
使用sklearn不同方法在digits手写数字数据集上聚类并用matplotlib呈现
碎片
10-27 7738
实验题目 Clustering with sklearn 实验内容 测试sklearn中以下聚类算法在以上两个数据集上的聚类效果。 使用不同的评估方法对实验结果进行评估。 实验准备 sklearn库 自2007年发布以来，scikit-learn已经成为Python重要的机器学习库了，scikit-learn简称sklearn，支持包括分类，回归，降维和聚类四大机器学习算法。还包括...
数据分析——算法——K-means聚类（天池：汽车产品聚类分析）
qq_42320048的博客
05-19 1万+
K-means聚类 目录 K-means聚类 1 简介 2 Python实战 1 简介 原理：通过计算不同样本间的距离来判断他们的相近关系的，相近的就会放到同一个类别中去。 适用数据：数值数据 优点：思想简单，容易实现，可解释度比较强 缺点：对噪音和异常点比较的敏感。k-means是在做凸优化，因此处理不了非凸的分布。如果两个类别距离比较近，k-means的效果也不会太好。初始中心点的选择以及k值的选择对结果影响较大，可能每次聚类结果都不一样。结果可能只是局部最优而不是全局最优 2 Py
org.apache.hadoop.fs.FSInputChecker - Problem opening checksum file
鸭梨的博客
01-15 333
WARN org.apache.hadoop.fs.FSInputChecker - Problem opening checksum file: /common/nlp/data/dictionary/custom/CustomDictionary.txt.bin. Ignoring exception: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.Dat
Hadoop put file 错误：fs.FSInputChecker: Found checksum error
FA的博客
03-30 371
Hadoop put file 错误：fs.FSInputChecker: Found checksum error 参考网址： https://www.cnblogs.com/gpcuster/archive/2011/01/26/1945363.html 错误原因： 启动任务的命令中包含一个参数“-files tb_steps_url_path_dim.txt” Hadoop客户端需要将机器本...
Kmeans方法Giraph实现V1.1.0
cloudeagle_bupt的专栏
06-24 300
package org.apache.giraph.benchmark.kmeans; import java.io.IOException; import java.util.regex.Pattern; import org.apache.giraph.graph.BasicComputation; import org.apache.giraph.graph.Vertex; import
Benchmark etcd性能测试工具
11-11
Benchmark etcd性能测试工具
benchmark-web-script-test:Alfresco Benchmark Framework 2.0自定义测试实现，调用Web脚本
05-09
Alfresco基准框架自定义测试实现此测试实现展示了如何在Alfresco Benchmark Framework测试中调用Web脚本。 文章： ：
benchmark:一个简单的Node.js基准测试库
03-22
@pizzafox/benchmark 一个简单的Node.js基准测试库。用法import { Benchmark , csvReporter } from '@pizzafox/benchmark' ;// Create a benchmarkconst benchmark = new Benchmark ( ) ;// Add different ...
benchmark.rar_Benchmark函数_benchmark
07-15
benchmark 函数,matlab编程
Benchmark 1.03
05-12
Tool measure phone performance for following categories: - 2D Graphic performance - CPU performance - Memory performance - File system performanc
GraphX简介及GraphFrames测试
ssyshenn的博客
07-21 7036
目录 概观 架构 存储 图的构造 GraphFrames 安装 测试 参考 概观 GraphX是Spark中用于图和图计算的组件，GraphX通过扩展Spark RDD引入了一个新的图抽象数据结构，一个将有效信息放入顶点和边的有向多重图。如同Spark的每一个模块一样，它们都有一个基于RDD的便于自己计算的抽象数据结构（如SQL的DataFrame，Str...
增强学习Reinforcement Learning经典算法梳理
jasmine_wxy的博客
08-25 294
https://blog.csdn.net/songrotek/article/details/51378582
Scikit-learn机器学习实战之Kmeans
沉心修炼
01-04 1万+
Scikit-learn中Kmeans实例
【Python实例第20讲】手写数字识别问题的K-Means聚类
wong2016的博客
11-28 6602
机器学习训练营——机器学习爱好者的自由交流空间（qq 群号：696721295） 在这个例子里，我们在手写数字识别数据集上，比较 K-means 聚类算法对于不同的初始化策略对运行时间和结果质量的影响。我们也利用不同的聚类质量测度判别聚类标签对于参考标签的拟合优度。这里使用的聚类评价测度有： homo (homogeneity score) compl (completeness sc...
HIVE自定义函数之UDF，UDAF和UDTF
dreamingfish2011的博客
04-29 1万+
UDF hive的udf允许用户使用自定义函数解决hive 自定义函数无法处理的逻辑。输入一行数据输出一行数据。 解决问题描述 想要比较两个逗号分隔的字符串是否相同。 -使用方法 如果ignoreNullFlag是1，则两个字符串都是空算相等，如果不是1，算不等 add jar /home/mart_wzyf/zhuhongmei/plist_udf_udaf.jar; CREATE TE
大数据领域的Benchmark介绍
热门推荐
行者小朱的博客
02-24 2万+
一、Benchmark简介 Benchmark是一个评价方式，在整个计算机领域有着长期的应用。正如维基百科上的解释“As computer architecture advanced, it became more difficult to compare the performance of various computer systems simply by looking at their
对Giraph的一些理解
hnhbwlp的专栏
04-27 3781
这两天又重新看了一下Giraph源码，对整体架构的理解又有了新的认识和理解，下面逐点来说。 一、Giraph本质的理解: 大家都知道，Giraph对用户来讲可能是一个基于Pregel模型的图运算项目，但是对于Hadoop来讲，其实它是一个普通的MapReduce任务。因此我们在运行时可以把他看成是一个mapreduce任务，只是这个任务有点特殊和复杂。 特殊在，它没有像普通mapreduce
Giraph Pagerank运行 SimplePagerank和PageRankBenchmark及自定义输入格式
cloudeagle_bupt的专栏
03-16 1348
测试数据: [0,0,[[1,1],[3,3]]] [1,0,[[0,1],[2,2],[3,1]]] [2,0,[[1,2],[4,4]]] [3,0,[[0,3],[1,1],[4,4]]] [4,0,[[3,4],[2,4]]] [liuqiang2@mu02 bin]$ giraph ../giraph-examples-1.1.0.jar   org.apache
Giraph源码分析（六）——Edge 分析
Hama White 的博客
04-21 2765
在Vertex类中，顶点的存储方式采用邻接表形式。每个顶点有 VertexId、VertexValue、OutgoingEdges和Halt，boolean型的halt变量用于记录顶点的状态，false时表示active，true表示inactive状态。
在Ubuntu20.04如何使用benchmarksql压测，五分钟一次
最新发布
06-09
默认情况下，BenchmarkSQL将执行一次测试，然后退出。如果你希望每5分钟运行一次测试，可以使用以下shell脚本： ``` #!/bin/bash while true do ./runBenchmark.sh sleep 300 done ``` 保存脚本，并使用以下命令使...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值