Giraph Pagerank运行 SimplePagerank和PageRankBenchmark及自定义输入格式

最新推荐文章于 2020-09-12 15:24:50 发布
最新推荐文章于 2020-09-12 15:24:50 发布
阅读量1.3k 收藏
点赞数 1
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/cloudeagle_bupt/article/details/50905165
版权 
测试数据:
[0,0,[[1,1],[3,3]]]
[1,0,[[0,1],[2,2],[3,1]]]
[2,0,[[1,2],[4,4]]]
[3,0,[[0,3],[1,1],[4,4]]]
[4,0,[[3,4],[2,4]]]

注意格式为 

JsonLongDoubleFloatDoubleVertexInputFormat

[source_id,source_value,[[dest_id, edge_value],...]]. 


执行命令:

[liuqiang2@mu02 bin]$ giraph ../giraph-examples-1.1.0.jar   org.apache.giraph.examples.SimplePageRankComputation  -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /test/test.txt  -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /output1 -w 1 -mc org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute


结果:


HADOOP_CONF_DIR=/opt/hadoop-1.2.1/conf
16/03/16 15:29:39 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
16/03/16 15:29:39 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
16/03/16 15:29:39 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4)
16/03/16 15:29:44 INFO job.GiraphJob: Tracking URL: http://mu02:50030/jobdetails.jsp?jobid=job_201603151735_0006
16/03/16 15:29:44 INFO job.GiraphJob: Waiting for resources... Job will start only when it gets all 2 mappers
16/03/16 15:30:13 INFO job.HaltApplicationUtils$DefaultHaltInstructionsWriter: writeHaltInstructions: To halt after next superstep execute: 'bin/halt-application --zkServer mu02:22181 --zkNode /_hadoopBsp/job_201603151735_0006/_haltComputation'
16/03/16 15:30:13 INFO mapred.JobClient: Running job: job_201603151735_0006
16/03/16 15:30:14 INFO mapred.JobClient:  map 50% reduce 0%
16/03/16 15:30:18 INFO mapred.JobClient:  map 100% reduce 0%
16/03/16 15:30:27 INFO mapred.JobClient: Job complete: job_201603151735_0006
16/03/16 15:30:27 INFO mapred.JobClient: Counters: 68
16/03/16 15:30:27 INFO mapred.JobClient:   Zookeeper halt node
16/03/16 15:30:27 INFO mapred.JobClient:     /_hadoopBsp/job_201603151735_0006/_haltComputation=0
16/03/16 15:30:27 INFO mapred.JobClient:   Zookeeper base path
16/03/16 15:30:27 INFO mapred.JobClient:     /_hadoopBsp/job_201603151735_0006=0
16/03/16 15:30:27 INFO mapred.JobClient:   Job Counters 
16/03/16 15:30:27 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=46775
16/03/16 15:30:27 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
16/03/16 15:30:27 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
16/03/16 15:30:27 INFO mapred.JobClient:     Launched map tasks=2
16/03/16 15:30:27 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
16/03/16 15:30:27 INFO mapred.JobClient:   Giraph Timers
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 29 SimplePageRankComputation (ms)=49
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 12 SimplePageRankComputation (ms)=58
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 16 SimplePageRankComputation (ms)=54
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 21 SimplePageRankComputation (ms)=38
16/03/16 15:30:27 INFO mapred.JobClient:     Shutdown (ms)=8942
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 0 SimplePageRankComputation (ms)=212
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 15 SimplePageRankComputation (ms)=43
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 13 SimplePageRankComputation (ms)=68
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 27 SimplePageRankComputation (ms)=52
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 17 SimplePageRankComputation (ms)=58
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 26 SimplePageRankComputation (ms)=58
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 23 SimplePageRankComputation (ms)=59
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 5 SimplePageRankComputation (ms)=66
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 18 SimplePageRankComputation (ms)=48
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 22 SimplePageRankComputation (ms)=42
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 10 SimplePageRankComputation (ms)=64
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 8 SimplePageRankComputation (ms)=60
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 3 SimplePageRankComputation (ms)=77
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 19 SimplePageRankComputation (ms)=40
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 4 SimplePageRankComputation (ms)=70
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 9 SimplePageRankComputation (ms)=61
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 30 SimplePageRankComputation (ms)=49
16/03/16 15:30:27 INFO mapred.JobClient:     Input superstep (ms)=366
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 25 SimplePageRankComputation (ms)=56
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 28 SimplePageRankComputation (ms)=42
16/03/16 15:30:27 INFO mapred.JobClient:     Initialize (ms)=3163
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 7 SimplePageRankComputation (ms)=51
16/03/16 15:30:27 INFO mapred.JobClient:     Setup (ms)=97
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 14 SimplePageRankComputation (ms)=58
16/03/16 15:30:27 INFO mapred.JobClient:     Total (ms)=11728
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 20 SimplePageRankComputation (ms)=40
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 24 SimplePageRankComputation (ms)=62
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 11 SimplePageRankComputation (ms)=65
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 2 SimplePageRankComputation (ms)=447
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 1 SimplePageRankComputation (ms)=97
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep 6 SimplePageRankComputation (ms)=67
16/03/16 15:30:27 INFO mapred.JobClient:   Zookeeper server:port
16/03/16 15:30:27 INFO mapred.JobClient:     mu02:22181=0
16/03/16 15:30:27 INFO mapred.JobClient:   Giraph Stats
16/03/16 15:30:27 INFO mapred.JobClient:     Aggregate edges=12
16/03/16 15:30:27 INFO mapred.JobClient:     Sent message bytes=0
16/03/16 15:30:27 INFO mapred.JobClient:     Superstep=31
16/03/16 15:30:27 INFO mapred.JobClient:     Last checkpointed superstep=0
16/03/16 15:30:27 INFO mapred.JobClient:     Current workers=1
16/03/16 15:30:27 INFO mapred.JobClient:     Aggregate sent messages=360
16/03/16 15:30:27 INFO mapred.JobClient:     Current master task partition=0
16/03/16 15:30:27 INFO mapred.JobClient:     Sent messages=0
16/03/16 15:30:27 INFO mapred.JobClient:     Aggregate finished vertices=5
16/03/16 15:30:27 INFO mapred.JobClient:     Aggregate sent message message bytes=6510
16/03/16 15:30:27 INFO mapred.JobClient:     Aggregate vertices=5
16/03/16 15:30:27 INFO mapred.JobClient:   File Output Format Counters 
16/03/16 15:30:27 INFO mapred.JobClient:     Bytes Written=0
16/03/16 15:30:27 INFO mapred.JobClient:   FileSystemCounters
16/03/16 15:30:27 INFO mapred.JobClient:     HDFS_BYTES_READ=200
16/03/16 15:30:27 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=268928
16/03/16 15:30:27 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=109
16/03/16 15:30:27 INFO mapred.JobClient:   File Input Format Counters 
16/03/16 15:30:27 INFO mapred.JobClient:     Bytes Read=0
16/03/16 15:30:27 INFO mapred.JobClient:   Map-Reduce Framework
16/03/16 15:30:27 INFO mapred.JobClient:     Map input records=2
16/03/16 15:30:27 INFO mapred.JobClient:     Physical memory (bytes) snapshot=793718784
16/03/16 15:30:27 INFO mapred.JobClient:     Spilled Records=0
16/03/16 15:30:27 INFO mapred.JobClient:     CPU time spent (ms)=13120
16/03/16 15:30:27 INFO mapred.JobClient:     Total committed heap usage (bytes)=1339031552
16/03/16 15:30:27 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=3435483136
16/03/16 15:30:27 INFO mapred.JobClient:     Map output records=0
16/03/16 15:30:27 INFO mapred.JobClient:     SPLIT_RAW_BYTES=88


注意参数最后的: -mc org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute

PageRankBenchmark版本

giraph ../giraph-core-1.1.0.jar  org.apache.giraph.benchmark.PageRankComputationNew -vif  org.apache.giraph.io.formats.LongDoubleDoubleAdjacencyListVertexInputFormat -vip /test/data.txt -op /output1  -w 1

16/03/16 21:59:38 INFO mapred.JobClient: Running job: job_201603162124_0003
16/03/16 21:59:39 INFO mapred.JobClient:  map 100% reduce 0%
16/03/16 21:59:44 INFO mapred.JobClient: Job complete: job_201603162124_0003
16/03/16 21:59:44 INFO mapred.JobClient: Counters: 37
16/03/16 21:59:44 INFO mapred.JobClient:   Zookeeper halt node
16/03/16 21:59:44 INFO mapred.JobClient:     /_hadoopBsp/job_201603162124_0003/_haltComputation=0
16/03/16 21:59:44 INFO mapred.JobClient:   Zookeeper base path
16/03/16 21:59:44 INFO mapred.JobClient:     /_hadoopBsp/job_201603162124_0003=0
16/03/16 21:59:44 INFO mapred.JobClient:   Job Counters 
16/03/16 21:59:44 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=34879
16/03/16 21:59:44 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
16/03/16 21:59:44 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
16/03/16 21:59:44 INFO mapred.JobClient:     Launched map tasks=2
16/03/16 21:59:44 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
16/03/16 21:59:44 INFO mapred.JobClient:   Giraph Timers
16/03/16 21:59:44 INFO mapred.JobClient:     Input superstep (ms)=319
16/03/16 21:59:44 INFO mapred.JobClient:     Total (ms)=9377
16/03/16 21:59:44 INFO mapred.JobClient:     Shutdown (ms)=8887
16/03/16 21:59:44 INFO mapred.JobClient:     Initialize (ms)=1054
16/03/16 21:59:44 INFO mapred.JobClient:     Superstep 0 PageRankComputationNew (ms)=94
16/03/16 21:59:44 INFO mapred.JobClient:     Setup (ms)=76
16/03/16 21:59:44 INFO mapred.JobClient:   Zookeeper server:port
16/03/16 21:59:44 INFO mapred.JobClient:     mu02:22181=0
16/03/16 21:59:44 INFO mapred.JobClient:   Giraph Stats
16/03/16 21:59:44 INFO mapred.JobClient:     Aggregate edges=2
16/03/16 21:59:44 INFO mapred.JobClient:     Sent message bytes=0
16/03/16 21:59:44 INFO mapred.JobClient:     Superstep=1
16/03/16 21:59:44 INFO mapred.JobClient:     Last checkpointed superstep=0
16/03/16 21:59:44 INFO mapred.JobClient:     Current workers=1
16/03/16 21:59:44 INFO mapred.JobClient:     Aggregate sent messages=0
16/03/16 21:59:44 INFO mapred.JobClient:     Current master task partition=0
16/03/16 21:59:44 INFO mapred.JobClient:     Sent messages=0
16/03/16 21:59:44 INFO mapred.JobClient:     Aggregate finished vertices=1
16/03/16 21:59:44 INFO mapred.JobClient:     Aggregate sent message message bytes=0
16/03/16 21:59:44 INFO mapred.JobClient:     Aggregate vertices=1
16/03/16 21:59:44 INFO mapred.JobClient:   File Output Format Counters 
16/03/16 21:59:44 INFO mapred.JobClient:     Bytes Written=0
16/03/16 21:59:44 INFO mapred.JobClient:   FileSystemCounters
16/03/16 21:59:44 INFO mapred.JobClient:     HDFS_BYTES_READ=110
16/03/16 21:59:44 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=267478
16/03/16 21:59:44 INFO mapred.JobClient:   File Input Format Counters 
16/03/16 21:59:44 INFO mapred.JobClient:     Bytes Read=0
16/03/16 21:59:44 INFO mapred.JobClient:   Map-Reduce Framework
16/03/16 21:59:44 INFO mapred.JobClient:     Map input records=2
16/03/16 21:59:44 INFO mapred.JobClient:     Physical memory (bytes) snapshot=741826560
16/03/16 21:59:44 INFO mapred.JobClient:     Spilled Records=0
16/03/16 21:59:44 INFO mapred.JobClient:     CPU time spent (ms)=8280
16/03/16 21:59:44 INFO mapred.JobClient:     Total committed heap usage (bytes)=1337982976
16/03/16 21:59:44 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=3484495872
16/03/16 21:59:44 INFO mapred.JobClient:     Map output records=0
16/03/16 21:59:44 INFO mapred.JobClient:     SPLIT_RAW_BYTES=88



读代码时发现有些数据不太合适目前的Benchmark中PagerankComputation的设置，即顶点格式:

public class PageRankComputation extends BasicComputation<IntWritable,
    FloatWritable, NullWritable, FloatWritable>

但是输入格式中未发现:  
public class IntFloatNullTextInputFormat extends
    TextVertexInputFormat<IntWritable, FloatWritable, NullWritable>

对于此类数据:

Each line consists of: vertex neighbor1 neighbor2 ..   

不太合适，因此进行了修改。

增加IntFloatNullTextInputFormat 类, 

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.giraph.io.formats;

import org.apache.giraph.edge.Edge;
import org.apache.giraph.edge.EdgeFactory;
import org.apache.hadoop.io.FloatWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.TaskAttemptContext;

import com.google.common.collect.Lists;

import java.io.IOException;
import java.util.List;
import java.util.regex.Pattern;

/**
 * Simple text-based {@link org.apache.giraph.io.VertexInputFormat} for
 * unweighted graphs with int ids.
 *
 * Each line consists of: vertex neighbor1 neighbor2 ...
 */
public class IntFloatNullTextInputFormat extends
    TextVertexInputFormat<IntWritable, FloatWritable, NullWritable> {
  /** Separator of the vertex and neighbors */
  private static final Pattern SEPARATOR = Pattern.compile("[\t ]");

  @Override
  public TextVertexReader createVertexReader(InputSplit split,
      TaskAttemptContext context)
    throws IOException {
    return new IntFloatNullVertexReader();
  }

  /**
   * Vertex reader associated with {@link IntIntNullTextInputFormat}.
   */
  public class IntFloatNullVertexReader extends
    TextVertexReaderFromEachLineProcessed<String[]> {
    /**
     * Cached vertex id for the current line
     */
    private IntWritable id;

    @Override
    protected String[] preprocessLine(Text line) throws IOException {
      String[] tokens = SEPARATOR.split(line.toString());
      id = new IntWritable(Integer.parseInt(tokens[0]));
      return tokens;
    }

    @Override
    protected IntWritable getId(String[] tokens) throws IOException {
      return id;
    }

    @Override
    protected FloatWritable getValue(String[] tokens) throws IOException {
      return new FloatWritable(new Float(id.get()));
    }

    @Override
    protected Iterable<Edge<IntWritable, NullWritable>> getEdges(
        String[] tokens) throws IOException {
      List<Edge<IntWritable, NullWritable>> edges =
          Lists.newArrayListWithCapacity(tokens.length - 1);
      for (int n = 1; n < tokens.length; n++) {
        edges.add(EdgeFactory.create(
            new IntWritable(Integer.parseInt(tokens[n]))));
      }
      return edges;
    }
  }
}

执行:


[liuqiang2@mu02 bin]$  giraph ../giraph-core-1.1.0.jar  org.apache.giraph.benchmark.PageRankComputation -vif  org.apache.giraph.io.formats.IntFloatNullTextInputFormat -vip /test/noWeight.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /output  -w 1
HADOOP_CONF_DIR=/opt/hadoop-1.2.1/conf
16/03/18 16:58:05 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
16/03/18 16:58:05 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
16/03/18 16:58:06 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4)
16/03/18 16:58:11 INFO job.GiraphJob: Tracking URL: http://mu02:50030/jobdetails.jsp?jobid=job_201603181605_0005
16/03/18 16:58:11 INFO job.GiraphJob: Waiting for resources... Job will start only when it gets all 2 mappers
16/03/18 16:58:46 INFO job.HaltApplicationUtils$DefaultHaltInstructionsWriter: writeHaltInstructions: To halt after next superstep execute: 'bin/halt-application --zkServer mu02:22181 --zkNode /_hadoopBsp/job_201603181605_0005/_haltComputation'
16/03/18 16:58:46 INFO mapred.JobClient: Running job: job_201603181605_0005
16/03/18 16:58:47 INFO mapred.JobClient:  map 100% reduce 0%
16/03/18 16:58:51 INFO mapred.JobClient: Job complete: job_201603181605_0005
16/03/18 16:58:51 INFO mapred.JobClient: Counters: 68
16/03/18 16:58:51 INFO mapred.JobClient:   Zookeeper halt node
16/03/18 16:58:51 INFO mapred.JobClient:     /_hadoopBsp/job_201603181605_0005/_haltComputation=0
16/03/18 16:58:51 INFO mapred.JobClient:   Zookeeper base path
16/03/18 16:58:51 INFO mapred.JobClient:     /_hadoopBsp/job_201603181605_0005=0
16/03/18 16:58:51 INFO mapred.JobClient:   Job Counters 
16/03/18 16:58:51 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=40535
16/03/18 16:58:51 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
16/03/18 16:58:51 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
16/03/18 16:58:51 INFO mapred.JobClient:     Launched map tasks=2
16/03/18 16:58:51 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
16/03/18 16:58:51 INFO mapred.JobClient:   Giraph Timers
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 27 PageRankComputation (ms)=52
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 3 PageRankComputation (ms)=66
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 1 PageRankComputation (ms)=98
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 11 PageRankComputation (ms)=62
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 23 PageRankComputation (ms)=41
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 20 PageRankComputation (ms)=59
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 8 PageRankComputation (ms)=62
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 13 PageRankComputation (ms)=62
16/03/18 16:58:51 INFO mapred.JobClient:     Shutdown (ms)=8972
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 2 PageRankComputation (ms)=56
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 25 PageRankComputation (ms)=60
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 29 PageRankComputation (ms)=51
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 12 PageRankComputation (ms)=63
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 15 PageRankComputation (ms)=40
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 26 PageRankComputation (ms)=58
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 18 PageRankComputation (ms)=58
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 9 PageRankComputation (ms)=48
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 10 PageRankComputation (ms)=66
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 14 PageRankComputation (ms)=65
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 0 PageRankComputation (ms)=102
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 24 PageRankComputation (ms)=52
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 16 PageRankComputation (ms)=36
16/03/18 16:58:51 INFO mapred.JobClient:     Input superstep (ms)=344
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 4 PageRankComputation (ms)=68
16/03/18 16:58:51 INFO mapred.JobClient:     Initialize (ms)=612
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 21 PageRankComputation (ms)=56
16/03/18 16:58:51 INFO mapred.JobClient:     Setup (ms)=68
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 5 PageRankComputation (ms)=60
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 30 PageRankComputation (ms)=51
16/03/18 16:58:51 INFO mapred.JobClient:     Total (ms)=11217
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 7 PageRankComputation (ms)=63
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 28 PageRankComputation (ms)=59
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 17 PageRankComputation (ms)=41
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 22 PageRankComputation (ms)=45
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 19 PageRankComputation (ms)=56
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep 6 PageRankComputation (ms)=66
16/03/18 16:58:51 INFO mapred.JobClient:   Zookeeper server:port
16/03/18 16:58:51 INFO mapred.JobClient:     mu02:22181=0
16/03/18 16:58:51 INFO mapred.JobClient:   Giraph Stats
16/03/18 16:58:51 INFO mapred.JobClient:     Aggregate edges=5
16/03/18 16:58:51 INFO mapred.JobClient:     Sent message bytes=0
16/03/18 16:58:51 INFO mapred.JobClient:     Superstep=31
16/03/18 16:58:51 INFO mapred.JobClient:     Last checkpointed superstep=0
16/03/18 16:58:51 INFO mapred.JobClient:     Current workers=1
16/03/18 16:58:51 INFO mapred.JobClient:     Aggregate sent messages=150
16/03/18 16:58:51 INFO mapred.JobClient:     Current master task partition=0
16/03/18 16:58:51 INFO mapred.JobClient:     Sent messages=0
16/03/18 16:58:51 INFO mapred.JobClient:     Aggregate finished vertices=4
16/03/18 16:58:51 INFO mapred.JobClient:     Aggregate sent message message bytes=1950
16/03/18 16:58:51 INFO mapred.JobClient:     Aggregate vertices=4
16/03/18 16:58:51 INFO mapred.JobClient:   File Output Format Counters 
16/03/18 16:58:51 INFO mapred.JobClient:     Bytes Written=0
16/03/18 16:58:51 INFO mapred.JobClient:   FileSystemCounters
16/03/18 16:58:51 INFO mapred.JobClient:     HDFS_BYTES_READ=106
16/03/18 16:58:51 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=262866
16/03/18 16:58:51 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=52
16/03/18 16:58:51 INFO mapred.JobClient:   File Input Format Counters 
16/03/18 16:58:51 INFO mapred.JobClient:     Bytes Read=0
16/03/18 16:58:51 INFO mapred.JobClient:   Map-Reduce Framework
16/03/18 16:58:51 INFO mapred.JobClient:     Map input records=2
16/03/18 16:58:51 INFO mapred.JobClient:     Physical memory (bytes) snapshot=299720704
16/03/18 16:58:51 INFO mapred.JobClient:     Spilled Records=0
16/03/18 16:58:51 INFO mapred.JobClient:     CPU time spent (ms)=12020
16/03/18 16:58:51 INFO mapred.JobClient:     Total committed heap usage (bytes)=406323200
16/03/18 16:58:51 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=1661681664
16/03/18 16:58:51 INFO mapred.JobClient:     Map output records=0
16/03/18 16:58:51 INFO mapred.JobClient:     SPLIT_RAW_BYTES=88




飞火流云
关注
  • 1
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Giraph原理与代码实例讲解
禅与计算机程序设计艺术
06-08 382
Giraph是一个基于Hadoop的分布式图计算框架，它可以处理大规模的图数据，例如社交网络、路网、生物网络等。Giraph的设计目标是提供一个高效、可扩展、易于使用的图计算框架，使得用户可以方便地进行图计算任务的开发和部署。Giraph最初由Yahoo!开发，后来成为Apache基金会的一个开源项目。目前，Giraph已经成为了大规模图计算领域的一个重要工具，被广泛应用于社交网络分析、推荐系统、网络安全等领域。
giraph:不可变的图数据结构
05-29
长颈鹿 不可变的图数据结构 安装 npm install -S giraph bower install -S giraph 用法 require ( 'giraph' ) ( ) . add ( 'a' , { my : 'data' } ) . add ( 'b' ) . add ( 'c' ) . connect ( 'a' , 'b' , 3 ) . connect ( 'a' , 'c' ) 文档 Giraph 带有许多不同的图形类型： 无向图 在根名称空间上作为工厂函数导出： .([options]) 选项和默认值 { immutable : true } require ( 'giraph' ) ( ) // => empty graph 无向图具有以下成员 属性： .map: {} 按ID散列顶点。 属性： .options
Giraph集群测试Pagerank
cloudeagle_bupt的专栏
03-18 1043
[liuqiang2@mu02 bin]$ giraph ../giraph-core-1.1.0.jar org.apache.giraph.benchmark.PageRankComputation -vif org.apache.giraph.io.formats.IntFloatNullTextInputFormat -vip /test/youTube.txt -vof org.
Giraph 简介
ikroal
08-12 4977
前言 本文主要阐述了 Giraph 由来及其作用，并根据 Giraph 的系统架构和计算模型简要介绍了 Giraph 的运行流程。 什么是 Giraph Giraph 是 Google 于 2010 年发布的论文 Pregel: a system for large-scale graph processing 的开源实现。Giraph 是以 Hadoop 为基础开发的上层应用，其系统架构和计算模...
Giraph的一些背景介绍
wjlwangluo的博客
02-20 1739
关于Giraph的一些背景介绍
从数据挖掘经典算法PageRank入门
科技创新与颠覆者
04-21 6296
从数据挖掘经典算法PageRank入门Graphene PageRank是Google研发的主要应用于评估网站可靠度和重要性的一种算法，是进行网页排名的考量指标之一。本文将对PageRank的原理进行讲解，并以此为出发点介绍如何利用Transwarp Data Hub的Graphene在实际中满足相关分析需求。 在开始PageRank原理介绍之前，请读者先阅读下面这则案例：
PageRank算法简介及Map-Reduce实现
02-26
PageRank对网页排名的算法，曾是Google发家致富的法宝。以前虽然有实验过，但理解还是不透彻，这几天又看了一下，这里总结一下PageRank算法的基本原理。PageRank的Page可是认为是网页，表示网页排名，也可以认为是...
链接分析--PageRank算法分析实现及优化.zip
06-10
朴素的 PageRank 算法在 Web 网页结构良好的环境下可以正常运行，通过迭代可以对不同的网页给出合理的打分。然而，研究表明，现实中的 Web 网页结构常常出现网页个体或网页群体没有出向链接，即网络中的 dead ends ...
Python基于随机游走模型的PageRank算法及应用.zip
最新发布
10-24
资源包含文件：课程论文word+源码 PageRank 问题，就是一种基于图的随机游走问题。在搜索引擎最初出现的时候，多采用的是分类目录的方法，详细介绍参考：https://blog.csdn.net/newlw/article/details/126875536
搜索引擎PageRank算法实现及测试数据
06-09
2. **计算新PageRank**：对于每个网页，根据其入链网页的PageRank值和阻尼因子进行计算。如果一个网页没有出链，它的PageRank会被平均分配给整个网络。 3. **迭代更新**：重复步骤2，直到PageRank值的变化低于某个...
Giraph Pagerank运行 SimplePagerank和PageRankBenchmark
cloudeagle_bupt的专栏
03-16 271
测试数据: [0,0,[[1,1],[3,3]]] [1,0,[[0,1],[2,2],[3,1]]] [2,0,[[1,2],[4,4]]] [3,0,[[0,3],[1,1],[4,4]]] [4,0,[[3,4],[2,4]]] 注意格式为 JsonLongDoubleFloatDoubleVertexInputFormat [source_id,source_value,[[d...
对Giraph的一些理解
hnhbwlp的专栏
04-27 3792
这两天又重新看了一下Giraph源码，对整体架构的理解又有了新的认识和理解，下面逐点来说。 一、Giraph本质的理解: 大家都知道，Giraph对用户来讲可能是一个基于Pregel模型的图运算项目，但是对于Hadoop来讲，其实它是一个普通的MapReduce任务。因此我们在运行时可以把他看成是一个mapreduce任务，只是这个任务有点特殊和复杂。 特殊在，它没有像普通mapreduce
Label Propagation算法Giraph实现
cloudeagle_bupt的专栏
08-18 782
package org.apache.giraph.examples.lp; import java.io.IOException; import java.util.ArrayList; import java.util.Collections; import java.util.List; import org.apache.giraph.comm.WorkerClientRequestP
giraph编程
qq_21877407的博客
07-09 361
Giraph的三个接口
TulieCS的博客
09-12 153
Giraph中的图仅使用Edge接口和Vertex接口构造。Edge包含目标顶点的值/权重和 id。在 Giraph 的图模型中，一个顶点包含一个 ID、一个值和它所有外出的边的列表。Giraph 提供了这些接口的默认实现：DefaultVertex 和 DefaultEdge。这些接口还有其他实现 一、Edge接口 Edge 界面的类型变量如图 3.1 所示。边包含一个目标顶点 ID，它是类型变量 I ，边数据是类型变量 E 。类型 I 和 E 取决于图形类型。为加载图数据而选择/实现的 InputFor
PageRank算法详解与应用
"PageRank算法讲解" PageRank算法是Google搜索引擎早期的核心技术之一，由Google的创始人之一拉里·佩奇（Larry Page）在1998年提出。这个算法主要利用互联网上的超链接结构来评估网页的重要性，从而为搜索引擎提供...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值