hadoop实例sort

参考文献：http://www.hadooper.cn/dct/page/65777

1排序实例

排序实例仅仅用 map/reduce框架来把输入目录排序放到输出目录。 输入和输出必须是顺序文件，键和值是BytesWritable.
mapper是预先定义的IdentityMapper，reducer 是预先定义的 IdentityReducer， 两个都是把输入直接的输出。
要运行这个例 子：bin/hadoop jar hadoop-*-examples.jar sort [-m <#maps>] [-r <#reduces>] <in-dir> <out-dir>

2运行排序基准测试

为了使得排序例子作为一个 基准测试，用 RandomWriter产 生10GB/node 的数据。然后用排序实例来进行排序。这个提供了一个可扩展性依赖于集群的大小的排序基准。默认情况下，排序实例用1.0*capacity作为 reduces的数量，依赖于你的集群的大小你可能会在1.75*capacity的情况下得到更好的结果。

To use the sort example as a benchmark, generate 10GB/node of random data using RandomWriter. Then sort the data using the sort example. This provides a sort benchmark that scales depending on the size of the cluster. By default, the sort example uses 1.0 * capacity for the number of reduces and depending on your cluster you may see better results at 1.75 * capacity.

命令是：

% bin/hadoop jar hadoop-*-examples.jar randomwriter rand
% bin/hadoop jar hadoop-*-examples.jar sort rand rand-sort

第一个命令会在rand 目录的生成没有排序的数据。第二个命令会读数据，排序，然后写入rand-sort 目录

排序支持一般的选项：参见DevelopmentCommandLineOptions

3具体实验

3.1代码实例Sort.java

/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.hadoop.examples;

import java.io.IOException;
import java.net.URI;
import java.util.*;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.lib.IdentityMapper;
import org.apache.hadoop.mapred.lib.IdentityReducer;
import org.apache.hadoop.mapred.lib.InputSampler;
import org.apache.hadoop.mapred.lib.TotalOrderPartitioner;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

/**
 * This is the trivial map/reduce program that does absolutely nothing
 * other than use the framework to fragment and sort the input values.
 *
 * To run: bin/hadoop jar build/hadoop-examples.jar sort
 *            [-m <i>maps</i>] [-r <i>reduces</i>]
 *            [-inFormat <i>input format class</i>] 
 *            [-outFormat <i>output format class</i>] 
 *            [-outKey <i>output key class</i>] 
 *            [-outValue <i>output value class</i>] 
 *            [-totalOrder <i>pcnt</i> <i>num samples</i> <i>max splits</i>]
 *            <i>in-dir</i> <i>out-dir</i> 
 */
public class Sort<K,V> extends Configured implements Tool {
  private RunningJob jobResult = null;

  static int printUsage() {
    System.out.println("sort [-m <maps>] [-r <reduces>] " +
                       "[-inFormat <input format class>] " +
                       "[-outFormat <output format class>] " + 
                       "[-outKey <output key class>] " +
                       "[-outValue <output value class>] " +
                       "[-totalOrder <pcnt> <num samples> <max splits>] " +
                       "<input> <output>");
    ToolRunner.printGenericCommandUsage(System.out);
    return -1;
  }

  /**
   * The main driver for sort program.
   * Invoke this method to submit the map/reduce job.
   * @throws IOException When there is communication problems with the 
   *                     job tracker.
   */
  public int run(String[] args) throws Exception {

    JobConf jobConf = new JobConf(getConf(), Sort.class);
    jobConf.setJobName("sorter");

    jobConf.setMapperClass(IdentityMapper.class);        
    jobConf.setReducerClass(IdentityReducer.class);

    JobClient client = new JobClient(jobConf);
    ClusterStatus cluster = client.getClusterStatus();
    int num_reduces = (int) (cluster.getMaxReduceTasks() * 0.9);
    String sort_reduces = jobConf.get("test.sort.reduces_per_host");
    if (sort_reduces != null) {
       num_reduces = cluster.getTaskTrackers() * 
                       Integer.parseInt(sort_reduces);
    }
    Class<? extends InputFormat> inputFormatClass = 
      SequenceFileInputFormat.class;
    Class<? extends OutputFormat> outputFormatClass = 
      SequenceFileOutputFormat.class;
    Class<? extends WritableComparable> outputKeyClass = BytesWritable.class;
    Class<? extends Writable> outputValueClass = BytesWritable.class;
    List<String> otherArgs = new ArrayList<String>();
    InputSampler.Sampler<K,V> sampler = null;
    for(int i=0; i < args.length; ++i) {
      try {
        if ("-m".equals(args[i])) {
          jobConf.setNumMapTasks(Integer.parseInt(args[++i]));
        } else if ("-r".equals(args[i])) {
          num_reduces = Integer.parseInt(args[++i]);
        } else if ("-inFormat".equals(args[i])) {
          inputFormatClass = 
            Class.forName(args[++i]).asSubclass(InputFormat.class);
        } else if ("-outFormat".equals(args[i])) {
          outputFormatClass = 
            Class.forName(args[++i]).asSubclass(OutputFormat.class);
        } else if ("-outKey".equals(args[i])) {
          outputKeyClass = 
            Class.forName(args[++i]).asSubclass(WritableComparable.class);
        } else if ("-outValue".equals(args[i])) {
          outputValueClass = 
            Class.forName(args[++i]).asSubclass(Writable.class);
        } else if ("-totalOrder".equals(args[i])) {
          double pcnt = Double.parseDouble(args[++i]);
          int numSamples = Integer.parseInt(args[++i]);
          int maxSplits = Integer.parseInt(args[++i]);
          if (0 >= maxSplits) maxSplits = Integer.MAX_VALUE;
          sampler =
            new InputSampler.RandomSampler<K,V>(pcnt, numSamples, maxSplits);
        } else {
          otherArgs.add(args[i]);
        }
      } catch (NumberFormatException except) {
        System.out.println("ERROR: Integer expected instead of " + args[i]);
        return printUsage();
      } catch (ArrayIndexOutOfBoundsException except) {
        System.out.println("ERROR: Required parameter missing from " +
            args[i-1]);
        return printUsage(); // exits
      }
    }

    // Set user-supplied (possibly default) job configs
    jobConf.setNumReduceTasks(num_reduces);

    jobConf.setInputFormat(inputFormatClass);
    jobConf.setOutputFormat(outputFormatClass);

    jobConf.setOutputKeyClass(outputKeyClass);
    jobConf.setOutputValueClass(outputValueClass);

    // Make sure there are exactly 2 parameters left.
    if (otherArgs.size() != 2) {
      System.out.println("ERROR: Wrong number of parameters: " +
          otherArgs.size() + " instead of 2.");
      return printUsage();
    }
    FileInputFormat.setInputPaths(jobConf, otherArgs.get(0));
    FileOutputFormat.setOutputPath(jobConf, new Path(otherArgs.get(1)));

    if (sampler != null) {
      System.out.println("Sampling input to effect total-order sort...");
      jobConf.setPartitionerClass(TotalOrderPartitioner.class);
      Path inputDir = FileInputFormat.getInputPaths(jobConf)[0];
      inputDir = inputDir.makeQualified(inputDir.getFileSystem(jobConf));
      Path partitionFile = new Path(inputDir, "_sortPartitioning");
      TotalOrderPartitioner.setPartitionFile(jobConf, partitionFile);
      InputSampler.<K,V>writePartitionFile(jobConf, sampler);
      URI partitionUri = new URI(partitionFile.toString() +
                                 "#" + "_sortPartitioning");
      DistributedCache.addCacheFile(partitionUri, jobConf);
      DistributedCache.createSymlink(jobConf);
    }

    System.out.println("Running on " +
        cluster.getTaskTrackers() +
        " nodes to sort from " + 
        FileInputFormat.getInputPaths(jobConf)[0] + " into " +
        FileOutputFormat.getOutputPath(jobConf) +
        " with " + num_reduces + " reduces.");
    Date startTime = new Date();
    System.out.println("Job started: " + startTime);
    jobResult = JobClient.runJob(jobConf);
    Date end_time = new Date();
    System.out.println("Job ended: " + end_time);
    System.out.println("The job took " + 
        (end_time.getTime() - startTime.getTime()) /1000 + " seconds.");
    return 0;
  }

//input attr:/home/hadoop/rand/part-00000 /home/hadoop/rand-sort

  public static void main(String[] args) throws Exception {
    int res = ToolRunner.run(new Configuration(), new Sort(), args);
    System.exit(res);
  }

  /**
   * Get the last job that was run using this instance.
   * @return the results of the last job that was run
   */
  public RunningJob getResult() {
    return jobResult;
  }
}

3.2在eclipse中设置参数：

/home/hadoop/rand/part-00000 /home/hadoop/rand-sort

其中/home/hadoop/rand/part-00000 表示输入路径，/home/hadoop/rand-sort表示输出路径

3.3数据来源

我们这里输入参数中的“/home/hadoop/rand/part-00000”是通过hadoop实例 RandomWriter 这个实例得到的。为了节省时间，hadoop实例 RandomWriter 中得到了两个文件，我们这里指使用了一个文件part-00000。如果要对两个文件都进行排序操作，那么输入路径只需要是目录即可。

4总结

本程序目前我测试只能在单机上执行，不能在集群上运行。即指可以run as ->java application，而不能run on hadoop，具体原因还没有找到，如果发现后续会更新本博客。

PS:2011-10-18

运行环境

1. run as java application

console中会输出信息

Running on 1 nodes to sort from hdfs://master:9000/home/hadoop/rand/part-00000 into hdfs://master:9000/home/hadoop/rand-sort with 1 reduces.

2.一个master和一个slave，run on hadoop

console输出信息

Running on 1 nodes to sort from hdfs://master:9000/home/hadoop/rand/part-00000 into hdfs://master:9000/home/hadoop/rand-sort with 1 reduces.

跟第一中情况一样。

3.一台主机即做master又做slave，另外一台单独做slave

console输出信息

11/10/18 09:24:35 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Running on 2 nodes to sort from hdfs://master:9000/home/hadoop/rand/part-00000 into hdfs://master:9000/home/hadoop/rand-sort with 3 reduces.
Job started: Tue Oct 18 09:24:35 CST 2011
11/10/18 09:24:35 INFO mapred.FileInputFormat: Total input paths to process : 1
11/10/18 09:24:36 INFO mapred.JobClient: Running job: job_201110180923_0001
11/10/18 09:24:37 INFO mapred.JobClient:  map 0% reduce 0%
11/10/18 09:24:50 INFO mapred.JobClient:  map 6% reduce 0%
11/10/18 09:24:51 INFO mapred.JobClient:  map 18% reduce 0%
11/10/18 09:24:53 INFO mapred.JobClient:  map 25% reduce 0%
11/10/18 09:24:56 INFO mapred.JobClient:  map 31% reduce 0%
11/10/18 09:25:01 INFO mapred.JobClient:  map 43% reduce 0%
11/10/18 09:25:02 INFO mapred.JobClient:  map 49% reduce 0%
11/10/18 09:25:04 INFO mapred.JobClient:  map 50% reduce 2%
11/10/18 09:25:08 INFO mapred.JobClient:  map 56% reduce 4%
11/10/18 09:25:09 INFO mapred.JobClient:  map 62% reduce 6%
11/10/18 09:25:11 INFO mapred.JobClient:  map 68% reduce 8%
11/10/18 09:25:12 INFO mapred.JobClient:  map 75% reduce 8%
11/10/18 09:25:14 INFO mapred.JobClient:  map 81% reduce 9%
11/10/18 09:25:20 INFO mapred.JobClient:  map 87% reduce 9%
11/10/18 09:25:23 INFO mapred.JobClient:  map 93% reduce 12%
11/10/18 09:25:26 INFO mapred.JobClient:  map 93% reduce 13%
11/10/18 09:25:27 INFO mapred.JobClient:  map 100% reduce 14%
11/10/18 09:25:29 INFO mapred.JobClient:  map 100% reduce 15%
11/10/18 09:25:35 INFO mapred.JobClient:  map 100% reduce 16%
11/10/18 09:25:36 INFO mapred.JobClient:  map 100% reduce 17%
11/10/18 09:27:49 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000000_0, Status : FAILED
Too many fetch-failures
11/10/18 09:27:49 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:27:49 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:28:05 INFO mapred.JobClient:  map 100% reduce 18%
11/10/18 09:32:51 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000003_0, Status : FAILED
Too many fetch-failures
11/10/18 09:32:55 INFO mapred.JobClient:  map 93% reduce 18%
11/10/18 09:32:58 INFO mapred.JobClient:  map 100% reduce 18%
11/10/18 09:33:04 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000001_0, Status : FAILED
Too many fetch-failures
11/10/18 09:33:04 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:33:04 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:33:11 INFO mapred.JobClient:  map 100% reduce 19%
11/10/18 09:33:20 INFO mapred.JobClient:  map 100% reduce 20%
11/10/18 09:38:19 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000005_0, Status : FAILED
Too many fetch-failures
11/10/18 09:38:19 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:38:19 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:38:23 INFO mapred.JobClient:  map 93% reduce 20%
11/10/18 09:38:26 INFO mapred.JobClient:  map 100% reduce 20%
11/10/18 09:38:35 INFO mapred.JobClient:  map 100% reduce 21%
11/10/18 09:38:41 INFO mapred.JobClient:  map 100% reduce 22%
11/10/18 09:43:10 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000002_0, Status : FAILED
Too many fetch-failures
11/10/18 09:43:14 INFO mapred.JobClient:  map 93% reduce 22%
11/10/18 09:43:17 INFO mapred.JobClient:  map 100% reduce 22%
11/10/18 09:43:35 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000006_0, Status : FAILED
Too many fetch-failures
11/10/18 09:43:35 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:43:35 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:43:51 INFO mapred.JobClient:  map 100% reduce 24%
11/10/18 09:48:50 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000009_0, Status : FAILED
Too many fetch-failures
11/10/18 09:48:50 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:48:50 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:49:06 INFO mapred.JobClient:  map 100% reduce 25%
11/10/18 09:53:21 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000004_0, Status : FAILED
Too many fetch-failures
11/10/18 09:53:25 INFO mapred.JobClient:  map 93% reduce 25%
11/10/18 09:53:28 INFO mapred.JobClient:  map 100% reduce 25%
11/10/18 09:53:37 INFO mapred.JobClient:  map 100% reduce 26%
11/10/18 09:54:05 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000011_0, Status : FAILED
Too many fetch-failures
11/10/18 09:54:05 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:54:05 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:54:21 INFO mapred.JobClient:  map 100% reduce 27%
11/10/18 09:59:20 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000015_0, Status : FAILED
Too many fetch-failures
11/10/18 09:59:20 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:59:20 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 09:59:36 INFO mapred.JobClient:  map 100% reduce 52%
11/10/18 09:59:42 INFO mapred.JobClient:  map 100% reduce 53%
11/10/18 09:59:45 INFO mapred.JobClient:  map 100% reduce 54%
11/10/18 09:59:48 INFO mapred.JobClient:  map 100% reduce 55%
11/10/18 09:59:51 INFO mapred.JobClient:  map 100% reduce 56%
11/10/18 09:59:54 INFO mapred.JobClient:  map 100% reduce 57%
11/10/18 09:59:57 INFO mapred.JobClient:  map 100% reduce 58%
11/10/18 10:00:00 INFO mapred.JobClient:  map 100% reduce 60%
11/10/18 10:00:03 INFO mapred.JobClient:  map 100% reduce 61%
11/10/18 10:00:06 INFO mapred.JobClient:  map 100% reduce 62%
11/10/18 10:00:09 INFO mapred.JobClient:  map 100% reduce 63%
11/10/18 10:00:12 INFO mapred.JobClient:  map 100% reduce 64%
11/10/18 10:00:15 INFO mapred.JobClient:  map 100% reduce 65%
11/10/18 10:00:18 INFO mapred.JobClient:  map 100% reduce 66%
11/10/18 10:00:21 INFO mapred.JobClient:  map 100% reduce 67%
11/10/18 10:00:24 INFO mapred.JobClient:  map 100% reduce 68%
11/10/18 10:00:27 INFO mapred.JobClient:  map 100% reduce 69%
11/10/18 10:00:30 INFO mapred.JobClient:  map 100% reduce 70%
11/10/18 10:00:33 INFO mapred.JobClient:  map 100% reduce 71%
11/10/18 10:00:36 INFO mapred.JobClient:  map 100% reduce 72%
11/10/18 10:00:39 INFO mapred.JobClient:  map 100% reduce 73%
11/10/18 10:00:52 INFO mapred.JobClient:  map 100% reduce 75%
11/10/18 10:03:41 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000007_0, Status : FAILED
Too many fetch-failures
11/10/18 10:03:45 INFO mapred.JobClient:  map 93% reduce 75%
11/10/18 10:03:48 INFO mapred.JobClient:  map 100% reduce 75%
11/10/18 10:08:34 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000003_1, Status : FAILED
Too many fetch-failures
11/10/18 10:08:34 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 10:08:34 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 10:08:50 INFO mapred.JobClient:  map 100% reduce 76%
11/10/18 10:13:53 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000008_0, Status : FAILED
Too many fetch-failures
11/10/18 10:13:57 INFO mapred.JobClient:  map 93% reduce 76%
11/10/18 10:14:00 INFO mapred.JobClient:  map 100% reduce 76%
11/10/18 10:18:49 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000002_1, Status : FAILED
Too many fetch-failures
11/10/18 10:18:49 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 10:18:49 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 10:19:05 INFO mapred.JobClient:  map 100% reduce 77%
11/10/18 10:24:09 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000010_0, Status : FAILED
Too many fetch-failures
11/10/18 10:24:13 INFO mapred.JobClient:  map 93% reduce 77%
11/10/18 10:24:16 INFO mapred.JobClient:  map 100% reduce 77%
11/10/18 10:29:04 INFO mapred.JobClient: Task Id : attempt_201110180923_0001_m_000004_1, Status : FAILED
Too many fetch-failures
11/10/18 10:29:04 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 10:29:04 WARN mapred.JobClient: Error reading task outputxuwei-laptop
11/10/18 10:29:20 INFO mapred.JobClient:  map 100% reduce 89%
11/10/18 10:29:23 INFO mapred.JobClient:  map 100% reduce 91%
11/10/18 10:29:26 INFO mapred.JobClient:  map 100% reduce 92%
11/10/18 10:29:29 INFO mapred.JobClient:  map 100% reduce 93%
11/10/18 10:29:32 INFO mapred.JobClient:  map 100% reduce 94%
11/10/18 10:29:35 INFO mapred.JobClient:  map 100% reduce 95%
11/10/18 10:29:38 INFO mapred.JobClient:  map 100% reduce 96%
11/10/18 10:29:41 INFO mapred.JobClient:  map 100% reduce 97%
11/10/18 10:29:44 INFO mapred.JobClient:  map 100% reduce 98%
11/10/18 10:29:50 INFO mapred.JobClient:  map 100% reduce 100%
11/10/18 10:29:52 INFO mapred.JobClient: Job complete: job_201110180923_0001
11/10/18 10:29:52 INFO mapred.JobClient: Counters: 18
11/10/18 10:29:52 INFO mapred.JobClient:   Job Counters 
11/10/18 10:29:52 INFO mapred.JobClient:     Launched reduce tasks=4
11/10/18 10:29:52 INFO mapred.JobClient:     Launched map tasks=32
11/10/18 10:29:52 INFO mapred.JobClient:     Data-local map tasks=32
11/10/18 10:29:52 INFO mapred.JobClient:   FileSystemCounters
11/10/18 10:29:52 INFO mapred.JobClient:     FILE_BYTES_READ=1075141899
11/10/18 10:29:52 INFO mapred.JobClient:     HDFS_BYTES_READ=1077495458
11/10/18 10:29:52 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=2150285276
11/10/18 10:29:52 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1077290017
11/10/18 10:29:52 INFO mapred.JobClient:   Map-Reduce Framework
11/10/18 10:29:52 INFO mapred.JobClient:     Reduce input groups=102334
11/10/18 10:29:52 INFO mapred.JobClient:     Combine output records=0
11/10/18 10:29:52 INFO mapred.JobClient:     Map input records=102334
11/10/18 10:29:52 INFO mapred.JobClient:     Reduce shuffle bytes=1031027235
11/10/18 10:29:52 INFO mapred.JobClient:     Reduce output records=102334
11/10/18 10:29:52 INFO mapred.JobClient:     Spilled Records=204668
11/10/18 10:29:52 INFO mapred.JobClient:     Map output bytes=1074566657
11/10/18 10:29:52 INFO mapred.JobClient:     Map input bytes=1077289249
11/10/18 10:29:52 INFO mapred.JobClient:     Combine input records=0
11/10/18 10:29:52 INFO mapred.JobClient:     Map output records=102334
11/10/18 10:29:52 INFO mapred.JobClient:     Reduce input records=102334
Job ended: Tue Oct 18 10:29:52 CST 2011
The job took 3916 seconds.