Hadoop 实战之运行AveragingWithCombiner(二)

环境:Vmware 8.0 和ubuntu11.04

Hadoop 实战之运行AveragingWithCombiner---使用计数器统计缺失值个数的MapClass

第一步:首先创建一个工程命名为HadoopTest.目录结构如下图:

第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件,每次启动虚拟机都要删除/tmp目录下的全部文件,重新格式化namenode,代码如下:
sudo rm -rf /tmp/*
rm -rf /home/tanglg1987/hadoop-0.20.2/logs
hadoop namenode -format
hadoop datanode -format
start-all.sh
hadoop fs -mkdir input 
hadoop dfsadmin -safemode leave

第三步:给start.sh增加执行权限并启动hadoop伪分布式集群,代码如下:

chmod 777 /home/tanglg1987/start.sh
./start.sh 

执行过程如下:

第四步:上传本地文件到hdfs

在专利局http://data.nber.org/patents/网站下载专利数据

http://data.nber.org/patents/apat63_99.zip

hadoop fs -put /home/tanglg1987/apat63_99.txt input

五步:新建一个AveragingWithCombiner2.java,代码如下:

package com.baison.action;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class AveragingWithCombiner2 extends Configured implements Tool {   
    public static class MapClass extends MapReduceBase
        implements Mapper<LongWritable, Text, Text, Text> {      
        static enum ClaimsCounters { MISSING, QUOTED };      
        public void map(LongWritable key, Text value,
                        OutputCollector<Text, Text> output,
                        Reporter reporter) throws IOException {                     
            String fields[] = value.toString().split(",", -20);
            String country = fields[4];
            String numClaims = fields[8];
            if (numClaims.length() == 0) {
                reporter.incrCounter(ClaimsCounters.MISSING, 1);
            } else if (numClaims.startsWith("\"")) {
                reporter.incrCounter(ClaimsCounters.QUOTED, 1);
            } else {
                output.collect(new Text(country), new Text(numClaims + ",1"));
            }           
        }
    }   
    public static class Combine extends MapReduceBase
        implements Reducer<Text, Text, Text, Text> {     
        public void reduce(Text key, Iterator<Text> values,
                           OutputCollector<Text, Text> output,
                           Reporter reporter) throws IOException {                         
            double sum = 0;
            int count = 0;
            while (values.hasNext()) {
                String fields[] = values.next().toString().split(",");
                sum += Double.parseDouble(fields[0]);
                count += Integer.parseInt(fields[1]);
            }
            output.collect(key, new Text(sum + "," + count));
        }
    }   
    public static class Reduce extends MapReduceBase
        implements Reducer<Text, Text, Text, DoubleWritable> {     
        public void reduce(Text key, Iterator<Text> values,
                           OutputCollector<Text, DoubleWritable> output,
                           Reporter reporter) throws IOException {                        
            double sum = 0;
            int count = 0;
            while (values.hasNext()) {
                String fields[] = values.next().toString().split(",");
                sum += Double.parseDouble(fields[0]);
                count += Integer.parseInt(fields[1]);
            }
            output.collect(key, new DoubleWritable(sum/count));
        }
    }   
    public int run(String[] args) throws Exception {
        // Configuration processed by ToolRunner
        Configuration conf = getConf();    
        // Create a JobConf using the processed conf
        JobConf job = new JobConf(conf, AveragingWithCombiner.class);    
        // Process custom command-line options
        Path in = new Path(args[0]);
        Path out = new Path(args[1]);
        FileInputFormat.setInputPaths(job, in);
        FileOutputFormat.setOutputPath(job, out);   
        // Specify various job-specific parameters     
        job.setJobName("AveragingWithCombiner");
        job.setMapperClass(MapClass.class);
        job.setCombinerClass(Combine.class);
        job.setReducerClass(Reduce.class);  
        job.setInputFormat(TextInputFormat.class);
        job.setOutputFormat(TextOutputFormat.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);     
        // Submit the job, then poll for progress until the job is complete
        JobClient.runJob(job); 
        return 0;
    }
    
    public static void main(String[] args) throws Exception {
        // Let ToolRunner handle generic command-line options 
     String [] arg={"hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt"
 ,"hdfs://localhost:9100/user/tanglg1987/output"};
        int res = ToolRunner.run(new Configuration(), new AveragingWithCombiner(), arg);      
        System.exit(res);
    }
}

第六步:Run On Hadoop,运行过程如下:

12/10/19 20:36:29 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/19 20:36:29 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/19 20:36:29 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/19 20:36:30 INFO mapred.JobClient: Running job: job_local_0001
12/10/19 20:36:30 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/19 20:36:30 INFO mapred.MapTask: numReduceTasks: 1
12/10/19 20:36:30 INFO mapred.MapTask: io.sort.mb = 100
12/10/19 20:36:30 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/19 20:36:30 INFO mapred.MapTask: record buffer = 262144/327680
12/10/19 20:36:31 INFO mapred.JobClient:  map 0% reduce 0%
12/10/19 20:36:33 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:0+67108864
12/10/19 20:36:34 INFO mapred.JobClient:  map 28% reduce 0%
12/10/19 20:36:36 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:0+67108864
12/10/19 20:36:37 INFO mapred.JobClient:  map 80% reduce 0%
12/10/19 20:36:37 INFO mapred.MapTask: Starting flush of map output
12/10/19 20:36:37 INFO mapred.MapTask: Finished spill 0
12/10/19 20:36:37 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/19 20:36:37 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:0+67108864
12/10/19 20:36:37 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/19 20:36:38 INFO mapred.MapTask: numReduceTasks: 1
12/10/19 20:36:38 INFO mapred.MapTask: io.sort.mb = 100
12/10/19 20:36:38 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/19 20:36:38 INFO mapred.MapTask: record buffer = 262144/327680
12/10/19 20:36:38 INFO mapred.JobClient:  map 100% reduce 0%
12/10/19 20:36:40 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:40 INFO mapred.MapTask: bufstart = 0; bufend = 2467890; bufvoid = 99614720
12/10/19 20:36:40 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/19 20:36:41 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:67108864+67108864
12/10/19 20:36:41 INFO mapred.JobClient:  map 67% reduce 0%
12/10/19 20:36:41 INFO mapred.MapTask: Finished spill 0
12/10/19 20:36:43 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:43 INFO mapred.MapTask: bufstart = 2467890; bufend = 4942611; bufvoid = 99614720
12/10/19 20:36:43 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/19 20:36:43 INFO mapred.MapTask: Finished spill 1
12/10/19 20:36:44 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:67108864+67108864
12/10/19 20:36:44 INFO mapred.JobClient:  map 89% reduce 0%
12/10/19 20:36:45 INFO mapred.MapTask: Starting flush of map output
12/10/19 20:36:45 INFO mapred.MapTask: Finished spill 2
12/10/19 20:36:45 INFO mapred.Merger: Merging 3 sorted segments
12/10/19 20:36:45 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 4411 bytes
12/10/19 20:36:45 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
12/10/19 20:36:45 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:67108864+67108864
12/10/19 20:36:45 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done.
12/10/19 20:36:45 INFO mapred.MapTask: numReduceTasks: 1
12/10/19 20:36:45 INFO mapred.MapTask: io.sort.mb = 100
12/10/19 20:36:45 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/19 20:36:45 INFO mapred.MapTask: record buffer = 262144/327680
12/10/19 20:36:46 INFO mapred.JobClient:  map 100% reduce 0%
12/10/19 20:36:47 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:47 INFO mapred.MapTask: bufstart = 0; bufend = 2496693; bufvoid = 99614720
12/10/19 20:36:47 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/19 20:36:48 INFO mapred.MapTask: Finished spill 0
12/10/19 20:36:48 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:134217728+67108864
12/10/19 20:36:49 INFO mapred.JobClient:  map 81% reduce 0%
12/10/19 20:36:49 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:50 INFO mapred.MapTask: bufstart = 2496693; bufend = 5000178; bufvoid = 99614720
12/10/19 20:36:50 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/19 20:36:51 INFO mapred.MapTask: Finished spill 1
12/10/19 20:36:51 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:134217728+67108864
12/10/19 20:36:52 INFO mapred.JobClient:  map 94% reduce 0%
12/10/19 20:36:52 INFO mapred.MapTask: Starting flush of map output
12/10/19 20:36:53 INFO mapred.MapTask: Finished spill 2
12/10/19 20:36:53 INFO mapred.Merger: Merging 3 sorted segments
12/10/19 20:36:53 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 4656 bytes
12/10/19 20:36:53 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
12/10/19 20:36:53 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:134217728+67108864
12/10/19 20:36:53 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000002_0' done.
12/10/19 20:36:53 INFO mapred.MapTask: numReduceTasks: 1
12/10/19 20:36:53 INFO mapred.MapTask: io.sort.mb = 100
12/10/19 20:36:53 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/19 20:36:53 INFO mapred.JobClient:  map 100% reduce 0%
12/10/19 20:36:53 INFO mapred.MapTask: record buffer = 262144/327680
12/10/19 20:36:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:54 INFO mapred.MapTask: bufstart = 0; bufend = 2526823; bufvoid = 99614720
12/10/19 20:36:54 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/19 20:36:55 INFO mapred.MapTask: Finished spill 0
12/10/19 20:36:56 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:201326592+35576587
12/10/19 20:36:56 INFO mapred.JobClient:  map 98% reduce 0%
12/10/19 20:36:56 INFO mapred.MapTask: Starting flush of map output
12/10/19 20:36:56 INFO mapred.MapTask: Finished spill 1
12/10/19 20:36:56 INFO mapred.Merger: Merging 2 sorted segments
12/10/19 20:36:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2613 bytes
12/10/19 20:36:56 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
12/10/19 20:36:56 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:201326592+35576587
12/10/19 20:36:56 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000003_0' done.
12/10/19 20:36:56 INFO mapred.LocalJobRunner:
12/10/19 20:36:56 INFO mapred.Merger: Merging 4 sorted segments
12/10/19 20:36:56 INFO mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 7858 bytes
12/10/19 20:36:56 INFO mapred.LocalJobRunner:
12/10/19 20:36:56 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/19 20:36:56 INFO mapred.LocalJobRunner:
12/10/19 20:36:56 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/19 20:36:56 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9100/user/tanglg1987/output
12/10/19 20:36:56 INFO mapred.LocalJobRunner: reduce > reduce
12/10/19 20:36:56 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/19 20:36:57 INFO mapred.JobClient:  map 100% reduce 100%
12/10/19 20:36:57 INFO mapred.JobClient: Job complete: job_local_0001
12/10/19 20:36:57 INFO mapred.JobClient: Counters: 15
12/10/19 20:36:57 INFO mapred.JobClient:   FileSystemCounters
12/10/19 20:36:57 INFO mapred.JobClient:     FILE_BYTES_READ=129472
12/10/19 20:36:57 INFO mapred.JobClient:     HDFS_BYTES_READ=876508703
12/10/19 20:36:57 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=236890
12/10/19 20:36:57 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=2658
12/10/19 20:36:57 INFO mapred.JobClient:   Map-Reduce Framework
12/10/19 20:36:57 INFO mapred.JobClient:     Reduce input groups=151
12/10/19 20:36:57 INFO mapred.JobClient:     Combine output records=1063
12/10/19 20:36:57 INFO mapred.JobClient:     Map input records=2923923
12/10/19 20:36:57 INFO mapred.JobClient:     Reduce shuffle bytes=0
12/10/19 20:36:57 INFO mapred.JobClient:     Reduce output records=151
12/10/19 20:36:57 INFO mapred.JobClient:     Spilled Records=1721
12/10/19 20:36:57 INFO mapred.JobClient:     Map output bytes=18862764
12/10/19 20:36:57 INFO mapred.JobClient:     Map input bytes=236903179
12/10/19 20:36:57 INFO mapred.JobClient:     Combine input records=1984625
12/10/19 20:36:57 INFO mapred.JobClient:     Map output records=1984055
12/10/19 20:36:57 INFO mapred.JobClient:     Reduce input records=493

第七步:查看结果集,运行结果如下:

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值