大家好,今天我们在Hadoop 实战之分析专利引用数据集(一)和Hadoop 实战之分析专利引用数据集(三)的基础上来实现计算不同引用次数专利的数目
环境:Vmware 8.0 和Ubuntu11.04
Hadoop 实战之运行CitationHistogram---计算不同引用次数专利的数目
第一步:首先创建一个工程命名为HadoopTest.目录结构如下图:
sudo rm -rf /tmp/*
rm -rf /home/tanglg1987/hadoop-0.20.2/logs
hadoop namenode -format
hadoop datanode -format
start-all.sh
hadoop fs -mkdir input
hadoop dfsadmin -safemode leave
第三步:给start.sh增加执行权限并启动hadoop伪分布式集群,代码如下:
chmod 777 /home/tanglg1987/start.sh
./start.sh
执行过程如下:
第四步:把我上一篇博客:linux下运行MyJob3的运行结果result当成这里的输入input
博客地址:http://blog.csdn.net/xiaotom5/article/details/8088304
五步:新建一个CitationHistogram.java,代码如下:
package com.baison.action;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.KeyValueTextInputFormat;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class CitationHistogram extends Configured implements Tool {
public static class MapClass extends MapReduceBase
implements Mapper<Text, Text, IntWritable, IntWritable> {
private final static IntWritable uno = new IntWritable(1);
private IntWritable citationCount = new IntWritable();
public void map(Text key, Text value,
OutputCollector<IntWritable, IntWritable> output,
Reporter reporter) throws IOException {
citationCount.set(Integer.parseInt(value.toString()));
output.collect(citationCount, uno);
}
}
public static class Reduce extends MapReduceBase
implements Reducer<IntWritable,IntWritable,IntWritable,IntWritable>
{
public void reduce(IntWritable key, Iterator<IntWritable> values,
OutputCollector<IntWritable, IntWritable>output,
Reporter reporter) throws IOException {
int count = 0;
while (values.hasNext()) {
count += values.next().get();
}
output.collect(key, new IntWritable(count));
}
}
public int run(String[] args) throws Exception {
Configuration conf = getConf();
JobConf job = new JobConf(conf, CitationHistogram.class);
Path in = new Path(args[0]);
Path out = new Path(args[1]);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
job.setJobName("CitationHistogram");
job.setMapperClass(MapClass.class);
job.setReducerClass(Reduce.class);
job.setInputFormat(KeyValueTextInputFormat.class);
job.setOutputFormat(TextOutputFormat.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(IntWritable.class);
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception {
String [] arg={"hdfs://localhost:9100/user/tanglg1987/output"
,"hdfs://localhost:9100/user/tanglg1987/output/CitationHistogram"};
int res = ToolRunner.run(new Configuration(),
new CitationHistogram(),
arg);
System.exit(res);
}
}
第六步:Run On Hadoop,运行过程如下:
12/10/18 23:26:49 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/18 23:26:49 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/18 23:26:49 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 23:26:50 INFO mapred.JobClient: Running job: job_local_0001
12/10/18 23:26:50 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 23:26:50 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 23:26:50 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 23:26:50 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 23:26:50 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 23:26:50 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:50 INFO mapred.MapTask: bufstart = 0; bufend = 2097152; bufvoid = 99614720
12/10/18 23:26:50 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 23:26:51 INFO mapred.JobClient: map 0% reduce 0%
12/10/18 23:26:51 INFO mapred.MapTask: Finished spill 0
12/10/18 23:26:51 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:51 INFO mapred.MapTask: bufstart = 2097152; bufend = 4194296; bufvoid = 99614720
12/10/18 23:26:51 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 23:26:52 INFO mapred.MapTask: Finished spill 1
12/10/18 23:26:52 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:52 INFO mapred.MapTask: bufstart = 4194296; bufend = 6291440; bufvoid = 99614720
12/10/18 23:26:52 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 23:26:52 INFO mapred.MapTask: Finished spill 2
12/10/18 23:26:53 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:53 INFO mapred.MapTask: bufstart = 6291440; bufend = 8388584; bufvoid = 99614720
12/10/18 23:26:53 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 23:26:53 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:26:53 INFO mapred.MapTask: Finished spill 3
12/10/18 23:26:53 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:53 INFO mapred.MapTask: bufstart = 8388584; bufend = 10485736; bufvoid = 99614720
12/10/18 23:26:53 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 23:26:54 INFO mapred.JobClient: map 31% reduce 0%
12/10/18 23:26:54 INFO mapred.MapTask: Finished spill 4
12/10/18 23:26:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:54 INFO mapred.MapTask: bufstart = 10485736; bufend = 12582880; bufvoid = 99614720
12/10/18 23:26:54 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 23:26:55 INFO mapred.MapTask: Finished spill 5
12/10/18 23:26:55 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:55 INFO mapred.MapTask: bufstart = 12582880; bufend = 14680024; bufvoid = 99614720
12/10/18 23:26:55 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 23:26:56 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:26:56 INFO mapred.MapTask: Finished spill 6
12/10/18 23:26:56 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:56 INFO mapred.MapTask: bufstart = 14680024; bufend = 16777168; bufvoid = 99614720
12/10/18 23:26:56 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 23:26:57 INFO mapred.JobClient: map 58% reduce 0%
12/10/18 23:26:57 INFO mapred.MapTask: Finished spill 7
12/10/18 23:26:57 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:57 INFO mapred.MapTask: bufstart = 16777168; bufend = 18874312; bufvoid = 99614720
12/10/18 23:26:57 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 23:26:58 INFO mapred.MapTask: Finished spill 8
12/10/18 23:26:58 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:58 INFO mapred.MapTask: bufstart = 18874312; bufend = 20971464; bufvoid = 99614720
12/10/18 23:26:58 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 23:26:58 INFO mapred.MapTask: Finished spill 9
12/10/18 23:26:59 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:26:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:59 INFO mapred.MapTask: bufstart = 20971464; bufend = 23068608; bufvoid = 99614720
12/10/18 23:26:59 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 23:26:59 INFO mapred.MapTask: Finished spill 10
12/10/18 23:26:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:59 INFO mapred.MapTask: bufstart = 23068608; bufend = 25165752; bufvoid = 99614720
12/10/18 23:26:59 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 23:27:00 INFO mapred.JobClient: map 88% reduce 0%
12/10/18 23:27:00 INFO mapred.MapTask: Finished spill 11
12/10/18 23:27:00 INFO mapred.MapTask: Starting flush of map output
12/10/18 23:27:00 INFO mapred.MapTask: Finished spill 12
12/10/18 23:27:00 INFO mapred.Merger: Merging 13 sorted segments
12/10/18 23:27:00 INFO mapred.Merger: Merging 4 intermediate segments out of a total of 13
12/10/18 23:27:01 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 32589860 bytes
12/10/18 23:27:02 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:27:03 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 23:27:05 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:27:05 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/18 23:27:05 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:27:05 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/18 23:27:05 INFO mapred.LocalJobRunner:
12/10/18 23:27:05 INFO mapred.Merger: Merging 1 sorted segments
12/10/18 23:27:05 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 32589842 bytes
12/10/18 23:27:05 INFO mapred.LocalJobRunner:
12/10/18 23:27:06 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/18 23:27:06 INFO mapred.LocalJobRunner:
12/10/18 23:27:06 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/18 23:27:06 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9100/user/tanglg1987/output/CitationHistogram
12/10/18 23:27:06 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 23:27:06 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/18 23:27:07 INFO mapred.JobClient: map 100% reduce 100%
12/10/18 23:27:07 INFO mapred.JobClient: Job complete: job_local_0001
12/10/18 23:27:07 INFO mapred.JobClient: Counters: 15
12/10/18 23:27:07 INFO mapred.JobClient: FileSystemCounters
12/10/18 23:27:07 INFO mapred.JobClient: FILE_BYTES_READ=118774812
12/10/18 23:27:07 INFO mapred.JobClient: HDFS_BYTES_READ=65717226
12/10/18 23:27:07 INFO mapred.JobClient: FILE_BYTES_WRITTEN=151399190
12/10/18 23:27:07 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1721
12/10/18 23:27:07 INFO mapred.JobClient: Map-Reduce Framework
12/10/18 23:27:07 INFO mapred.JobClient: Reduce input groups=258
12/10/18 23:27:07 INFO mapred.JobClient: Combine output records=0
12/10/18 23:27:07 INFO mapred.JobClient: Map input records=3258984
12/10/18 23:27:07 INFO mapred.JobClient: Reduce shuffle bytes=0
12/10/18 23:27:07 INFO mapred.JobClient: Reduce output records=258
12/10/18 23:27:07 INFO mapred.JobClient: Spilled Records=10825525
12/10/18 23:27:07 INFO mapred.JobClient: Map output bytes=26071872
12/10/18 23:27:07 INFO mapred.JobClient: Map input bytes=32858613
12/10/18 23:27:07 INFO mapred.JobClient: Combine input records=0
12/10/18 23:27:07 INFO mapred.JobClient: Map output records=3258984
12/10/18 23:27:07 INFO mapred.JobClient: Reduce input records=3258984
第七步:查看结果集,运行结果如下:
![](https://img-my.csdn.net/uploads/201210/19/1350610831_1587.png)