Hadoop 实战之分析专利引用数据集（二）

最新推荐文章于 2020-07-10 15:13:29 发布

置顶

xiaocaichonga

最新推荐文章于 2020-07-10 15:13:29 发布

阅读量5.6k

点赞数

分类专栏： hadoop 文章标签： linux output buffer input string csv

本文链接：https://blog.csdn.net/xiaocaichonga/article/details/8088612

版权

环境：Vmware 8.0 和Ubuntu11.04

Hadoop 实战之分析专利引用数据集（二）---计算专利引用数据并排序

第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:

第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件，每次启动虚拟机都要删除/tmp目录下的全部文件，重新格式化namenode，代码如下：

sudo rm -rf /tmp/*
rm -rf /home/tanglg1987/hadoop-0.20.2/logs
hadoop namenode -format
hadoop datanode -format
start-all.sh
hadoop fs -mkdir input 
hadoop dfsadmin -safemode leave

第三步：给start.sh增加执行权限并启动hadoop伪分布式集群，代码如下:

chmod 777 /home/tanglg1987/ start.sh
./start.sh

执行过程如下：

第四步：上传本地文件到hdfs

在专利局http://data.nber.org/patents/网站下载专利数据

http://data.nber.org/patents/apat63_99.zip

hadoop fs -put /home/tanglg1987/cite75_99.txt input

五步：新建一个MyJob2.java，代码如下：

package com.baison.action;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class MyJob2 extends Configured implements Tool {  
    public static class MapClass extends Mapper<LongWritable, Text, Text, Text> {       
        public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {                      
            String[] citation = value.toString().split(",");
            context.write(new Text(citation[1]), new Text(citation[0]));
        }
    }    
    public static class Reduce extends Reducer<Text, Text, Text, Text> {       
        public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {                         
            String csv = "";
            for (Text val:values) {
                if (csv.length() > 0) csv += ",";
                csv += val.toString();
            }         
            context.write(key, new Text(csv));
        }
    }
    
    public int run(String[] args) throws Exception {
        Configuration conf = getConf();   
        Job job = new Job(conf, "MyJob");
        job.setJarByClass(MyJob.class);     
        Path in = new Path(args[0]);
        Path out = new Path(args[1]);
        FileInputFormat.setInputPaths(job, in);
        FileOutputFormat.setOutputPath(job, out);   
        job.setMapperClass(MapClass.class);
        job.setReducerClass(Reduce.class);    
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);       
        System.exit(job.waitForCompletion(true)?0:1);    
        return 0;
    }  
    public static void main(String[] args) throws Exception { 
    	String [] arg={"hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt","hdfs://localhost:9100/user/tanglg1987/output"};
        int res = ToolRunner.run(new Configuration(), new MyJob(), arg);        
        System.exit(res);
    }
}

第六步：Run On Hadoop，运行过程如下：

hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt
hdfs://localhost:9100/user/tanglg1987/output
12/10/18 22:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/18 22:17:45 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/18 22:17:46 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 22:17:46 INFO mapred.JobClient: Running job: job_local_0001
12/10/18 22:17:46 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 22:17:46 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:17:46 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:17:46 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:17:46 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:17:47 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:47 INFO mapred.MapTask: bufstart = 0; bufend = 4185926; bufvoid = 99614720
12/10/18 22:17:47 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:17:47 INFO mapred.JobClient: map 0% reduce 0%
12/10/18 22:17:48 INFO mapred.MapTask: Finished spill 0
12/10/18 22:17:48 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:48 INFO mapred.MapTask: bufstart = 4185926; bufend = 8372612; bufvoid = 99614720
12/10/18 22:17:48 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:17:49 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:49 INFO mapred.MapTask: Finished spill 1
12/10/18 22:17:50 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:50 INFO mapred.MapTask: bufstart = 8372612; bufend = 12558934; bufvoid = 99614720
12/10/18 22:17:50 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:17:50 INFO mapred.JobClient: map 14% reduce 0%
12/10/18 22:17:51 INFO mapred.MapTask: Finished spill 2
12/10/18 22:17:51 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:51 INFO mapred.MapTask: bufstart = 12558934; bufend = 16745318; bufvoid = 99614720
12/10/18 22:17:51 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:17:52 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:52 INFO mapred.MapTask: Finished spill 3
12/10/18 22:17:52 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:52 INFO mapred.MapTask: bufstart = 16745318; bufend = 20931889; bufvoid = 99614720
12/10/18 22:17:52 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:17:53 INFO mapred.JobClient: map 26% reduce 0%
12/10/18 22:17:53 INFO mapred.MapTask: Finished spill 4
12/10/18 22:17:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:54 INFO mapred.MapTask: bufstart = 20931889; bufend = 25118734; bufvoid = 99614720
12/10/18 22:17:54 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:17:55 INFO mapred.MapTask: Finished spill 5
12/10/18 22:17:55 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:55 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:55 INFO mapred.MapTask: bufstart = 25118734; bufend = 29305269; bufvoid = 99614720
12/10/18 22:17:55 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:17:56 INFO mapred.JobClient: map 42% reduce 0%
12/10/18 22:17:56 INFO mapred.MapTask: Finished spill 6
12/10/18 22:17:56 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:56 INFO mapred.MapTask: bufstart = 29305269; bufend = 33492764; bufvoid = 99614720
12/10/18 22:17:56 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:17:57 INFO mapred.MapTask: Finished spill 7
12/10/18 22:17:57 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:57 INFO mapred.MapTask: bufstart = 33492764; bufend = 37680391; bufvoid = 99614720
12/10/18 22:17:57 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:17:58 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:58 INFO mapred.MapTask: Finished spill 8
12/10/18 22:17:59 INFO mapred.JobClient: map 57% reduce 0%
12/10/18 22:17:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:59 INFO mapred.MapTask: bufstart = 37680391; bufend = 41868206; bufvoid = 99614720
12/10/18 22:17:59 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:18:00 INFO mapred.MapTask: Finished spill 9
12/10/18 22:18:00 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:00 INFO mapred.MapTask: bufstart = 41868206; bufend = 46056257; bufvoid = 99614720
12/10/18 22:18:00 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:18:01 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:02 INFO mapred.MapTask: Finished spill 10
12/10/18 22:18:02 INFO mapred.JobClient: map 70% reduce 0%
12/10/18 22:18:02 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:02 INFO mapred.MapTask: bufstart = 46056257; bufend = 50244288; bufvoid = 99614720
12/10/18 22:18:02 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:18:04 INFO mapred.MapTask: Finished spill 11
12/10/18 22:18:04 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:04 INFO mapred.MapTask: bufstart = 50244288; bufend = 54432271; bufvoid = 99614720
12/10/18 22:18:04 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:18:04 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:05 INFO mapred.JobClient: map 82% reduce 0%
12/10/18 22:18:05 INFO mapred.MapTask: Finished spill 12
12/10/18 22:18:07 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:07 INFO mapred.MapTask: bufstart = 54432271; bufend = 58620635; bufvoid = 99614720
12/10/18 22:18:07 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:18:07 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:07 INFO mapred.JobClient: map 88% reduce 0%
12/10/18 22:18:08 INFO mapred.MapTask: Finished spill 13
12/10/18 22:18:08 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:08 INFO mapred.MapTask: bufstart = 58620635; bufend = 62808941; bufvoid = 99614720
12/10/18 22:18:08 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:18:10 INFO mapred.MapTask: Finished spill 14
12/10/18 22:18:10 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:10 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:10 INFO mapred.MapTask: bufstart = 62808941; bufend = 66997060; bufvoid = 99614720
12/10/18 22:18:10 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:18:10 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:18:10 INFO mapred.JobClient: map 99% reduce 0%
12/10/18 22:18:11 INFO mapred.MapTask: Finished spill 15
12/10/18 22:18:11 INFO mapred.MapTask: Finished spill 16
12/10/18 22:18:11 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:18:11 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:18:13 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:13 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:18:15 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75511476 bytes
12/10/18 22:18:16 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:19 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:21 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/18 22:18:21 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:21 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/18 22:18:22 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:18:22 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:18:22 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:18:22 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:18:22 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:22 INFO mapred.MapTask: bufstart = 0; bufend = 4188247; bufvoid = 99614720
12/10/18 22:18:22 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:18:23 INFO mapred.MapTask: Finished spill 0
12/10/18 22:18:23 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:23 INFO mapred.MapTask: bufstart = 4188247; bufend = 8376832; bufvoid = 99614720
12/10/18 22:18:23 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:18:24 INFO mapred.MapTask: Finished spill 1
12/10/18 22:18:25 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:25 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:25 INFO mapred.MapTask: bufstart = 8376832; bufend = 12565282; bufvoid = 99614720
12/10/18 22:18:25 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length =

最低0.47元/天解锁文章

xiaocaichonga

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
Hadoop 实战之分析专利引用数据集（二）

环境：Vmware 8.0 和Ubuntu11.04Hadoop 实战之分析专利引用数据集（二）---计算专利引用数据并排序第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件，每次启动虚拟机都要删除/tmp目录下的全部文件，重新格式化namenode，代码如下：sudo
复制链接

扫一扫