MapReduce编程开发之倒排索引

最新推荐文章于 2024-04-28 18:20:22 发布
最新推荐文章于 2024-04-28 18:20:22 发布
阅读量743 收藏 7
点赞数 1
分类专栏： hadoop 文章标签： MapReduce 倒排索引
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/feinifi/article/details/100191499
版权

    倒排索引是词频统计的一个变种，其实也是做一个词频统计，不过这个词频统计需要加上文件的名称。倒排索引被广泛用来做全文检索。倒排索引最终的结果是一个单词在文件中出现的次数的集合，以下面的数据为例：

file1.txt

hdfs hadoop mapreduce
hdfs bigdata
hadoop mapreduce

file2.txt

mapreduce hdfs
hadoop bigdata mapreduce
hdfs hadoop
hdfs mapreduce
bigdata hadoop

file3.txt

bigdata hadoop mapreduce hdfs
hadoop hdfs mapreduce
bigdata hadoop

最终的结果：

bigdata	file3.txt:2;file2.txt:2;file1.txt:1;
hadoop	file1.txt:2;file3.txt:3;file2.txt:3;
hdfs	file2.txt:3;file1.txt:2;file3.txt:2;
mapreduce	file3.txt:2;file1.txt:2;file2.txt:3;

倒排索引的设计思路和词频统计有些类似，但是不完全一样，这里我们需要对单词的文件也加进来做统计。要达到这个效果，首先是需要对单词以及他所出现的文件做一个统计，所以map中，我们和做词频统计一样，只不过输出的词不是单个的词，而是词+":"+文件，类似这样：

<hdfs:file1.txt , 1>
<hdfs:file2.txt , 1>
<hdfs:file3.txt , 1>
<hadoop:file1.txt , 1>
<mapreduce:file3.txt , 1>

在reduce之前，我们需要进行一个combine的过程，这个过程里面，我们的输入数据是这样的<词：文件 ， <统计次数>>：

<hdfs:file1.txt , <1,1>>
<hadoop:file1.txt , <1,1>>
<hdfs:file2.txt , <1,1,1>>
<hdfs:file3.txt , <1,1>>

我们做个改动，输出为<词，文件：统计次数和>

<hdfs , file1.txt:2>
<hdfs , file2.txt:3>
<hdfs , file3.txt:2>

在reduce阶段，我们就将values拼接即可。用一张图来总结这个过程就是：

下面给出倒排索引的代码：

package com.xxx.hadoop.mapred;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
 * 倒排索引
 *
 */
public class InvertIndexApp {

	public static class Map extends Mapper<LongWritable, Text, Text, Text>{
		private Text keyinfo = new Text();
		private Text valueinfo = new Text();
		private FileSplit split;
		protected void map(LongWritable key, Text value,
				Context context) throws IOException ,InterruptedException {
			split = (FileSplit)context.getInputSplit();
			StringTokenizer tokenizer = new StringTokenizer(value.toString());
			while(tokenizer.hasMoreTokens()) {
				int splitIndex = split.getPath().toString().indexOf("file");
				keyinfo.set(tokenizer.nextToken()+":"+split.getPath().toString().substring(splitIndex));
				valueinfo.set("1");
				context.write(keyinfo, valueinfo);
			}
		}
	}
	
	
	public static class Combine extends Reducer<Text, Text, Text, Text>{
		private Text info = new Text();
		protected void reduce(Text key, java.lang.Iterable<Text> values, 
				Context context) throws IOException ,InterruptedException {
			int sum = 0;
			for(Text value:values) {
				sum += Integer.parseInt(value.toString());
			}
			int splitIndex = key.toString().indexOf(":");
			info.set(key.toString().substring(splitIndex+1)+":"+sum);
			key.set(key.toString().substring(0,splitIndex));
			context.write(key, info);
		}
	}
	
	public static class Reduce extends Reducer<Text, Text, Text, Text>{
		private Text result = new Text();
		protected void reduce(Text key, Iterable<Text> values,
				Context context) throws java.io.IOException ,InterruptedException {	
			String filelist = new String();
			for(Text value:values) {
				filelist += value.toString()+";";
			}
			result.set(filelist);
			context.write(key, result);
		};
	}
	public static void main(String[] args) throws Exception{
		String input = "/user/root/invertindex/input";
		String output = "/user/root/invertindex/output";
		System.setProperty("HADOOP_USER_NAME", "root");
		Configuration conf = new Configuration();
		conf.set("fs.defaultFS", "hdfs://192.168.56.202:9000");
		FileSystem fs = FileSystem.get(conf);
		boolean exists = fs.exists(new Path(output));
		if(exists) {
			fs.delete(new Path(output), true);
		}
		Job job = Job.getInstance(conf);
		job.setJarByClass(InvertIndexApp.class);
		
		job.setMapperClass(Map.class);
		job.setCombinerClass(Combine.class);
		job.setReducerClass(Reduce.class);
		
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(Text.class);
		
		FileInputFormat.addInputPath(job, new Path(input));
		FileOutputFormat.setOutputPath(job, new Path(output));
		System.exit(job.waitForCompletion(true)?0:1);
	}

}

运行之前的准备数据：

运行程序，控制台打印信息如下：

2019-09-02 09:56:22 [INFO ]  [main]  [org.apache.hadoop.conf.Configuration.deprecation] session.id is deprecated. Instead, use dfs.metrics.session-id
2019-09-02 09:56:22 [INFO ]  [main]  [org.apache.hadoop.metrics.jvm.JvmMetrics] Initializing JVM Metrics with processName=JobTracker, sessionId=
2019-09-02 09:56:22 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-09-02 09:56:22 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2019-09-02 09:56:22 [INFO ]  [main]  [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] Total input paths to process : 3
2019-09-02 09:56:22 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] number of splits:3
2019-09-02 09:56:22 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] Submitting tokens for job: job_local1888565320_0001
2019-09-02 09:56:23 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] The url to track the job: http://localhost:8080/
2019-09-02 09:56:23 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Running job: job_local1888565320_0001
2019-09-02 09:56:23 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter set in config null
2019-09-02 09:56:23 [INFO ]  [Thread-3]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-09-02 09:56:23 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-09-02 09:56:23 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] Waiting for map tasks
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1888565320_0001_m_000000_0
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@e65aa68
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/invertindex/input/file2.txt:0+82
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 214; bufvoid = 104857600
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1888565320_0001_m_000000_0 is done. And is in the process of committing
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1888565320_0001_m_000000_0' done.
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1888565320_0001_m_000000_0
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1888565320_0001_m_000001_0
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4012ecfc
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/invertindex/input/file3.txt:0+67
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 175; bufvoid = 104857600
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1888565320_0001_m_000001_0 is done. And is in the process of committing
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1888565320_0001_m_000001_0' done.
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1888565320_0001_m_000001_0
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1888565320_0001_m_000002_0
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@157add31
2019-09-02 09:56:23 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/invertindex/input/file1.txt:0+52
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 136; bufvoid = 104857600
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214372(104857488); length = 25/6553600
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1888565320_0001_m_000002_0 is done. And is in the process of committing
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1888565320_0001_m_000002_0' done.
2019-09-02 09:56:24 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1888565320_0001_m_000002_0
2019-09-02 09:56:24 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] map task executor complete.
2019-09-02 09:56:24 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Job job_local1888565320_0001 running in uber mode : false
2019-09-02 09:56:24 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job]  map 100% reduce 0%
2019-09-02 09:56:24 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] Waiting for reduce tasks
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1888565320_0001_r_000000_0
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@27512f1a
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.ReduceTask] Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4ac60620
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] MergerManager: memoryLimit=1265788544, maxSingleShuffleLimit=316447136, mergeThreshold=835420480, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2019-09-02 09:56:24 [INFO ]  [EventFetcher for fetching Map Completion Events]  [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] attempt_local1888565320_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2019-09-02 09:56:24 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1888565320_0001_m_000001_0 decomp: 88 len: 92 to MEMORY
2019-09-02 09:56:24 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 88 bytes from map-output for attempt_local1888565320_0001_m_000001_0
2019-09-02 09:56:24 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 88, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->88
2019-09-02 09:56:24 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1888565320_0001_m_000002_0 decomp: 88 len: 92 to MEMORY
2019-09-02 09:56:24 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 88 bytes from map-output for attempt_local1888565320_0001_m_000002_0
2019-09-02 09:56:24 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 88, inMemoryMapOutputs.size() -> 2, commitMemory -> 88, usedMemory ->176
2019-09-02 09:56:24 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1888565320_0001_m_000000_0 decomp: 88 len: 92 to MEMORY
2019-09-02 09:56:24 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 88 bytes from map-output for attempt_local1888565320_0001_m_000000_0
2019-09-02 09:56:24 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 88, inMemoryMapOutputs.size() -> 3, commitMemory -> 176, usedMemory ->264
2019-09-02 09:56:24 [INFO ]  [EventFetcher for fetching Map Completion Events]  [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] EventFetcher is interrupted.. Returning
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Merging 3 sorted segments
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 3 segments left of total size: 234 bytes
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merged 3 segments, 264 bytes to disk to satisfy reduce memory limit
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 1 files, 264 bytes from disk
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 0 segments, 0 bytes from memory into reduce
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Merging 1 sorted segments
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 1 segments left of total size: 250 bytes
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.conf.Configuration.deprecation] mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task:attempt_local1888565320_0001_r_000000_0 is done. And is in the process of committing
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task attempt_local1888565320_0001_r_000000_0 is allowed to commit now
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] Saved output of task 'attempt_local1888565320_0001_r_000000_0' to hdfs://192.168.56.202:9000/user/root/invertindex/output/_temporary/0/task_local1888565320_0001_r_000000
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] reduce > reduce
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1888565320_0001_r_000000_0' done.
2019-09-02 09:56:24 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1888565320_0001_r_000000_0
2019-09-02 09:56:24 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] reduce task executor complete.
2019-09-02 09:56:25 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job]  map 100% reduce 100%
2019-09-02 09:56:25 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Job job_local1888565320_0001 completed successfully
2019-09-02 09:56:25 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Counters: 35
	File System Counters
		FILE: Number of bytes read=4510
		FILE: Number of bytes written=1098080
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=633
		HDFS: Number of bytes written=178
		HDFS: Number of read operations=37
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=10
	Map-Reduce Framework
		Map input records=11
		Map output records=27
		Map output bytes=525
		Map output materialized bytes=276
		Input split bytes=387
		Combine input records=27
		Combine output records=12
		Reduce input groups=4
		Reduce shuffle bytes=276
		Reduce input records=12
		Reduce output records=4
		Spilled Records=24
		Shuffled Maps =3
		Failed Shuffles=0
		Merged Map outputs=3
		GC time elapsed (ms)=8
		Total committed heap usage (bytes)=1485307904
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=201
	File Output Format Counters 
		Bytes Written=178

查看运行结果：

    这个和普通的MapReduce程序不太一样的地方是，我们在map,reduce之间显示的方式指定了一个combine的过程，这个过程里面，我们修改了<key,value>，让他在reduce阶段更简单。其实普通的MapReduce程序也有combine的过程，只不过这个combine别我们忽略了，因为它的默认实现就是reduce的实现。

luffy5459
关注
  • 1
    点赞
  • 踩
  • 7
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
MapReduce排序--倒排序索引
Junds0的博客
11-15 476
但在Shuffle过程时将面临一个问题：所有具有相同单词的记录（由单词、URL和词频组成）应该交由同一个Reducer处理，但当前的key值无法保证这一点，所以必须修改 key值和value 值。它主要是用来存储某个单词（或词组）在一个文档或一组文档中的存储位置的映射，即提 供了一种根据内容来查找文档的方式。由于不是根据文档来确定文档所包含的内容，而是进行相反的操作，因而称为倒排索引。首先通过输入的文件进行处理，得到文件每行的偏移量和内容，分析输入键值对，得到倒排序索引中所需的单词、文档路径以及单词频率。
Hadoop编程实验4 倒排索引
最新发布
m0_63099685的博客
08-11 18
实验目的和要求编写MapReduce程序，实现倒排索引。理解Map、Reduce端键值对形式的输入输出。操作系统软件:JDK-1.8Hadoop-2.7Eclipse数据存放路径：tar包路径：tar包压缩路径：软件安装路径:/opt实验设计创建文件：1、实验内容（1）在Eclipse等编译器中编写map程序；（2）在Eclipse等编译器中编写combine程序；（3）在Eclipse等编译器中编写reduce程序；
用MapReduce实现倒排索引（多job串联）
weixin_43087634的博客
12-27 651
需求：有大量的文本（文档、网页），需要建立搜索索引 所谓倒排索引就是把多个文件的字段进行逐次统计，进行展示。 比如说一个test文件夹下有多个文件：a.txt b.txt c.txt a.txt 中的字段为 ： ni ss ni b.txt 中的字段为：ni dd ni c.txt 中的字段为：ni hao ss (1),第一次MapReduce方法预期输出为 ...
MapReduce实现倒排索引
Running-小猛的博客
03-25 2995
分为两步进行 第一步： package com.jym.hadoop.mr.inverindex; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoo...
MapReduce编程之倒排索引的实现
爱笑的博客
01-18 863
转：http://www.cnblogs.com/aijianiula/p/3870664.html 倒排索引简单地就是：根据单词，返回它在哪个文件中出现过，而且频率是多少的结果。这就像百度里的搜索，你输入一个关键字，那么百度引擎就迅速的在它的服务器里找到有该关键字的文件，并根据频率和其他一些策略（如页面点击投票率）等来给你返回结果。这个过程中，倒排索引就起到很关键的作用。 ---
MapReduce编程之倒排索引
sunlei1980的专栏
06-12 1564
任务要求： //输入文件格式 18661629496 110 13107702446 110 1234567 120 2345678 120 987654 110 2897839274 18661629496 //输出文件格式格式 11018661629496|13107702446|987654|18661629496|13107702446
MapReduce之倒排索引
qq_43653581的博客
03-31 355
倒排索引算法 预备环境 网上对于下面的配置和安装都比较多。 Maven的安装以及在IDEA中对Maven进行了配置； 搜索对应系统版本下（比如windows7）编译过的hadoop jar包并进行相关环境变量配置； 总之就是先要在windows搭建hadoop环境。 操作步骤 1.新建maven项目，在pom.xml中新增依赖： <build> <plugin...
MapReduce经典案例-倒排索引的实现流程
m0_51679196的博客
09-03 1258
mapreduce经典案例倒排索引
Hadoop编程基于MR程序实现倒排索引示例
09-30
在本文中，我们将探讨如何使用Hadoop的MapReduce（MR）框架实现倒排索引，这是一种在大规模数据处理中常见的技术，特别是在搜索引擎和文本分析领域。倒排索引允许快速定位文档中包含特定关键词的位置，极大地提高了...
mapreduce编程 倒排索引
09-08
在MapReduce编程中，倒排索引是一种常见的应用案例。倒排索引（Inverted index）是一种索引方法，用于存储某个单词在一个文档或一组文档中的位置映射。它是文档检索系统中最常用的数据结构。 在实现倒排索引的...
大数据系统开发实践项目——安装完全分布式Hadoop集群并利用MapReduce实现倒排索引（北理工大三小学期）
2301_79058506的博客
04-28 839
接着，在倒排索引的构建过程中，再次运用MapReduce任务，应用TF-IDF算法，将词项映射到包含它的文档，并记录词项在文档中的位置信息，以计算每个词项的在文档中的重要性，确保了对文档的全面索引，为后续的搜索排序提供基础。Reduce阶段首先以键值对形式接收来自Map部分的输出，其中键为词项，值为文档ID和文档内容的列表，然后针对相同的词项，将它们对应的文档ID和文档内容列表合并，形成倒排索引的条目，最后生成输出最终的键值对，其中键为词项，值为包含该词项的文档ID和位置信息。
Mapreduce实现倒排索引建立
成都往右的博客
10-16 1072
需求：有大量的文本（文档、网页），需要建立搜索索引
MapReduce倒排索引实现
BinChasing的博客
12-05 1360
输入数据 输出数据代码package com.test.ReversedIndex;import java.io.IOException; import java.util.StringTokenizer;import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org
MapReduce入门（三）倒排索引
weixin_30856965的博客
10-13 348
什么是倒排索引？ 倒排索引源于实际应用中需要根据属性的值来查找记录。这种索引表中的每一项都包括一个属性值和具有该属性值的各记录的地址。由于不是由记录来确定属性值，而是由属性值来确定记录的位置，因而称为倒排索引(inverted index)。带有倒排索引的文件我们称为倒排索引文件，简称倒排文件(inverted file)。 我感觉搜索引...
MapReduce编程——文档倒排索引
weixin_45835541的博客
12-30 1610
目录实验要求代码第一次mapreducemapper类reducer类main第二次mapreducemapper类reducer类partitionermain实验数据运行结果第一次mapreduce结果第二次mapreduce结果总结 实验要求 实现倒排索引效果：统计每个单词在不同文件中的出现次数；倒排索引的原理参考实验说明； 输入：自己编辑几个文件，例如 a.txt,b.txt,c.txt。每个文件的内容为若干行单词，单词之间以空格分开，并将这些文件上传到 hdfs 的/in 目录下；例如：a.tx
MapReducer——实现倒排索引（11）
付之一笑的专栏
09-20 809
倒排索引数据处理的过程 使用MapReduce实现倒排索引 RevertedIndexMapper.java package com.revertedindex; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

luffy5459

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值