MapReduce编程开发之数据去重

最新推荐文章于 2024-03-16 13:36:01 发布
最新推荐文章于 2024-03-16 13:36:01 发布
阅读量1.6k 收藏 12
点赞数 1
分类专栏： hadoop 文章标签： MapReduce 数据去重
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/feinifi/article/details/100172673
版权

    MapReduce就是一个利用分而治之的思想做计算的框架，所谓分，就是将数据打散，分成可以计算的小份，治就是将数据合并，相同键的数据合并成一个集合。MapReduce并不能解决所有的问题，因为他的数据类型是键值对，只能解决特定范围的问题。

    数据去重算法，其实就是词频统计的一个变种，词频统计是统计文本中的单词出现的次数，如果一个单词出现两次，就算重复，去掉重复的办法就是保留一个副本，键值对表示的话，我们只需要保留键的值就可以了。其实MapReduce最适合干的工作就是数据去重，因为在reduce阶段，所有的数据的键都是唯一的，正好满足数据去重的要求。

    在map阶段，我们将value作为key输出，将new Text()作为value就是空输出，这样我们就把所有的数据遍历了一遍，这里面，我们几乎不用什么复杂的算法。

    在shuffle阶段，这些值相同的key会被合并到一块，value-list是一个new Text()的集合，如果有重复的记录，那么这个集合的长度会大于1，而且这些key还会进行默认的排序。

    在reduce阶段，我们的输入会变为<key , <value-list>>，我们只需要将key一一输出即可，这样就达到了数据去重的目的。思路其实很简单。

算法代码：

package com.xxx.hadoop.mapred;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
 * 数据去重
 *
 */
public class DeDuplicationApp {
	
	public static class Map extends Mapper<Object, Text, Text, Text>{
		protected void map(Object key, Text value,
				Context context) throws IOException ,InterruptedException {
			context.write(value, new Text());
		};
	}
	
	public static class Reduce extends Reducer<Text, Text, Text, Text>{
		protected void reduce(Text key, java.lang.Iterable<Text> values,
				Context context) throws IOException ,InterruptedException {
			context.write(key, new Text());
		};
	}

	public static void main(String[] args) throws Exception {
		String input = "/user/root/deduplication/input",
			  output = "/user/root/deduplication/output";
		System.setProperty("HADOOP_USER_NAME", "root");
		Configuration conf = new Configuration();
		conf.set("fs.defaultFS", "hdfs://192.168.56.202:9000");
		
		Job job = Job.getInstance(conf);
		job.setJarByClass(DeDuplicationApp.class);
		job.setMapperClass(Map.class);
		job.setReducerClass(Reduce.class);
		FileInputFormat.addInputPath(job, new Path(input));
		FileOutputFormat.setOutputPath(job, new Path(output));
		
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(Text.class);
		
		System.exit(job.waitForCompletion(true)?0:1);
	}

}

原始数据：

运行程序，控制台打印信息如下：

2019-08-31 14:33:56 [INFO ]  [main]  [org.apache.hadoop.conf.Configuration.deprecation] session.id is deprecated. Instead, use dfs.metrics.session-id
2019-08-31 14:33:56 [INFO ]  [main]  [org.apache.hadoop.metrics.jvm.JvmMetrics] Initializing JVM Metrics with processName=JobTracker, sessionId=
2019-08-31 14:33:57 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-08-31 14:33:57 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2019-08-31 14:33:57 [INFO ]  [main]  [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] Total input paths to process : 3
2019-08-31 14:33:57 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] number of splits:3
2019-08-31 14:33:57 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] Submitting tokens for job: job_local1661418487_0001
2019-08-31 14:33:58 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] The url to track the job: http://localhost:8080/
2019-08-31 14:33:58 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Running job: job_local1661418487_0001
2019-08-31 14:33:58 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter set in config null
2019-08-31 14:33:58 [INFO ]  [Thread-3]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:58 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-08-31 14:33:58 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] Waiting for map tasks
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1661418487_0001_m_000000_0
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@3c3c2e72
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/deduplication/input/a.txt:0+12
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 15; bufvoid = 104857600
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1661418487_0001_m_000000_0 is done. And is in the process of committing
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1661418487_0001_m_000000_0' done.
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1661418487_0001_m_000000_0
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1661418487_0001_m_000001_0
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:58 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4b250759
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/deduplication/input/b.txt:0+12
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 15; bufvoid = 104857600
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1661418487_0001_m_000001_0 is done. And is in the process of committing
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1661418487_0001_m_000001_0' done.
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1661418487_0001_m_000001_0
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1661418487_0001_m_000002_0
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@77ad94ef
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/deduplication/input/c.txt:0+12
2019-08-31 14:33:59 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Job job_local1661418487_0001 running in uber mode : false
2019-08-31 14:33:59 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job]  map 100% reduce 0%
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 15; bufvoid = 104857600
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1661418487_0001_m_000002_0 is done. And is in the process of committing
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1661418487_0001_m_000002_0' done.
2019-08-31 14:33:59 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1661418487_0001_m_000002_0
2019-08-31 14:33:59 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] map task executor complete.
2019-08-31 14:33:59 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] Waiting for reduce tasks
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1661418487_0001_r_000000_0
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@651b16b8
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.ReduceTask] Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@12884be
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] MergerManager: memoryLimit=1265788544, maxSingleShuffleLimit=316447136, mergeThreshold=835420480, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2019-08-31 14:33:59 [INFO ]  [EventFetcher for fetching Map Completion Events]  [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] attempt_local1661418487_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2019-08-31 14:33:59 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1661418487_0001_m_000002_0 decomp: 23 len: 27 to MEMORY
2019-08-31 14:33:59 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 23 bytes from map-output for attempt_local1661418487_0001_m_000002_0
2019-08-31 14:33:59 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 23, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->23
2019-08-31 14:33:59 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1661418487_0001_m_000001_0 decomp: 23 len: 27 to MEMORY
2019-08-31 14:33:59 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 23 bytes from map-output for attempt_local1661418487_0001_m_000001_0
2019-08-31 14:33:59 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 23, inMemoryMapOutputs.size() -> 2, commitMemory -> 23, usedMemory ->46
2019-08-31 14:33:59 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1661418487_0001_m_000000_0 decomp: 23 len: 27 to MEMORY
2019-08-31 14:33:59 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 23 bytes from map-output for attempt_local1661418487_0001_m_000000_0
2019-08-31 14:33:59 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 23, inMemoryMapOutputs.size() -> 3, commitMemory -> 46, usedMemory ->69
2019-08-31 14:33:59 [INFO ]  [EventFetcher for fetching Map Completion Events]  [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] EventFetcher is interrupted.. Returning
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Merging 3 sorted segments
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 3 segments left of total size: 51 bytes
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merged 3 segments, 69 bytes to disk to satisfy reduce memory limit
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 1 files, 69 bytes from disk
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 0 segments, 0 bytes from memory into reduce
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Merging 1 sorted segments
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 1 segments left of total size: 59 bytes
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.conf.Configuration.deprecation] mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task:attempt_local1661418487_0001_r_000000_0 is done. And is in the process of committing
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task attempt_local1661418487_0001_r_000000_0 is allowed to commit now
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] Saved output of task 'attempt_local1661418487_0001_r_000000_0' to hdfs://192.168.56.202:9000/user/root/deduplication/output/_temporary/0/task_local1661418487_0001_r_000000
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] reduce > reduce
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1661418487_0001_r_000000_0' done.
2019-08-31 14:33:59 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1661418487_0001_r_000000_0
2019-08-31 14:33:59 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] reduce task executor complete.
2019-08-31 14:34:00 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job]  map 100% reduce 100%
2019-08-31 14:34:00 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Job job_local1661418487_0001 completed successfully
2019-08-31 14:34:00 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Counters: 35
	File System Counters
		FILE: Number of bytes read=4066
		FILE: Number of bytes written=1095628
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=108
		HDFS: Number of bytes written=30
		HDFS: Number of read operations=33
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=6
	Map-Reduce Framework
		Map input records=9
		Map output records=9
		Map output bytes=45
		Map output materialized bytes=81
		Input split bytes=381
		Combine input records=0
		Combine output records=0
		Reduce input groups=6
		Reduce shuffle bytes=81
		Reduce input records=9
		Reduce output records=6
		Spilled Records=18
		Shuffled Maps =3
		Failed Shuffles=0
		Merged Map outputs=3
		GC time elapsed (ms)=8
		Total committed heap usage (bytes)=1476919296
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=36
	File Output Format Counters 
		Bytes Written=30

计算之后的数据：

    这里面数据去重，其实是利用了MapReduce的特点，没有复杂的设计思路。普通的数组去重，我们会遍历数组，然后保存遍历的值，后续遍历的值如果在已经遍历过的值集合中，那么就丢弃。而MapReduce框架是一个分而治之的计算框架，分为几个阶段，每一个阶段都有自己的任务，正是这每些阶段将我们的数据做了分组和排序，所以数据就自动的去重了。

luffy5459
关注
  • 1
    点赞
  • 踩
  • 12
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
mapreduce 丢数据_大数据之MapReduce详解
weixin_39896839的博客
12-17 518
1.什么是Map/Reduce，看下面的各种解释：(1)MapReduce是hadoop的核心组件之一，hadoop要分布式包括两部分，一是分布式文件系统hdfs,一部是分布式计算框，就是mapreduce,缺一不可，也就是说，可以通过mapreduce很容易在hadoop平台上进行分布式的计算编程。(2)Mapreduce是一种编程模型，是一种编程方法，抽象理论。(3)下面是一个关于一个程序员是...
007_hadoop中MapReduce应用案例_1_数据去重
08-06
NULL 博文链接：https://zc985552943.iteye.com/blog/2088221
MapReduce功能实现五---去重(Distinct)、计数(Count)
小强签名设计 的博客
07-26 1万+
一、去重 类似于db中的select distinct(x) from table , 去重处理甚至比WordCount还要简单 [hadoop@h71 q1]$ vi hello.txt hello world hello hadoop hello hive hello hadoop hello world hello world [hadoop@h71 q1]$ hadoop fs -
【Hadoop大数据技术】——MapReduce经典案例实战（倒排索引、数据去重、TopN）
最新发布
HinsCoder的博客
03-16 2342
MapReduce是一种分布式并行编程模型，是Hadoop核心子项目之一。
MapReduce实现数据去重
weixin_33872660的博客
12-02 1002
一、原理分析 　　Mapreduce的处理过程，由于Mapreduce会在Map~reduce中，将重复的Key合并在一起，所以Mapreduce很容易就去除重复的行。Map无须做任何处理，设置Map中写入context的东西为不作任何处理的行，也就是Map中最初处理的value即可，而Reduce同样无须做任何处理，写入输出文件的东西就是，最初得到的K...
MapReduce 基础案例 之 数据去重
努力吧闫小胖的博客
09-04 1133
数据去重 "数据去重"主要是为了掌握和利用并行化思想来对数据进行有意义的筛选。统计大数据集上的数据种类个数、从网站日志中计算访问地等这些看似庞杂的任务都会涉及数据去重。下面就进入这个实例的MapReduce程序设计。 思想 对于数据去重问题，每个元素出现了多少次是不重要，因为不需要多次的进行判断是否已经存在这个元素，我只要确定每个元素只要出现过，并且最后能够显示出来就可以。那么通过map阶...
大数据之Hadoop分布式计算框架MapReduce（三）
weixin_45666566的博客
12-15 478
MapReduce一、MapReduce概述1.1 MapReduce定义1.2 MapReduce优缺点1.3 应用场景二、MapReduce框架原理2.1 执行流程2.2 MapReduce编程模型2.3 单词统计小案例 一、MapReduce概述 1.1 MapReduce定义 \quad \quadMapReduce是一个用于处理海量数据的分布式计算框架，并发运行在一个Hadoop集群上。 1.2 MapReduce优缺点 优点 MapReduce易于编程 它简单的实现一些接口，就可以完成一个分
LocalJobRunner
hsh8819的专栏
07-08 600
       hadoop作业分本地模式和分布式模式两种执行模式，JobClient初始化时会读取配置项mapred.job.tracker(默认为local)，如果该配置项的值为local，则hadoop采本地模式执行作业，否则采用分布式模式执行。本地模式使用LocalJobRuner提交并执行作业。对LocalJobRunner实例调用submitJob( )方法会创建Job(LocalJ...
MapReduce-合并去重实验
m0_52931616的博客
11-17 1024
本次Mapreduce-合并去重实验在windows操作系统下进行，使用IDEA构建Maven工程编译运行。
Python Mapreduce实例——去重
weixin_44087585的博客
08-25 2530
任务目标 1.准确理解mapreduce去重的设计原理 2.熟练掌握mapreduce去重的程序编写 3.学会自己编写MapReduce去重代码解决实际问题 相关知识 “数据去重”主要是为了掌握和利用并行化思想来对数据进行有意义的筛选。统计大数据集上的数据种类个数、从网站日志中计算访问地等这些看似庞杂的任务都会涉及数据去重。 数据去重的最终目标是让原始数据中出现次数超过一次的数据在输出文件中只出现...
Sqoop使用
Adolph的专栏
07-16 509
  Sqoop的本质还是一个命令行工具，和HDFS，MapReduce相比，并没有什么高深的理论。 我们可以通过sqoop help命令来查看sqoop的命令选项，如下：   16/11/13 20:10:17 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 usage: sqoop COMMAND [ARGS]   Available ...
Hadoop-MapReduce实践示例
06-20
初学大数据hadoop，自己写实践练手的示例，示例非常全，带目录，非常清晰。
MapReduce基础开发之二数据去重和排序
医疗影像检索
06-23 6074
因Hadoop集群平台网络限制，只能在eclipse里先写好代码再提交jar到集群平台namenode上执行，不能实时调试，所以没有配置eclipse的hadoop开发环境，只是引入了hadoop的lib包。 eclipse的hadoop开发环境配置可参考：http://www.cnblogs.com/xia520pi/archive/2012/05/20/2510723.html MapRe
MapReduce实例----数据去重
w_t_y_y的博客
05-31 7606
数据去重的最终目标是让原始数据中出现次数超过一次的数据在输出文件中只出现一次。我们自然而然会想到将同一个数据的所有记录都交给一台reduce机器，无论这个数据出现多少次，只要在最终结果中输出一次就可以了。具体就是reduce的输入应该以数据作为key，而对value-list则没有要求。当reduce接收到一个时就直接将key复制到输出的key中，并将value设置成空值。 　　在MapRedu
mapreduce 数据去重
あずにゃん梓喵的博客
04-30 424
代码下载链接： https://pan.baidu.com/s/1VhxRXB63vz_4QZyKVFqoZQ 提取码：2v0l 1.经过了maptask：把一行数据作为key，value为空 2.每个各自的mapreduce中的相同顺序的分区中的数据都传到同一个reducetask中进行全局合并：key=一行数据。value=为空的迭代器 ...
Hadoop--MapReduce2--自定义序列化数据类型
大魔王
02-27 1317
在MapReduce编程模型中，Map阶段以及Reduce阶段都需要对输入输出的数据进行序列化以及反序列化，因此输入输出的参数必须实现特定的接口Writable，具体来说Hadoop对基本数据类型已做封装如Text,LongWritable,IntWritable,FloatWritable,NullWritable等。 Writable接口定义如下： @Public @Sta...
Hadoop - Map/Reduce 通过理解org.apache.hadoop.mapreduce.Job类来学习hadoop的执行逻辑
热门推荐
Derekjiang的笔记簿
10-08 1万+
在我的上篇文章“Hadoop - Map/Reduce 通过WordCount例子的变化来了解新版hadoop接口的变化”中，已经跟大家提到，在Hadoop新的版本中，使用org.apache.hadoop.mapreduce.Job类替代了JobClient类。现在，我想通过对
mapreduce 去重的问题怎么解决
微尘的世界
06-25 300
  登录 | 注册     http://bbs.csdn.net/topics/390811736?page=1#post-397617777   返回列表 管理菜单 结帖 发帖 回复 关注 mapreduce 去重的问题怎么解决？ [问题点数：40分] 快速回复只显...
运用MapReduce编程实现数据去重
05-20
MapReduce是一种分布式计算框架，用于处理大规模数据集。在MapReduce中，数据被分成小块，并分配给多个计算节点进行处理。在进行数据去重时，可以使用MapReduce编程模型来实现。 以下是使用MapReduce编程实现数据去重的步骤： 1. Map阶段：将输入数据切分成小块，并对每个小块进行处理。在处理时，可以使用哈希表来进行数据去重，将每个不同的数据作为键值对的键，将任意值作为值存储在哈希表中。这样，相同的数据将被哈希表合并为一个键值对，完成数据去重。 2. Shuffle阶段：将Map阶段输出的键值对按照键进行排序，将相同的键值对聚合在一起。 3. Reduce阶段：对Shuffle阶段输出的键值对进行处理，将相同的键值对合并为一个键值对输出。这样，就完成了数据去重操作。 下面是使用Python实现MapReduce程序的示例代码： ``` from mrjob.job import MRJob class DataDeduplication(MRJob): def mapper(self, _, line): data = line.strip() yield data, 1 def reducer(self, data, _): yield data, None if __name__ == '__main__': DataDeduplication.run() ``` 在上面的代码中，mapper函数将输入数据作为键值对的键，将任意值作为值输出。在reducer函数中，将相同的键值对合并为一个键值对，输出不包含值的键值对，完成数据去重操作。 以上就是使用MapReduce编程实现数据去重的步骤和示例代码。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

luffy5459

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值