Linux 下运行wordcount的过程详解

最新推荐文章于 2022-01-05 00:04:27 发布
最新推荐文章于 2022-01-05 00:04:27 发布
阅读量2.3k 收藏 2
点赞数
分类专栏： hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xfg0218/article/details/52740327
版权
[root@hadoop1 ~]# hadoop jar wordcount.jar bigDate.LinuxWordCount    // 运行的命令
16/10/05 01:40:03 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/10/05 01:40:03 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/10/05 01:40:04 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/10/05 01:40:04 INFO input.FileInputFormat: Total input paths to process : 8     // 获取输入的文件的数量
16/10/05 01:40:04 INFO mapreduce.JobSubmitter: number of splits:8     // 切片的个数，也就是说一个文件一片
16/10/05 01:40:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1040123772_0001   // job的ID
16/10/05 01:40:06 INFO mapreduce.Job: The url to track the job: http://localhost:8080/       // jobTrack 运行
16/10/05 01:40:06 INFO mapreduce.Job: Running job: job_local1040123772_0001
16/10/05 01:40:06 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/10/05 01:40:06 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/10/05 01:40:07 INFO mapred.LocalJobRunner: Waiting for map tasks
16/10/05 01:40:07 INFO mapred.LocalJobRunner: Starting task: attempt_local1040123772_0001_m_000000_0
16/10/05 01:40:07 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:40:07 INFO mapred.MapTask: Processing split: hdfs://hadoop1:9000/wordcount/hadoop-policy.xml:0+9683
16/10/05 01:40:07 INFO mapreduce.Job: Job job_local1040123772_0001 running in uber mode : false
16/10/05 01:40:07 INFO mapreduce.Job:  map 0% reduce 0%             // map与reduce的进度
16/10/05 01:40:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:40:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:40:08 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:40:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:40:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:40:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:40:08 INFO mapred.LocalJobRunner: 
16/10/05 01:40:08 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:40:08 INFO mapred.MapTask: Spilling map output
16/10/05 01:40:08 INFO mapred.MapTask: bufstart = 0; bufend = 16910; bufvoid = 104857600
16/10/05 01:40:08 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26207164(104828656); length = 7233/6553600
16/10/05 01:40:08 INFO mapred.MapTask: Finished spill 0
16/10/05 01:40:08 INFO mapred.Task: Task:attempt_local1040123772_0001_m_000000_0 is done. And is in the process of committing
16/10/05 01:40:08 INFO mapred.LocalJobRunner: map
16/10/05 01:40:08 INFO mapred.Task: Task 'attempt_local1040123772_0001_m_000000_0' done.
16/10/05 01:40:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local1040123772_0001_m_000000_0
16/10/05 01:40:08 INFO mapred.LocalJobRunner: Starting task: attempt_local1040123772_0001_m_000001_0
16/10/05 01:40:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:40:08 INFO mapred.MapTask: Processing split: hdfs://hadoop1:9000/wordcount/kms-site.xml:0+5511
16/10/05 01:40:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:40:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:40:08 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:40:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:40:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:40:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:40:08 INFO mapred.LocalJobRunner: 
16/10/05 01:40:08 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:40:08 INFO mapred.MapTask: Spilling map output
16/10/05 01:40:08 INFO mapred.MapTask: bufstart = 0; bufend = 9727; bufvoid = 104857600
16/10/05 01:40:08 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26210184(104840736); length = 4213/6553600
16/10/05 01:40:08 INFO mapred.MapTask: Finished spill 0
16/10/05 01:40:08 INFO mapreduce.Job:  map 13% reduce 0%            // map  与reduce 的进度
16/10/05 01:40:08 INFO mapred.Task: Task:attempt_local1040123772_0001_m_000001_0 is done. And is in the process of committing
16/10/05 01:40:08 INFO mapred.LocalJobRunner: map
16/10/05 01:40:08 INFO mapred.Task: Task 'attempt_local1040123772_0001_m_000001_0' done.
16/10/05 01:40:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local1040123772_0001_m_000001_0
16/10/05 01:40:08 INFO mapred.LocalJobRunner: Starting task: attempt_local1040123772_0001_m_000002_0
16/10/05 01:40:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:40:08 INFO mapred.MapTask: Processing split: hdfs://hadoop1:9000/wordcount/capacity-scheduler.xml:0+4436
16/10/05 01:40:09 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:40:09 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:40:09 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:40:09 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:40:09 INFO mapred.LocalJobRunner: 
16/10/05 01:40:09 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:40:09 INFO mapred.MapTask: Spilling map output
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufend = 7953; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26210876(104843504); length = 3521/6553600
16/10/05 01:40:09 INFO mapred.MapTask: Finished spill 0
16/10/05 01:40:09 INFO mapred.Task: Task:attempt_local1040123772_0001_m_000002_0 is done. And is in the process of committing
16/10/05 01:40:09 INFO mapred.LocalJobRunner: map
16/10/05 01:40:09 INFO mapred.Task: Task 'attempt_local1040123772_0001_m_000002_0' done.
16/10/05 01:40:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local1040123772_0001_m_000002_0
16/10/05 01:40:09 INFO mapred.LocalJobRunner: Starting task: attempt_local1040123772_0001_m_000003_0
16/10/05 01:40:09 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:40:09 INFO mapred.MapTask: Processing split: hdfs://hadoop1:9000/wordcount/kms-acls.xml:0+3523
16/10/05 01:40:09 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:40:09 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:40:09 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:40:09 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:40:09 INFO mapred.LocalJobRunner: 
16/10/05 01:40:09 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:40:09 INFO mapred.MapTask: Spilling map output
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufend = 6587; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26211336(104845344); length = 3061/6553600
16/10/05 01:40:09 INFO mapred.MapTask: Finished spill 0
16/10/05 01:40:09 INFO mapred.Task: Task:attempt_local1040123772_0001_m_000003_0 is done. And is in the process of committing
16/10/05 01:40:09 INFO mapred.LocalJobRunner: map
16/10/05 01:40:09 INFO mapred.Task: Task 'attempt_local1040123772_0001_m_000003_0' done.
16/10/05 01:40:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local1040123772_0001_m_000003_0
16/10/05 01:40:09 INFO mapred.LocalJobRunner: Starting task: attempt_local1040123772_0001_m_000004_0
16/10/05 01:40:09 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:40:09 INFO mapred.MapTask: Processing split: hdfs://hadoop1:9000/wordcount/hdfs-site.xml:0+1149
16/10/05 01:40:09 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:40:09 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:40:09 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:40:09 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:40:09 INFO mapred.LocalJobRunner: 
16/10/05 01:40:09 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:40:09 INFO mapred.MapTask: Spilling map output
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufend = 1713; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213836(104855344); length = 561/6553600
16/10/05 01:40:09 INFO mapred.MapTask: Finished spill 0
16/10/05 01:40:09 INFO mapred.Task: Task:attempt_local1040123772_0001_m_000004_0 is done. And is in the process of committing
16/10/05 01:40:09 INFO mapred.LocalJobRunner: map
16/10/05 01:40:09 INFO mapred.Task: Task 'attempt_local1040123772_0001_m_000004_0' done.
16/10/05 01:40:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local1040123772_0001_m_000004_0
16/10/05 01:40:09 INFO mapred.LocalJobRunner: Starting task: attempt_local1040123772_0001_m_000005_0
16/10/05 01:40:09 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:40:09 INFO mapred.MapTask: Processing split: hdfs://hadoop1:9000/wordcount/core-site.xml:0+952
16/10/05 01:40:09 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:40:09 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:40:09 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:40:09 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:40:09 INFO mapred.LocalJobRunner: 
16/10/05 01:40:09 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:40:09 INFO mapred.MapTask: Spilling map output
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufend = 1484; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213868(104855472); length = 529/6553600
16/10/05 01:40:09 INFO mapred.MapTask: Finished spill 0
16/10/05 01:40:09 INFO mapred.Task: Task:attempt_local1040123772_0001_m_000005_0 is done. And is in the process of committing
16/10/05 01:40:09 INFO mapred.LocalJobRunner: map
16/10/05 01:40:09 INFO mapred.Task: Task 'attempt_local1040123772_0001_m_000005_0' done.
16/10/05 01:40:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local1040123772_0001_m_000005_0
16/10/05 01:40:09 INFO mapred.LocalJobRunner: Starting task: attempt_local1040123772_0001_m_000006_0
16/10/05 01:40:09 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:40:09 INFO mapred.MapTask: Processing split: hdfs://hadoop1:9000/wordcount/yarn-site.xml:0+823
16/10/05 01:40:09 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:40:09 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:40:09 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:40:09 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:40:09 INFO mapreduce.Job:  map 100% reduce 0%      // map 与reduce的进度
16/10/05 01:40:09 INFO mapred.LocalJobRunner: 
16/10/05 01:40:09 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:40:09 INFO mapred.MapTask: Spilling map output
16/10/05 01:40:09 INFO mapred.MapTask: bufstart = 0; bufend = 1295; bufvoid = 104857600
16/10/05 01:40:09 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213928(104855712); length = 469/6553600
16/10/05 01:40:09 INFO mapred.MapTask: Finished spill 0
16/10/05 01:40:09 INFO mapred.Task: Task:attempt_local1040123772_0001_m_000006_0 is done. And is in the process of committing
16/10/05 01:40:10 INFO mapred.LocalJobRunner: map
16/10/05 01:40:10 INFO mapred.Task: Task 'attempt_local1040123772_0001_m_000006_0' done.
16/10/05 01:40:10 INFO mapred.LocalJobRunner: Finishing task: attempt_local1040123772_0001_m_000006_0
16/10/05 01:40:10 INFO mapred.LocalJobRunner: Starting task: attempt_local1040123772_0001_m_000007_0
16/10/05 01:40:10 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:40:10 INFO mapred.MapTask: Processing split: hdfs://hadoop1:9000/wordcount/httpfs-site.xml:0+620
16/10/05 01:40:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:40:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:40:10 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:40:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:40:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:40:10 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:40:10 INFO mapred.LocalJobRunner: 
16/10/05 01:40:10 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:40:10 INFO mapred.MapTask: Spilling map output
16/10/05 01:40:10 INFO mapred.MapTask: bufstart = 0; bufend = 1044; bufvoid = 104857600
16/10/05 01:40:10 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213976(104855904); length = 421/6553600
16/10/05 01:40:10 INFO mapred.MapTask: Finished spill 0
16/10/05 01:40:10 INFO mapred.Task: Task:attempt_local1040123772_0001_m_000007_0 is done. And is in the process of committing
16/10/05 01:40:10 INFO mapred.LocalJobRunner: map
16/10/05 01:40:10 INFO mapred.Task: Task 'attempt_local1040123772_0001_m_000007_0' done.
16/10/05 01:40:10 INFO mapred.LocalJobRunner: Finishing task: attempt_local1040123772_0001_m_000007_0
16/10/05 01:40:10 INFO mapred.LocalJobRunner: map task executor complete.
16/10/05 01:40:10 INFO mapred.LocalJobRunner: Waiting for reduce tasks
16/10/05 01:40:10 INFO mapred.LocalJobRunner: Starting task: attempt_local1040123772_0001_r_000000_0
16/10/05 01:40:10 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:40:10 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@43e3a075
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/10/05 01:40:10 INFO reduce.EventFetcher: attempt_local1040123772_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
16/10/05 01:40:10 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1040123772_0001_m_000003_0 decomp: 8121 len: 8125 to MEMORY
16/10/05 01:40:10 INFO reduce.InMemoryMapOutput: Read 8121 bytes from map-output for attempt_local1040123772_0001_m_000003_0
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 8121, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->8121
16/10/05 01:40:10 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1040123772_0001_m_000000_0 decomp: 20530 len: 20534 to MEMORY
16/10/05 01:40:10 INFO reduce.InMemoryMapOutput: Read 20530 bytes from map-output for attempt_local1040123772_0001_m_000000_0
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 20530, inMemoryMapOutputs.size() -> 2, commitMemory -> 8121, usedMemory ->28651
16/10/05 01:40:10 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1040123772_0001_m_000004_0 decomp: 1997 len: 2001 to MEMORY
16/10/05 01:40:10 INFO reduce.InMemoryMapOutput: Read 1997 bytes from map-output for attempt_local1040123772_0001_m_000004_0
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1997, inMemoryMapOutputs.size() -> 4, commitMemory -> 40488, usedMemory ->42485
16/10/05 01:40:10 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1040123772_0001_m_000006_0 decomp: 1533 len: 1537 to MEMORY
16/10/05 01:40:10 WARN io.ReadaheadPool: Failed readahead on ifile
16/10/05 01:40:10 INFO reduce.InMemoryMapOutput: Read 1533 bytes from map-output for attempt_local1040123772_0001_m_000006_0
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1533, inMemoryMapOutputs.size() -> 5, commitMemory -> 42485, usedMemory ->44018
16/10/05 01:40:10 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1040123772_0001_m_000005_0 decomp: 1752 len: 1756 to MEMORY
16/10/05 01:40:10 INFO reduce.InMemoryMapOutput: Read 1752 bytes from map-output for attempt_local1040123772_0001_m_000005_0
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1752, inMemoryMapOutputs.size() -> 6, commitMemory -> 44018, usedMemory ->45770
16/10/05 01:40:10 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1040123772_0001_m_000002_0 decomp: 9717 len: 9721 to MEMORY
16/10/05 01:40:10 INFO reduce.InMemoryMapOutput: Read 9717 bytes from map-output for attempt_local1040123772_0001_m_000002_0
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 9717, inMemoryMapOutputs.size() -> 7, commitMemory -> 45770, usedMemory ->55487
16/10/05 01:40:10 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1040123772_0001_m_000007_0 decomp: 1258 len: 1262 to MEMORY
16/10/05 01:40:10 INFO reduce.InMemoryMapOutput: Read 1258 bytes from map-output for attempt_local1040123772_0001_m_000007_0
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1258, inMemoryMapOutputs.size() -> 8, commitMemory -> 55487, usedMemory ->56745
16/10/05 01:40:10 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/10/05 01:40:10 INFO mapred.LocalJobRunner: 8 / 8 copied.
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: finalMerge called with 8 in-memory map-outputs and 0 on-disk map-outputs
16/10/05 01:40:10 INFO mapred.Merger: Merging 8 sorted segments
16/10/05 01:40:10 INFO mapred.Merger: Down to the last merge-pass, with 8 segments left of total size: 56721 bytes
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: Merged 8 segments, 56745 bytes to disk to satisfy reduce memory limit
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: Merging 1 files, 56735 bytes from disk
16/10/05 01:40:10 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/10/05 01:40:10 INFO mapred.Merger: Merging 1 sorted segments
16/10/05 01:40:10 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 56728 bytes
16/10/05 01:40:10 INFO mapred.LocalJobRunner: 8 / 8 copied.
16/10/05 01:40:10 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
16/10/05 01:40:12 INFO mapred.Task: Task:attempt_local1040123772_0001_r_000000_0 is done. And is in the process of committing
16/10/05 01:40:12 INFO mapred.LocalJobRunner: 8 / 8 copied.
16/10/05 01:40:12 INFO mapred.Task: Task attempt_local1040123772_0001_r_000000_0 is allowed to commit now
16/10/05 01:40:12 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1040123772_0001_r_000000_0' to hdfs://hadoop1:9000/wordcount/output/_temporary/0/task_local1040123772_0001_r_000000
16/10/05 01:40:12 INFO mapred.LocalJobRunner: reduce > reduce
16/10/05 01:40:12 INFO mapred.Task: Task 'attempt_local1040123772_0001_r_000000_0' done.
16/10/05 01:40:12 INFO mapred.LocalJobRunner: Finishing task: attempt_local1040123772_0001_r_000000_0
16/10/05 01:40:12 INFO mapred.LocalJobRunner: reduce task executor complete.
16/10/05 01:40:12 INFO mapreduce.Job:  map 100% reduce 100%     // map  与reduce的进度
16/10/05 01:40:13 INFO mapreduce.Job: Job job_local1040123772_0001 completed successfully
16/10/05 01:40:14 INFO mapreduce.Job: Counters: 38
File System Counters
FILE: Number of bytes read=241244      // 总共读的字节数
FILE: Number of bytes written=2935620    // 总共输出的字节数
FILE: Number of read operations=0    // 读的操作 
FILE: Number of large read operations=0   // 大规模的读取的操作
FILE: Number of write operations=0
HDFS: Number of bytes read=196687    // HDFS读取的个数
HDFS: Number of bytes written=10550   // HDFS写入的个数
HDFS: Number of read operations=127
HDFS: Number of large read operations=0
HDFS: Number of write operations=11
Map-Reduce Framework
Map input records=773      // map  输入的记录
Map output records=5008    // reduce 输出的记录
Map output bytes=46713  
Map output materialized bytes=56777
Input split bytes=877
Combine input records=0
Combine output records=0
Reduce input groups=603
Reduce shuffle bytes=56777     // reduce的shuffer的字节数
Reduce input records=5008      // reduce 输入的记录
Reduce output records=603     // reduce输出的记录
Spilled Records=10016
Shuffled Maps =8                        // shuffer的map
Failed Shuffles=0
Merged Map outputs=8
GC time elapsed (ms)=540
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1370157056
Shuffle Errors          // 错误的计数器
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters     // input的计数器
Bytes Read=26697
File Output Format Counters   // 输出的计数器
Bytes Written=10550
盒马coding
关注
专栏目录
WordCount运行详解
wenyusuran的专栏
06-19 1386
1、MapReduce理论简介   1.1 MapReduce编程模型 　　MapReduce采用"分而治之"的思想，把对大规模数据集的操作，分发给一个主节点管理下的各个分节点共同完成，然后通过整合各个节点的中间结果，得到最终结果。简单地说，MapReduce就是"任务的分解与结果的汇总"。 　　在Hadoop中，用于执行MapReduce任务的机器角色有两个：一个是JobTracker；
MapReduce编程模式——WordCount程序详解
weixin_44319333的博客
04-24 583
首先介绍一下Hadoop的四大组件： - HDFS：分布式存储系统 - MapReduce：分布式计算系统 - YARN：hadoop的资源调度系统 - Common：以上三大组件的底层支撑组件，主要提供工具包和RPC框架等 MapReduce是一个分布式运算程序的编程框架，是用户开发“基于Hadoop的数据分析应用”的核心框架。 其核心功能是将用户编写的业务逻辑代码和自带默认组件整合成一个完整的...
linux环境下运行Eclipse用来测试WordCount示例
03-08
linux环境下，创建eclipse的快捷方式，并且在eclipse中运行示例，查看hdfs中的文件目录
linux 运行wordcount.jar
javaMylife的博客
12-13 242
hdplib=`find /opt/aps/hadoop-2.8.5/ -name "*.jar" | xargs | sed 's/ /:/g'` 定义变量，拼接classPath java -cp task.jar:${hdplib} com.ws.wordcount.WordCountMap arg0arg1arg2 [root@dream4 testhdp]# find /opt/apps/hadoop-2.8.5/ -name "*.jar" /opt/apps/hadoo...
Linux上运行WordCount单词统计程序
czy的博客
10-16 969
本次在Linux上运行WordCount程序，因为在Eclipse上运行会涉及到很多配置问题，在Linux上运行就方便很多，时候初学者。本次单词程序分为三个Java文件：主函数WordCount.java、Mapper类WCMapper.java、Reducer类WCReducer。新建一个package，下面放建立这三个class文件。 MapReduce实现的大致流程为： 主函数Wo...
在Linux环境实现wordcount：mapper，reducer的代码创建，脚本实现map，reduce
博樽
10-22 829
数据准备、map创建的初始、在map.py基础下创建red.py、map.py 重修版、用脚本run.sh 一步到位 执行map.py与red.py。
linux运行wordcount,Hadoop之本地运行WordCount
weixin_33859931的博客
05-11 276
本文主要记录在windows搭建Hadoop开发环境并编写一个WordCount的mapreduce在本地环境执行。主要内容：1.搭建本地环境2.编写WordCount并在本地运行1.搭建本地环境1.1.解压去官网下载指定的hadoop版本hadoop-2.7.3.tar.gz将下载好的hadoop压缩包解压到任意目录拷贝winutils.exe 到 hadoop-2.7.3/bin 目录下1.2...
Linux下编写运行自己的WordCount程序
彩虹柠檬水
04-29 2088
一、实践环境 Ubuntu14.04+JDK1.8.0_25 +Eclipse3.8+ Hadoop2.5.1 一共三台linux机器（virtualbox虚拟机，桥接网络配置静态ip），已经部署好的完全分布式环境。 二、编写WordCount程序 1.启动Eclipse，创建Java Project。 2.配置Java Project，这一步很重要，折腾了半天才配好
Hadoop集群-WordCount运行详解
09-02
在linux环境下部署的Hadoop集群-WordCount运行详解。
linux运行hadoop实例wordcount详解
DaiDaiHan1的专栏
02-24 1714
在服务器上配置了hadoop准备自己从wordcount.java开始到跑起wordcount程序，但是就这简单的几步遇到了很多问题，现在总结一下我碰到的问题： 整个过程步骤是： 1.首先创建文件目录： mkdir  Test 2.编译wordcount.java文件生成.class文件，存放在第一步创建的文件夹Test中： javac  -classpath /hadoop*core*
CentOS下用Eclipse和命令行方式编译运行WordCount程序
qq_41630866的博客
06-08 2370
安装Eclipse 一、下载Eclipse软件 二、安装Eclipse软件 三、安装Eclipse插件Hadoop-Eclipse-Plugin 四、配置Hadoop-Eclipse-Plugin WordCount程序 一、在eclipse中写个WordCount吧 1.建立项目 2.在项目里面建立WordCount类 3.写入代码 4.通过 Eclipse 运行 Map...
linux运行wordcount,Ubuntu16.04上运行Hadoop2.7.3自带example wordCount摸索记录
weixin_32394015的博客
05-11 110
首先最最重要的写在最前面，也是我觉得个人踩得最深的坑，刚接触hadoop的人，缺少的认识：hdfs的理解：它是一个文件系统，跟linux的文件系统是类似的结构，拥有类似的语法，大概就是你在linux上ls查看文件列表，那么hdfs的无非就是hadoop fs -ls。hadoop的输入输出，都是从hdfs读取和写入的，那么比如运行hadoop的word count例子的时候，网上各种大坑教程中完全...
Linux Ubuntu安装单机Hadoop并运行WordCount测试
飞花舞者的博客
03-23 738
1.安装java JDK 在终端root下，直接输入java,提示没有安装java的话，根据提示可以选择apt install default-jre那个，安装完后使用java -version出现版本提示信息则安装成功，此时的默认路径一般为 /usr/lib/jvm/default-java,记住这个路径，等会配置hadoop的时候要用。 2.安装Hadoop 下载Hadoop,这里使用的版本为2.7.3,下载hadoop-2.7.3.tar.gz，放在/home/bluecap/desktop/目录下，
linux云服务器实现wordcount案例
weixin_44999258的博客
01-05 1273
在hadoop安装目录下创建wcinput文件夹mkdir input 在input文件夹下创建文件touch wc.input 在 wc.input文件内添加文本内容vi wc.input hadoop yarn hadoop mapreduce atguigu atguigu 在hadoop安装目录下执行wordcount程序hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.1.jar wordcount wcinpu.
linux hadoop实例应用,Hadoop示例程序WordCount运行及详解
weixin_33764069的博客
05-13 510
最近在学习云计算，研究Haddop框架，费了一整天时间将Hadoop在Linux下完全运行起来，看到官方的map-reduce的demo程序WordCount，仔细研究了一下，算做入门了。运行方法：假设：• /home/cq/wordcount/input - 是 HDFS 中的输入路径• /home/cq/wordcount/output - 是 HDFS 中的输出路径用示例文本文件做为输入：$...
linux wc (Word Count)命令
qq_36740035的博客
08-23 550
Linux系统中的wc(Word Count)命令的功能为统计指定文件中的字节数、字数、行数，并将统计结果显示输出。       命令参数：     -c 统计字节数。     -l 统计行数。     -m 统计字符数。这个标志不能与 -c 标志一起使用。     -w 统计字数。一个字被定义为由空白、跳格或换行字符分隔的字符串。     -L 打印最长行的长度。
Linux下执行Hadoop WordCount.jar
红苹果乐园
09-11 3151
Linux执行 Hadoop WordCount ubuntu 终端进入快捷键 ：ctrl + Alt +t hadoop启动命令：start-all.sh   正常执行效果如下： hadoop@HADOOP:~$ start-all.sh Warning: $HADOOP_HOME is deprecated.   starting namenode, logging to /h
Linux下Hadoop分布式配置详解与实战
- **作业提交**：讲解了如何在客户端提交MapReduce任务，提供了一个WordCount示例，并说明了在不同模式下编译和运行应用的步骤。 7. **附录**：包含了具体的程序代码和总结，以及针对Hadoop配置的一些建议，如使用...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

盒马coding

你的支持是我最大的动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值