Hadoop-2.9.2不同模式下hadoop-mapreduce-examples-2.9.2.jar跑的日志

最新推荐文章于 2022-08-13 11:37:01 发布
最新推荐文章于 2022-08-13 11:37:01 发布
阅读量891 收藏
点赞数 1
分类专栏： Hadoop 文章标签： Hadoop Yarn HDFS
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/asran1/article/details/89021641
版权

Standalone

$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar grep input output 'dfs[a-z.]+'
19/04/04 14:04:20 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
19/04/04 14:04:20 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
19/04/04 14:04:20 INFO input.FileInputFormat: Total input files to process : 8
19/04/04 14:04:20 INFO mapreduce.JobSubmitter: number of splits:8
19/04/04 14:04:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local931706435_0001
19/04/04 14:04:20 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
19/04/04 14:04:20 INFO mapreduce.Job: Running job: job_local931706435_0001
19/04/04 14:04:20 INFO mapred.LocalJobRunner: OutputCommitter set in config null
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Waiting for map tasks
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Starting task: attempt_local931706435_0001_m_000000_0
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:20 INFO mapred.MapTask: Processing split: file:/home/admin/ws/hadoop-2.9.2/input/hadoop-policy.xml:0+10206
19/04/04 14:04:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/04/04 14:04:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/04/04 14:04:20 INFO mapred.MapTask: soft limit at 83886080
19/04/04 14:04:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/04/04 14:04:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/04/04 14:04:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/04/04 14:04:20 INFO mapred.LocalJobRunner: 
19/04/04 14:04:20 INFO mapred.MapTask: Starting flush of map output
19/04/04 14:04:20 INFO mapred.MapTask: Spilling map output
19/04/04 14:04:20 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
19/04/04 14:04:20 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
19/04/04 14:04:20 INFO mapred.MapTask: Finished spill 0
19/04/04 14:04:20 INFO mapred.Task: Task:attempt_local931706435_0001_m_000000_0 is done. And is in the process of committing
19/04/04 14:04:20 INFO mapred.LocalJobRunner: map
19/04/04 14:04:20 INFO mapred.Task: Task 'attempt_local931706435_0001_m_000000_0' done.
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local931706435_0001_m_000000_0
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Starting task: attempt_local931706435_0001_m_000001_0
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:20 INFO mapred.MapTask: Processing split: file:/home/admin/ws/hadoop-2.9.2/input/capacity-scheduler.xml:0+7861
19/04/04 14:04:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/04/04 14:04:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/04/04 14:04:20 INFO mapred.MapTask: soft limit at 83886080
19/04/04 14:04:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/04/04 14:04:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/04/04 14:04:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/04/04 14:04:20 INFO mapred.LocalJobRunner: 
19/04/04 14:04:20 INFO mapred.MapTask: Starting flush of map output
19/04/04 14:04:20 INFO mapred.Task: Task:attempt_local931706435_0001_m_000001_0 is done. And is in the process of committing
19/04/04 14:04:20 INFO mapred.LocalJobRunner: map
19/04/04 14:04:20 INFO mapred.Task: Task 'attempt_local931706435_0001_m_000001_0' done.
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local931706435_0001_m_000001_0
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Starting task: attempt_local931706435_0001_m_000002_0
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:20 INFO mapred.MapTask: Processing split: file:/home/admin/ws/hadoop-2.9.2/input/kms-site.xml:0+5939
19/04/04 14:04:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/04/04 14:04:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/04/04 14:04:20 INFO mapred.MapTask: soft limit at 83886080
19/04/04 14:04:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/04/04 14:04:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/04/04 14:04:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/04/04 14:04:20 INFO mapred.LocalJobRunner: 
19/04/04 14:04:20 INFO mapred.MapTask: Starting flush of map output
19/04/04 14:04:20 INFO mapred.Task: Task:attempt_local931706435_0001_m_000002_0 is done. And is in the process of committing
19/04/04 14:04:20 INFO mapred.LocalJobRunner: map
19/04/04 14:04:20 INFO mapred.Task: Task 'attempt_local931706435_0001_m_000002_0' done.
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local931706435_0001_m_000002_0
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Starting task: attempt_local931706435_0001_m_000003_0
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:20 INFO mapred.MapTask: Processing split: file:/home/admin/ws/hadoop-2.9.2/input/kms-acls.xml:0+3518
19/04/04 14:04:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/04/04 14:04:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/04/04 14:04:20 INFO mapred.MapTask: soft limit at 83886080
19/04/04 14:04:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/04/04 14:04:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/04/04 14:04:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/04/04 14:04:20 INFO mapred.LocalJobRunner: 
19/04/04 14:04:20 INFO mapred.MapTask: Starting flush of map output
19/04/04 14:04:20 INFO mapred.Task: Task:attempt_local931706435_0001_m_000003_0 is done. And is in the process of committing
19/04/04 14:04:20 INFO mapred.LocalJobRunner: map
19/04/04 14:04:20 INFO mapred.Task: Task 'attempt_local931706435_0001_m_000003_0' done.
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local931706435_0001_m_000003_0
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Starting task: attempt_local931706435_0001_m_000004_0
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:20 INFO mapred.MapTask: Processing split: file:/home/admin/ws/hadoop-2.9.2/input/hdfs-site.xml:0+775
19/04/04 14:04:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/04/04 14:04:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/04/04 14:04:20 INFO mapred.MapTask: soft limit at 83886080
19/04/04 14:04:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/04/04 14:04:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/04/04 14:04:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/04/04 14:04:20 INFO mapred.LocalJobRunner: 
19/04/04 14:04:20 INFO mapred.MapTask: Starting flush of map output
19/04/04 14:04:20 INFO mapred.Task: Task:attempt_local931706435_0001_m_000004_0 is done. And is in the process of committing
19/04/04 14:04:20 INFO mapred.LocalJobRunner: map
19/04/04 14:04:20 INFO mapred.Task: Task 'attempt_local931706435_0001_m_000004_0' done.
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local931706435_0001_m_000004_0
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Starting task: attempt_local931706435_0001_m_000005_0
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:20 INFO mapred.MapTask: Processing split: file:/home/admin/ws/hadoop-2.9.2/input/core-site.xml:0+774
19/04/04 14:04:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/04/04 14:04:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/04/04 14:04:20 INFO mapred.MapTask: soft limit at 83886080
19/04/04 14:04:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/04/04 14:04:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/04/04 14:04:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/04/04 14:04:20 INFO mapred.LocalJobRunner: 
19/04/04 14:04:20 INFO mapred.MapTask: Starting flush of map output
19/04/04 14:04:20 INFO mapred.Task: Task:attempt_local931706435_0001_m_000005_0 is done. And is in the process of committing
19/04/04 14:04:20 INFO mapred.LocalJobRunner: map
19/04/04 14:04:20 INFO mapred.Task: Task 'attempt_local931706435_0001_m_000005_0' done.
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local931706435_0001_m_000005_0
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Starting task: attempt_local931706435_0001_m_000006_0
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:20 INFO mapred.MapTask: Processing split: file:/home/admin/ws/hadoop-2.9.2/input/yarn-site.xml:0+690
19/04/04 14:04:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/04/04 14:04:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/04/04 14:04:20 INFO mapred.MapTask: soft limit at 83886080
19/04/04 14:04:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/04/04 14:04:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/04/04 14:04:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/04/04 14:04:20 INFO mapred.LocalJobRunner: 
19/04/04 14:04:20 INFO mapred.MapTask: Starting flush of map output
19/04/04 14:04:20 INFO mapred.Task: Task:attempt_local931706435_0001_m_000006_0 is done. And is in the process of committing
19/04/04 14:04:20 INFO mapred.LocalJobRunner: map
19/04/04 14:04:20 INFO mapred.Task: Task 'attempt_local931706435_0001_m_000006_0' done.
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local931706435_0001_m_000006_0
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Starting task: attempt_local931706435_0001_m_000007_0
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:20 INFO mapred.MapTask: Processing split: file:/home/admin/ws/hadoop-2.9.2/input/httpfs-site.xml:0+620
19/04/04 14:04:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/04/04 14:04:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/04/04 14:04:20 INFO mapred.MapTask: soft limit at 83886080
19/04/04 14:04:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/04/04 14:04:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/04/04 14:04:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/04/04 14:04:20 INFO mapred.LocalJobRunner: 
19/04/04 14:04:20 INFO mapred.MapTask: Starting flush of map output
19/04/04 14:04:20 INFO mapred.Task: Task:attempt_local931706435_0001_m_000007_0 is done. And is in the process of committing
19/04/04 14:04:20 INFO mapred.LocalJobRunner: map
19/04/04 14:04:20 INFO mapred.Task: Task 'attempt_local931706435_0001_m_000007_0' done.
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local931706435_0001_m_000007_0
19/04/04 14:04:20 INFO mapred.LocalJobRunner: map task executor complete.
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Waiting for reduce tasks
19/04/04 14:04:20 INFO mapred.LocalJobRunner: Starting task: attempt_local931706435_0001_r_000000_0
19/04/04 14:04:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:20 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:20 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7b4297cf
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=364432576, maxSingleShuffleLimit=91108144, mergeThreshold=240525504, ioSortFactor=10, memToMemMergeOutputsThreshold=10
19/04/04 14:04:20 INFO reduce.EventFetcher: attempt_local931706435_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
19/04/04 14:04:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local931706435_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
19/04/04 14:04:20 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local931706435_0001_m_000000_0
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21
19/04/04 14:04:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local931706435_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
19/04/04 14:04:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local931706435_0001_m_000003_0
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 21, usedMemory ->23
19/04/04 14:04:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local931706435_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
19/04/04 14:04:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local931706435_0001_m_000006_0
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 23, usedMemory ->25
19/04/04 14:04:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local931706435_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
19/04/04 14:04:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local931706435_0001_m_000002_0
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 25, usedMemory ->27
19/04/04 14:04:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local931706435_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
19/04/04 14:04:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local931706435_0001_m_000005_0
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 27, usedMemory ->29
19/04/04 14:04:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local931706435_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
19/04/04 14:04:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local931706435_0001_m_000001_0
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 29, usedMemory ->31
19/04/04 14:04:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local931706435_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
19/04/04 14:04:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local931706435_0001_m_000004_0
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 31, usedMemory ->33
19/04/04 14:04:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local931706435_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
19/04/04 14:04:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local931706435_0001_m_000007_0
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->35
19/04/04 14:04:20 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
19/04/04 14:04:20 INFO mapred.LocalJobRunner: 8 / 8 copied.
19/04/04 14:04:20 INFO reduce.MergeManagerImpl: finalMerge called with 8 in-memory map-outputs and 0 on-disk map-outputs
19/04/04 14:04:20 INFO mapred.Merger: Merging 8 sorted segments
19/04/04 14:04:20 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
19/04/04 14:04:21 INFO reduce.MergeManagerImpl: Merged 8 segments, 35 bytes to disk to satisfy reduce memory limit
19/04/04 14:04:21 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
19/04/04 14:04:21 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
19/04/04 14:04:21 INFO mapred.Merger: Merging 1 sorted segments
19/04/04 14:04:21 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
19/04/04 14:04:21 INFO mapred.LocalJobRunner: 8 / 8 copied.
19/04/04 14:04:21 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
19/04/04 14:04:21 INFO mapred.Task: Task:attempt_local931706435_0001_r_000000_0 is done. And is in the process of committing
19/04/04 14:04:21 INFO mapred.LocalJobRunner: 8 / 8 copied.
19/04/04 14:04:21 INFO mapred.Task: Task attempt_local931706435_0001_r_000000_0 is allowed to commit now
19/04/04 14:04:21 INFO output.FileOutputCommitter: Saved output of task 'attempt_local931706435_0001_r_000000_0' to file:/home/admin/ws/hadoop-2.9.2/grep-temp-2108165286/_temporary/0/task_local931706435_0001_r_000000
19/04/04 14:04:21 INFO mapred.LocalJobRunner: reduce > reduce
19/04/04 14:04:21 INFO mapred.Task: Task 'attempt_local931706435_0001_r_000000_0' done.
19/04/04 14:04:21 INFO mapred.LocalJobRunner: Finishing task: attempt_local931706435_0001_r_000000_0
19/04/04 14:04:21 INFO mapred.LocalJobRunner: reduce task executor complete.
19/04/04 14:04:21 INFO mapreduce.Job: Job job_local931706435_0001 running in uber mode : false
19/04/04 14:04:21 INFO mapreduce.Job:  map 100% reduce 100%
19/04/04 14:04:21 INFO mapreduce.Job: Job job_local931706435_0001 completed successfully
19/04/04 14:04:21 INFO mapreduce.Job: Counters: 30
        File System Counters
                FILE: Number of bytes read=2997505
                FILE: Number of bytes written=6940406
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
        Map-Reduce Framework
                Map input records=840
                Map output records=1
                Map output bytes=17
                Map output materialized bytes=67
                Input split bytes=949
                Combine input records=1
                Combine output records=1
                Reduce input groups=1
                Reduce shuffle bytes=67
                Reduce input records=1
                Reduce output records=1
                Spilled Records=2
                Shuffled Maps =8
                Failed Shuffles=0
                Merged Map outputs=8
                GC time elapsed (ms)=26
                Total committed heap usage (bytes)=4343201792
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=30383
        File Output Format Counters 
                Bytes Written=123
19/04/04 14:04:21 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
19/04/04 14:04:21 INFO input.FileInputFormat: Total input files to process : 1
19/04/04 14:04:21 INFO mapreduce.JobSubmitter: number of splits:1
19/04/04 14:04:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1791972909_0002
19/04/04 14:04:21 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
19/04/04 14:04:21 INFO mapred.LocalJobRunner: OutputCommitter set in config null
19/04/04 14:04:21 INFO mapreduce.Job: Running job: job_local1791972909_0002
19/04/04 14:04:21 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:21 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:21 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
19/04/04 14:04:21 INFO mapred.LocalJobRunner: Waiting for map tasks
19/04/04 14:04:21 INFO mapred.LocalJobRunner: Starting task: attempt_local1791972909_0002_m_000000_0
19/04/04 14:04:21 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/04/04 14:04:21 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
19/04/04 14:04:21 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/04/04 14:04:21 INFO mapred.MapTask: Processing split: file:/home/admin/ws/hadoop-2.9.2/grep-temp-2108165286/part-r-00000:0+111
19/04/04 14:04:21 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/04/04 14:04:21 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/04/04 14:04:21 INFO mapred.MapTask: soft limit at 83886080
19/04/04 14:04:21 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/04/04 14:04:21 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/04/04 14:04:21 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/04/04 14:04:21 INFO mapred.LocalJobRunner: 
19/04/04 14:04:21 INFO mapred.MapTask: Starting flush of map output
19/04/04 14:04:21 INFO mapred.MapTask: Spilling map output
19/04/04 14:04:21 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
19/04/04 14:04:21 INFO mapred.MapTask: kvstart = 26214

最低0.47元/天 解锁文章
AsranD
关注
  • 1
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
hadoop之hadoop-mapreduce-examples-2.7.0.jar
yibei8811的专栏
07-06 1万+
前2篇blog中测试hadoop代码的时候都用到了这个jar，那么很有必要去分析一下源码。 分析源码之前很有必要先写一个wordcount，代码如下 package mytest; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration;
hadoop-mapreduce-examples-2.6.5.jar
10-23
hadoop-mapreduce-examples-2.6.5.jar 官方案例源码
hadoop-mapreduce-examples-2.7.1-sources.jar
08-14
hadoop中的demo，wordcount列子用到的JAR包 用法： # 在容器里运行WordCount程序，该程序需要2个参数：HDFS输入目录和HDFS输出目录 hadoop jar hadoop-mapreduce-examples-2.7.1-sources.jar org.apache.hadoop.examples.WordCount input output
hadoop-mapreduce-examples-2.7.1.jar
06-03
hadoop-mapreduce-examples-2.7.1.jar
使用hadoop-mapreduce-examples-3.2.2.jar来计算文件中的不同字符个数
weixin_43719616的博客
05-07 948
学习hadoop第一天，好牛呀！！！其实还是自己菜 1. mkdir wcinput cd /wcinput 3.新建一个测试文件 vim word.txt 4.在这个文件中随便写点测试数据，待会就会输出这个文件里面每个字符各有多少个 5.返回上一级 cd .. bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar wordcount wcinput/ ./wcoutput 7.然后进入
Hadoop下载 hadoop-2.9.2.tar.gz
07-06
Hadoop 采用 MapReduce 分布式计算框架，根据 GFS 原理开发了 HDFS（分布式文件系统），并根据 BigTable 原理开发了 HBase 数据存储系统。 Hadoop 和 Google 内部使用的分布式计算系统原理相同，其开源特性使其成为...
hadoop-eclipse-plugin-2.9.2
12-20
适用于Eclipse IDE（4.10.0）版本+jdk1.8版本，这个真的跟eclipse本身有关，我自己编译打的包，本来用在eclipse mars2上，结果一直不显示，没办法，只能重新下载eclipse然后放到dropins目录下就显示了
hadoop-eclipse-plugin-2.9.2.jar
04-04
hadoop-eclipse-plugin-2.9.2.jar。hadoop的eclipse插件，适用于hadoop2.9.2，自己编译的，亲测可用
MapReduce
JAX_fire的博客
08-13 748
分布式计算是一种计算方法和集中式计算是相对的随着计算技术的发展，有些应用需要非常巨大的计算能力才能完成，如果采用集中式计算，需要耗费相当长的时间来完成。分布式计算将该应用分解成许多小的部分，分配给多台计算机进行处理。这样可以节约整体计算时间，大大提高计算效率。Hadoop MapReduce是一个分布式计算框架，用于青松编写分布式应用程序，这些应用程序以可靠，容错的方式并行处理大型硬件集群（数千个节点）上的大量数据（多TB数据集）。...
hadoop-mapreduce-examples-2.6.0
04-20
Hadoop实现了一个分布式文件系统（Hadoop Distributed File System），简称HDFS。HDFS有高容错性的特点，并且设计用来部署在低廉的（low-cost）硬件上；而且它提供高吞吐量（high throughput）来访问应用程序的数据，适合那些有着超大数据集（large data set）的应用程序。HDFS放宽了（relax）POSIX的要求，可以以流的形式访问（streaming access）文件系统中的数据。
Linux执行hdfs上的jar包,如何运行Hadoop自带的Wordcount案例
weixin_39851809的博客
05-14 678
1,在运行这个案例之前,首先启动你的Hadoop集群,我的Hadoop集群搭建的是完全分布式,三台机器,保证都启动起来2,进入Hadoop的安装目录3,重点来了,如果你不知道这个examples这个jar包在哪里的情况下,可以在hadoop的安装目录执行find ./ -name example.jarimage.png4,上传相关文件到hdfs上面hdfs dfs -put /home/a.t...
执行 bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /user/input出错
simply
05-26 1万+
[root@node1 hadoop-2.7.3]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /user/input /user/output 19/05/26 09:44:33 INFO client.RMProxy: Connecting to ResourceMan...
【图文解析 】MapReduce 程序运行演示 ——hadoopd第二核心组建！
谦卑t
11-14 294
在 MapReduce 组件里，官方给我们提供了一些样例程序，其中非常有名的就是 wordcount 和 pi 程序。这些 MapReduce 程序的代码都在 hadoop-mapreduce-examples-2.6.4.jar 包里，这 个 jar 包在 hadoop 安装目录下的/share/hadoop/mapreduce/目录里    下面我们使用 hadoop 命令来试跑例子程序，看看...
bin/hadoop jar hadoop-*-examples.jar wordcount 路径错误。
谭运华的专栏
06-12 8303
初学hadoop，准备在linux上单机运行wordcount例子，结果初学错误。 根据例子第一次运行hadoop，运行命令“bin/hadoop jar hadoop-*-examples.jar wordcount”，结果发生了异常，如下：        Exception in thread "main" java.io.IOException: Error opening job
MapReduce简介
intmiss的博客
10-29 916
MapReduce MapReduce是一种编程模型，用于大规模数据集（大于1TB）的并行运算。概念"Map（映射）“和"Reduce（归约）”，是它们的主要思想，都是从函数式编程语言里借来的，还有从矢量编程语言里借来的特性。它极大地方便了编程人员在不会分布式并行编程的情况下，将自己的程序运行在分布式系统上。 当前的软件实现是指定一个Map（映射）函数，用来把一组键值对映射成一组新的键值对，指定并...
hadoop个人学习笔记
weixin_41627634的博客
04-18 223
hadoop安装 一、单个namenode集群搭建 1、简单概念 机器 node1 node2 node3 node4 uri概念–统一资源定位符 hdfs://node1:9000 uri field/// jdbc:mysql://localhost:3306 安装jdk JAVA_HOME=/usr/local/jdk1.7.0_45/ 2、解压文件 tar -zxvf hadoop-2.6.4.tar.gz -C /apps ln -s /apps/hadoop-2.6.4 hadoop 修改配置文
Hadoop之hadoop-mapreduce-examples测试执行及报错处理
热门推荐
月生的静心苑
12-16 1万+
Hadoop Map/Reduce是一个使用简易的软件框架，基于它写出来的应用程序能够运行在由上千个商用机器组成的大型集群上，并以一种可靠容错的方式并行处理上T级别的数据集。
【02】Hadoop简介及安装
忻辰的博客
04-14 1082
Hadoop简介及安装 一、认识Hadoop 1.Hadoop简介（Hadoop是什么）？ Hadoop 是一个开源的，可靠的、可扩展的分布式存储和计算框架，它基于 Java 语言开发，具有很好的跨平台特性，它允许用户使用简单的编程模型在廉价的计算机集群上对大规模数据集进行分布式处理。 2.Hadoop特点 1.高可靠性。 采用冗余数据存储方式，即使一个副本发生故障，其它副本也可以保证正常对外提供服务。 2.高可扩展性。 Hadoop 集群可以很方便地添加机器节点，从而提高整体数据处理能力。 3. 高性能。
hadoop3.2.1集群搭建
fanghailiang2016的博客
05-27 493
hadoop目前最新版本为3.2.1，本教程将使用hadoop最新版本搭建大数据集群，并通过yarn运行wordcount的mapreduce程序 软件和服务器准备工作 软件准备 hadoop 3,2,1。 下载地址点击下载 java8。 下载地址点击下载（经本人测试jdk最新版本 java14 运行hadoop集群会报错，所以并不是jdk版本越高越好） 服务器准备 hadoop服务器 ip（主机名） 外网ip 功能 172.19.86.64（host01） 139....
ln -s hadoop-2.9.2 hadoop
最新发布
04-26
ln -s是Linux系统中的一个命令，用于创建软链接（symbolic link）。软链接是一个指向另一个...所以，执行"ln -s hadoop-2.9.2 hadoop"命令后，会在当前目录下创建一个名为hadoop的软链接，指向hadoop-2.9.2文件或目录。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值