运行hadoop案例

最新推荐文章于 2024-04-25 19:00:29 发布
最新推荐文章于 2024-04-25 19:00:29 发布
阅读量2.5k 收藏 1
点赞数 1
分类专栏： 大数据 文章标签： hadoop helloword hadoop wordcount
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/chen18677338530/article/details/92830138
版权

运行Grep案例

  1. 创建input文件夹
    在这里插入图片描述
  2. 准备文件到input文件夹中
    在这里插入图片描述
  3. 执行share目录下的MapReduce程序
./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar  grep /data/tmp/input/ /data/tmp/output 'dfs[a-z.]+'

结果：

19/06/19 13:53:16 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
19/06/19 13:53:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
19/06/19 13:53:16 INFO input.FileInputFormat: Total input paths to process : 8
19/06/19 13:53:16 INFO mapreduce.JobSubmitter: number of splits:8
19/06/19 13:53:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local28590477_0001
19/06/19 13:53:16 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
19/06/19 13:53:16 INFO mapreduce.Job: Running job: job_local28590477_0001
19/06/19 13:53:16 INFO mapred.LocalJobRunner: OutputCommitter set in config null
19/06/19 13:53:16 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:16 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
19/06/19 13:53:16 INFO mapred.LocalJobRunner: Waiting for map tasks
19/06/19 13:53:16 INFO mapred.LocalJobRunner: Starting task: attempt_local28590477_0001_m_000000_0
19/06/19 13:53:16 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:16 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:16 INFO mapred.MapTask: Processing split: file:/data/tmp/input/hadoop-policy.xml:0+9683
19/06/19 13:53:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/19 13:53:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/19 13:53:17 INFO mapred.MapTask: soft limit at 83886080
19/06/19 13:53:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/19 13:53:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/06/19 13:53:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/06/19 13:53:17 INFO mapred.LocalJobRunner: 
19/06/19 13:53:17 INFO mapred.MapTask: Starting flush of map output
19/06/19 13:53:17 INFO mapred.MapTask: Spilling map output
19/06/19 13:53:17 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
19/06/19 13:53:17 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
19/06/19 13:53:17 INFO mapred.MapTask: Finished spill 0
19/06/19 13:53:17 INFO mapred.Task: Task:attempt_local28590477_0001_m_000000_0 is done. And is in the process of committing
19/06/19 13:53:17 INFO mapred.LocalJobRunner: map
19/06/19 13:53:17 INFO mapred.Task: Task 'attempt_local28590477_0001_m_000000_0' done.
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local28590477_0001_m_000000_0
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Starting task: attempt_local28590477_0001_m_000001_0
19/06/19 13:53:17 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:17 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:17 INFO mapred.MapTask: Processing split: file:/data/tmp/input/kms-site.xml:0+5511
19/06/19 13:53:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/19 13:53:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/19 13:53:17 INFO mapred.MapTask: soft limit at 83886080
19/06/19 13:53:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/19 13:53:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/06/19 13:53:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/06/19 13:53:17 INFO mapred.LocalJobRunner: 
19/06/19 13:53:17 INFO mapred.MapTask: Starting flush of map output
19/06/19 13:53:17 INFO mapred.Task: Task:attempt_local28590477_0001_m_000001_0 is done. And is in the process of committing
19/06/19 13:53:17 INFO mapred.LocalJobRunner: map
19/06/19 13:53:17 INFO mapred.Task: Task 'attempt_local28590477_0001_m_000001_0' done.
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local28590477_0001_m_000001_0
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Starting task: attempt_local28590477_0001_m_000002_0
19/06/19 13:53:17 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:17 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:17 INFO mapred.MapTask: Processing split: file:/data/tmp/input/capacity-scheduler.xml:0+4436
19/06/19 13:53:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/19 13:53:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/19 13:53:17 INFO mapred.MapTask: soft limit at 83886080
19/06/19 13:53:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/19 13:53:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/06/19 13:53:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/06/19 13:53:17 INFO mapred.LocalJobRunner: 
19/06/19 13:53:17 INFO mapred.MapTask: Starting flush of map output
19/06/19 13:53:17 INFO mapred.Task: Task:attempt_local28590477_0001_m_000002_0 is done. And is in the process of committing
19/06/19 13:53:17 INFO mapred.LocalJobRunner: map
19/06/19 13:53:17 INFO mapred.Task: Task 'attempt_local28590477_0001_m_000002_0' done.
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local28590477_0001_m_000002_0
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Starting task: attempt_local28590477_0001_m_000003_0
19/06/19 13:53:17 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:17 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:17 INFO mapred.MapTask: Processing split: file:/data/tmp/input/kms-acls.xml:0+3518
19/06/19 13:53:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/19 13:53:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/19 13:53:17 INFO mapred.MapTask: soft limit at 83886080
19/06/19 13:53:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/19 13:53:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/06/19 13:53:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/06/19 13:53:17 INFO mapred.LocalJobRunner: 
19/06/19 13:53:17 INFO mapred.MapTask: Starting flush of map output
19/06/19 13:53:17 INFO mapred.Task: Task:attempt_local28590477_0001_m_000003_0 is done. And is in the process of committing
19/06/19 13:53:17 INFO mapred.LocalJobRunner: map
19/06/19 13:53:17 INFO mapred.Task: Task 'attempt_local28590477_0001_m_000003_0' done.
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local28590477_0001_m_000003_0
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Starting task: attempt_local28590477_0001_m_000004_0
19/06/19 13:53:17 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:17 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:17 INFO mapred.MapTask: Processing split: file:/data/tmp/input/hdfs-site.xml:0+775
19/06/19 13:53:17 INFO mapreduce.Job: Job job_local28590477_0001 running in uber mode : false
19/06/19 13:53:17 INFO mapreduce.Job:  map 100% reduce 0%
19/06/19 13:53:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/19 13:53:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/19 13:53:17 INFO mapred.MapTask: soft limit at 83886080
19/06/19 13:53:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/19 13:53:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/06/19 13:53:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/06/19 13:53:17 INFO mapred.LocalJobRunner: 
19/06/19 13:53:17 INFO mapred.MapTask: Starting flush of map output
19/06/19 13:53:17 INFO mapred.Task: Task:attempt_local28590477_0001_m_000004_0 is done. And is in the process of committing
19/06/19 13:53:17 INFO mapred.LocalJobRunner: map
19/06/19 13:53:17 INFO mapred.Task: Task 'attempt_local28590477_0001_m_000004_0' done.
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local28590477_0001_m_000004_0
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Starting task: attempt_local28590477_0001_m_000005_0
19/06/19 13:53:17 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:17 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:17 INFO mapred.MapTask: Processing split: file:/data/tmp/input/core-site.xml:0+774
19/06/19 13:53:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/19 13:53:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/19 13:53:17 INFO mapred.MapTask: soft limit at 83886080
19/06/19 13:53:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/19 13:53:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/06/19 13:53:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/06/19 13:53:17 INFO mapred.LocalJobRunner: 
19/06/19 13:53:17 INFO mapred.MapTask: Starting flush of map output
19/06/19 13:53:17 INFO mapred.Task: Task:attempt_local28590477_0001_m_000005_0 is done. And is in the process of committing
19/06/19 13:53:17 INFO mapred.LocalJobRunner: map
19/06/19 13:53:17 INFO mapred.Task: Task 'attempt_local28590477_0001_m_000005_0' done.
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local28590477_0001_m_000005_0
19/06/19 13:53:17 INFO mapred.LocalJobRunner: Starting task: attempt_local28590477_0001_m_000006_0
19/06/19 13:53:17 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:17 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:17 INFO mapred.MapTask: Processing split: file:/data/tmp/input/yarn-site.xml:0+690
19/06/19 13:53:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/19 13:53:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/19 13:53:17 INFO mapred.MapTask: soft limit at 83886080
19/06/19 13:53:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/19 13:53:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/06/19 13:53:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/06/19 13:53:17 INFO mapred.LocalJobRunner: 
19/06/19 13:53:17 INFO mapred.MapTask: Starting flush of map output
19/06/19 13:53:17 INFO mapred.Task: Task:attempt_local28590477_0001_m_000006_0 is done. And is in the process of committing
19/06/19 13:53:18 INFO mapred.LocalJobRunner: map
19/06/19 13:53:18 INFO mapred.Task: Task 'attempt_local28590477_0001_m_000006_0' done.
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local28590477_0001_m_000006_0
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Starting task: attempt_local28590477_0001_m_000007_0
19/06/19 13:53:18 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:18 INFO mapred.MapTask: Processing split: file:/data/tmp/input/httpfs-site.xml:0+620
19/06/19 13:53:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/19 13:53:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/19 13:53:18 INFO mapred.MapTask: soft limit at 83886080
19/06/19 13:53:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/19 13:53:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/06/19 13:53:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/06/19 13:53:18 INFO mapred.LocalJobRunner: 
19/06/19 13:53:18 INFO mapred.MapTask: Starting flush of map output
19/06/19 13:53:18 INFO mapred.Task: Task:attempt_local28590477_0001_m_000007_0 is done. And is in the process of committing
19/06/19 13:53:18 INFO mapred.LocalJobRunner: map
19/06/19 13:53:18 INFO mapred.Task: Task 'attempt_local28590477_0001_m_000007_0' done.
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local28590477_0001_m_000007_0
19/06/19 13:53:18 INFO mapred.LocalJobRunner: map task executor complete.
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Waiting for reduce tasks
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Starting task: attempt_local28590477_0001_r_000000_0
19/06/19 13:53:18 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:18 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3d12eadd
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
19/06/19 13:53:18 INFO reduce.EventFetcher: attempt_local28590477_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
19/06/19 13:53:18 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local28590477_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
19/06/19 13:53:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local28590477_0001_m_000006_0
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
19/06/19 13:53:18 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local28590477_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
19/06/19 13:53:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local28590477_0001_m_000003_0
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
19/06/19 13:53:18 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local28590477_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
19/06/19 13:53:18 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local28590477_0001_m_000000_0
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->25
19/06/19 13:53:18 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local28590477_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
19/06/19 13:53:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local28590477_0001_m_000007_0
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 25, usedMemory ->27
19/06/19 13:53:18 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local28590477_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
19/06/19 13:53:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local28590477_0001_m_000004_0
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 27, usedMemory ->29
19/06/19 13:53:18 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/06/19 13:53:18 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local28590477_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
19/06/19 13:53:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local28590477_0001_m_000001_0
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 29, usedMemory ->31
19/06/19 13:53:18 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local28590477_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
19/06/19 13:53:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local28590477_0001_m_000005_0
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 31, usedMemory ->33
19/06/19 13:53:18 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local28590477_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
19/06/19 13:53:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local28590477_0001_m_000002_0
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->35
19/06/19 13:53:18 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
19/06/19 13:53:18 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/06/19 13:53:18 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/06/19 13:53:18 INFO mapred.LocalJobRunner: 8 / 8 copied.
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: finalMerge called with 8 in-memory map-outputs and 0 on-disk map-outputs
19/06/19 13:53:18 INFO mapred.Merger: Merging 8 sorted segments
19/06/19 13:53:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: Merged 8 segments, 35 bytes to disk to satisfy reduce memory limit
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
19/06/19 13:53:18 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
19/06/19 13:53:18 INFO mapred.Merger: Merging 1 sorted segments
19/06/19 13:53:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
19/06/19 13:53:18 INFO mapred.LocalJobRunner: 8 / 8 copied.
19/06/19 13:53:18 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
19/06/19 13:53:18 INFO mapred.Task: Task:attempt_local28590477_0001_r_000000_0 is done. And is in the process of committing
19/06/19 13:53:18 INFO mapred.LocalJobRunner: 8 / 8 copied.
19/06/19 13:53:18 INFO mapred.Task: Task attempt_local28590477_0001_r_000000_0 is allowed to commit now
19/06/19 13:53:18 INFO output.FileOutputCommitter: Saved output of task 'attempt_local28590477_0001_r_000000_0' to file:/data/app/hadoop-2.7.2/grep-temp-1290670728/_temporary/0/task_local28590477_0001_r_000000
19/06/19 13:53:18 INFO mapred.LocalJobRunner: reduce > reduce
19/06/19 13:53:18 INFO mapred.Task: Task 'attempt_local28590477_0001_r_000000_0' done.
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local28590477_0001_r_000000_0
19/06/19 13:53:18 INFO mapred.LocalJobRunner: reduce task executor complete.
19/06/19 13:53:18 INFO mapreduce.Job:  map 100% reduce 100%
19/06/19 13:53:18 INFO mapreduce.Job: Job job_local28590477_0001 completed successfully
19/06/19 13:53:18 INFO mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=2688230
		FILE: Number of bytes written=4974527
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=745
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=67
		Input split bytes=805
		Combine input records=1
		Combine output records=1
		Reduce input groups=1
		Reduce shuffle bytes=67
		Reduce input records=1
		Reduce output records=1
		Spilled Records=2
		Shuffled Maps =8
		Failed Shuffles=0
		Merged Map outputs=8
		GC time elapsed (ms)=487
		Total committed heap usage (bytes)=2748841984
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=26007
	File Output Format Counters 
		Bytes Written=123
19/06/19 13:53:18 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
19/06/19 13:53:18 INFO input.FileInputFormat: Total input paths to process : 1
19/06/19 13:53:18 INFO mapreduce.JobSubmitter: number of splits:1
19/06/19 13:53:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1171342348_0002
19/06/19 13:53:18 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
19/06/19 13:53:18 INFO mapreduce.Job: Running job: job_local1171342348_0002
19/06/19 13:53:18 INFO mapred.LocalJobRunner: OutputCommitter set in config null
19/06/19 13:53:18 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:18 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Waiting for map tasks
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1171342348_0002_m_000000_0
19/06/19 13:53:18 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:18 INFO mapred.MapTask: Processing split: file:/data/app/hadoop-2.7.2/grep-temp-1290670728/part-r-00000:0+111
19/06/19 13:53:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/06/19 13:53:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/06/19 13:53:18 INFO mapred.MapTask: soft limit at 83886080
19/06/19 13:53:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/06/19 13:53:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/06/19 13:53:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/06/19 13:53:18 INFO mapred.LocalJobRunner: 
19/06/19 13:53:18 INFO mapred.MapTask: Starting flush of map output
19/06/19 13:53:18 INFO mapred.MapTask: Spilling map output
19/06/19 13:53:18 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
19/06/19 13:53:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
19/06/19 13:53:18 INFO mapred.MapTask: Finished spill 0
19/06/19 13:53:18 INFO mapred.Task: Task:attempt_local1171342348_0002_m_000000_0 is done. And is in the process of committing
19/06/19 13:53:18 INFO mapred.LocalJobRunner: map
19/06/19 13:53:18 INFO mapred.Task: Task 'attempt_local1171342348_0002_m_000000_0' done.
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local1171342348_0002_m_000000_0
19/06/19 13:53:18 INFO mapred.LocalJobRunner: map task executor complete.
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Waiting for reduce tasks
19/06/19 13:53:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1171342348_0002_r_000000_0
19/06/19 13:53:18 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/06/19 13:53:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/06/19 13:53:18 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7a0d1c38
19/06/19 13:53:19 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
19/06/19 13:53:19 INFO reduce.EventFetcher: attempt_local1171342348_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
19/06/19 13:53:19 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local1171342348_0002_m_000000_0 decomp: 21 len: 25 to MEMORY
19/06/19 13:53:19 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1171342348_0002_m_000000_0
19/06/19 13:53:19 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21
19/06/19 13:53:19 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
19/06/19 13:53:19 INFO mapred.LocalJobRunner: 1 / 1 copied.
19/06/19 13:53:19 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
19/06/19 13:53:19 INFO mapred.Merger: Merging 1 sorted segments
19/06/19 13:53:19 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
19/06/19 13:53:19 INFO reduce.MergeManagerImpl: Merged 1 segments, 21 bytes to disk to satisfy reduce memory limit
19/06/19 13:53:19 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
19/06/19 13:53:19 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
19/06/19 13:53:19 INFO mapred.Merger: Merging 1 sorted segments
19/06/19 13:53:19 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
19/06/19 13:53:19 INFO mapred.LocalJobRunner: 1 / 1 copied.
19/06/19 13:53:19 INFO mapred.Task: Task:attempt_local1171342348_0002_r_000000_0 is done. And is in the process of committing
19/06/19 13:53:19 INFO mapred.LocalJobRunner: 1 / 1 copied.
19/06/19 13:53:19 INFO mapred.Task: Task attempt_local1171342348_0002_r_000000_0 is allowed to commit now
19/06/19 13:53:19 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1171342348_0002_r_000000_0' to file:/data/tmp/output/_temporary/0/task_local1171342348_0002_r_000000
19/06/19 13:53:19 INFO mapred.LocalJobRunner: reduce > reduce
19/06/19 13:53:19 INFO mapred.Task: Task 'attempt_local1171342348_0002_r_000000_0' done.
19/06/19 13:53:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local1171342348_0002_r_000000_0
19/06/19 13:53:19 INFO mapred.LocalJobRunner: reduce task executor complete.
19/06/19 13:53:19 INFO mapreduce.Job: Job job_local1171342348_0002 running in uber mode : false
19/06/19 13:53:19 INFO mapreduce.Job:  map 100% reduce 100%
19/06/19 13:53:19 INFO mapreduce.Job: Job job_local1171342348_0002 completed successfully
19/06/19 13:53:19 INFO mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=1157660
		FILE: Number of bytes written=2213150
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=1
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=25
		Input split bytes=126
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=25
		Reduce input records=1
		Reduce output records=1
		Spilled Records=2
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=493879296
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=123
	File Output Format Counters 
		Bytes Written=23
  1. 查看执行结果
cat /data/tmp/output/*

在这里插入图片描述

运行WordCount案例

  1. 新建wcinout文件夹
    在这里插入图片描述
  2. 新建wc.input文件，并写入数据
    在这里插入图片描述
  3. 执行命令
hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /data/tmp/wcinput/ /data/tmp/wcoutput
  1. 查看结果
cat /data/tmp/wcoutput/*

在这里插入图片描述

chen18677338530
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
hadoop常见错误整理
zyj8170的专栏
11-27 1884
版本信息： hadoop 2.3.0 hive 0.11.0 1. Application Master 无法访问 点击application mater 链接，出现 http 500 错误，java.lang.Connect.exception： 问题是由于设定web ui时，50030 端口对应的ip地址为0.0.0.0，导致application master 链接无法定位。 解决办法
hadoop的运行案例
03-26
我的博客 Linux环境下，Eclipse运行hadoop的本地模式 ：https://blog.csdn.net/Henry_Lin_Wind/article/details/88812421 的运行案例
hadoop的平台搭建和运行
最新发布
2401_83263392的博客
04-25 1816
rw-------. 1 hadoop hadoop 395 11月 14 16:18 authorized_keys。-rw-------. 1 hadoop hadoop 395 11月 14 16:18 authorized_keys。-rw-------. 1 hadoop hadoop 395 11月 14 16:19 authorized_keys。-rw-r--r--. 1 hadoop hadoop 33 11月 10 23:50 part-r-00000。
Hadoop实例运行
Simon的博客
05-28 639
1、要使用 HDFS，首先需要在 HDFS 中创建用户目录，也可以使用默认根目录： zhaojunyandeMacBook-Pro:~ zhaojunyan$ hdfs dfs -mkdir -p /usr/hadoop 默认根目录是hdfs://localhost:9000/user/zhaojunyan 2、创建input目录，并拷贝文件 zhaojunyandeMacBook-Pro
day_01小笔记----本地运行Hadoop 案例、伪分布式运行Hadoop 案例
web15285868498的博客
03-31 1221
0、hadoop1.x和hadoop2.x区别 Hadoop1.x的核心组件 与Hadoop2.x核心组件不一样 Hadoop1.x：hdfs、common、mapreduce（mapreduce同时处理业务逻辑 运算和资源调度、耦合性较大） Hadoop2.x：hdfs、yarn、common、mapreduce（增加yarn、yarn只负责资源的调度、mapreduce只负责运算） 1、概念理解 HDFS组成：namenode（存储文件的元数据）、datanode（本地储存数据）、secondary n
hadoop运行案例
weixin_30312557的博客
04-09 96
运行一个example [hadoop@hadoop01 hadoop]$ hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 5 10 运行的过程中报错 然后去查看jobhistory 发现以下错误 Error: java.lang.RuntimeException: ...
大数据-hadoop2.6运行例子1
杨文的专栏（南京Java讲师）
05-06 253
本人刚刚录制的hadoop视频（在本人QQ空间上有）
Hadoop Yarn 案例 2
07-16
本文将围绕"Hadoop Yarn案例2"这一主题，深入探讨YARN在实际应用中的工作原理和优势，以及如何利用它来实现好友推荐系统。 好友推荐系统是社交网络服务中常见的一种功能，其背后通常涉及到大量的数据处理和分析。在...
Hadoop中单词统计案例运行的代码
03-02
在大数据处理领域，Hadoop是一个不可或缺的开源框架，它主要用于分布式存储和计算。...通过这个案例，我们可以学习到如何在Hadoop环境中编写和运行MapReduce作业，从而更好地理解和应用大数据处理技术。
浅谈七种常见的Hadoop和Spark项目案例
09-30
大型组织在运行多个Hadoop集群后，可能会寻求将其集中管理，形成Hadoop即服务（HaaS）。这可以优化资源利用率，减少运维成本。Docker容器化技术是实现这一目标的常见手段，尽管内部安全和政治因素可能构成挑战。一些...
hadoop-demo
11-30
通过分析和运行这些示例，学习者可以理解如何配置Hadoop环境，如何提交作业，以及如何处理分布式计算中的错误和异常。 至于文件名"hadoopSecond"，这可能是一个目录或文件，表示这是第二个Hadoop相关的练习或者项目...
本地运行Hadoop的官方案例：Grep与WordCount
凯撒袁六兽的blog
09-18 582
这里的Hadoop版本Hadoop2.7.2，基于Windows端中的Linux里的Hadoop运行，而Hadoop2.7.2的官方文档：https://hadoop.apache.org/docs/r2.7.2/
本地文件运行Hadoop 案例-02
kxj19980524的博客
04-01 517
hadoop运行环境搭建-01https://blog.csdn.net/kxj19980524/article/details/88934339 grep案例 grep案例就是执行一个mapreduce程序,从一堆文件里面找出符合那个正则的单词,输出到一个文件夹里 在hadoop目录下创建input文件夹 把hadoop下面的所有xml文件复制到自己创建的input目录下,这些文件做...
Hadoop集群运行简单实例
欢迎你的到来，我们一起加油！！！
04-27 1239
集群配置信息 请看这篇文章Hadoop3.3.0完全分布式集群部署 这里我们简单在Hadoop运行一下实例。 在一开始运行时，就遇到了一个报错信息. hadoop 找不到或无法加载org.apache.hadoop.mapreduce.v2.app.MRAppMaster 解决方案：我们到mapred-site.xml 配置文件添加配置 <property> <name>mapreduce.admin.user.env</name> <valu
Hadoop（三、hadoop运行模式与案例）
wise_nut的博客
06-14 474
1、Hadoop运行模式 Hadoop运行模式包括：本地模式、伪分布式模式以及完全分布式模式。 Hadoop官方网站： 链接: link. hadoop工程模块、相关工程包括哪些？如下图 hadoop 生态中，相关工程中：Hbase、Hive、spark、zookeeper是必学的 ...
Hadoop案例实践
weixin_34143774的博客
12-04 115
2019独角兽企业重金招聘Python工程师标准>>> ...
大数据实训笔记2：hadoop完全分布式模式
Roslin_v的博客
07-01 630
介绍了hadoop的完全分布式模式，重点是三台客户机的集群配置与启动，并介绍了一个简单案例。
Hadoop入门小例子WordCount
Lisam Blog
09-14 664
参考自书籍《Hadoop+Spark 大数据巨量分析与机器学习》 1 编写测试程序例子 import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Te...
Ubuntu系统下的Hadoop集群（3）_Hadoop单机版自定义实现类以及编译运行
wang_zhenwei的博客
08-10 2152
Hadoop 2.4.1单机版 自定义实现类以及编译运行 概述 博主最近在学hadoop，而且在本实验室一位大神的指导下，我已配置好hadoop2.4.1开发环境，还没有配置或者不会配置的，请看链接hadoop单机版配置。由于之前运行的都是hadoop自带的实例，但是对于个人学习而言，肯定是要自己编写实现类以及编译运行实现类，因此博主就撰写了这篇文章，希望对学习hadoop的同道中
wordcount案例Hadoop Linux
06-11
好的，您的问题是关于WordCount案例在Hadoop和Linux环境下的实现。 WordCount案例是Hadoop中的经典案例之一，它的作用是统计文本中单词的出现次数。在Linux环境下，我们可以使用grep、awk、sed等命令来实现这个功能...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值