单机版部署Hadoop
- 实验环境:
在server2上部署 - 步骤一:创建普通用户,解压软件包,创建软链接,编辑hadoop环境变量文件
- 步骤二:运行hadoop,进行测试
[bdkl@server2 bin]$ ./hadoop
Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
or hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]
where CLASSNAME is a user-provided Java class
OPTIONS is none or any of:
buildpaths attempt to add class files from build tree
--config dir Hadoop config directory
--debug turn on shell script debug mode
--help usage information
hostnames list[,of,host,names] hosts to use in slave mode
hosts filename list of hosts to use in slave mode
loglevel level set the log4j level for this command
workers turn on worker mode
SUBCOMMAND is one of:
Admin Commands:
daemonlog get/set the log level for each daemon
Client Commands:
archive create a Hadoop archive
checknative check native Hadoop and compression libraries availability
classpath prints the class path needed to get the Hadoop jar and the required libraries
conftest validate configuration XML files
credential interact with credential providers
distch distributed metadata changer
distcp copy file or directories recursively
dtutil operations related to delegation tokens
envvars display computed Hadoop environment variables
fs run a generic filesystem user client
gridmix submit a mix of synthetic job, modeling a profiled from production load
jar <jar> run a jar file. NOTE: please use "yarn jar" to launch YARN applications, not this command.
jnipath prints the java.library.path
kdiag Diagnose Kerberos Problems
kerbname show auth_to_local principal conversion
key manage keys via the KeyProvider
rumenfolder scale a rumen input trace
rumentrace convert logs into a rumen trace
s3guard manage metadata on S3
trace view and modify Hadoop tracing settings
version print the version
Daemon Commands:
kms run KMS, the Key Management Server
SUBCOMMAND may print help when invoked w/o parameters or with -h.
- 步骤三:进行测试
#进行运算,output目录必须不存在
[bdkl@server2 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar grep input output 'dfs[a-z.]+'
2021-01-06 00:27:56,455 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2021-01-06 00:27:56,721 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2021-01-06 00:27:56,722 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2021-01-06 00:27:58,870 INFO input.FileInputFormat: Total input files to process : 9
2021-01-06 00:27:59,100 INFO mapreduce.JobSubmitter: number of splits:9
2021-01-06 00:27:59,701 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local575103824_0001
2021-01-06 00:27:59,701 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-01-06 00:28:00,624 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2021-01-06 00:28:00,629 INFO mapreduce.Job: Running job: job_local575103824_0001
2021-01-06 00:28:00,652 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2021-01-06 00:28:00,754 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:00,754 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:00,764 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2021-01-06 00:28:01,042 INFO mapred.LocalJobRunner: Waiting for map tasks
2021-01-06 00:28:01,046 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000000_0
2021-01-06 00:28:01,201 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:01,201 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:01,333 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:01,346 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/hadoop-policy.xml:0+11392
2021-01-06 00:28:01,714 INFO mapreduce.Job: Job job_local575103824_0001 running in uber mode : false
2021-01-06 00:28:01,715 INFO mapreduce.Job: map 0% reduce 0%
2021-01-06 00:28:01,919 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:01,920 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:01,920 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:01,920 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:01,920 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:01,935 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,005 INFO mapred.LocalJobRunner:
2021-01-06 00:28:02,005 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,005 INFO mapred.MapTask: Spilling map output
2021-01-06 00:28:02,005 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2021-01-06 00:28:02,005 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2021-01-06 00:28:02,078 INFO mapred.MapTask: Finished spill 0
2021-01-06 00:28:02,131 INFO mapred.Task: Task:attempt_local575103824_0001_m_000000_0 is done. And is in the process of committing
2021-01-06 00:28:02,166 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,166 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000000_0' done.
2021-01-06 00:28:02,191 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000000_0: Counters: 18
File System Counters
FILE: Number of bytes read=329155
FILE: Number of bytes written=838284
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=265
Map output records=1
Map output bytes=17
Map output materialized bytes=25
Input split bytes=117
Combine input records=1
Combine output records=1
Spilled Records=1
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=107
Total committed heap usage (bytes)=126259200
File Input Format Counters
Bytes Read=11392
2021-01-06 00:28:02,191 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000000_0
2021-01-06 00:28:02,193 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000001_0
2021-01-06 00:28:02,199 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,199 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,200 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,201 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/capacity-scheduler.xml:0+8260
2021-01-06 00:28:02,270 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,270 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,270 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,270 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,270 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,273 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,278 INFO mapred.LocalJobRunner:
2021-01-06 00:28:02,278 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,298 INFO mapred.Task: Task:attempt_local575103824_0001_m_000001_0 is done. And is in the process of committing
2021-01-06 00:28:02,314 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,314 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000001_0' done.
2021-01-06 00:28:02,316 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000001_0: Counters: 18
File System Counters
FILE: Number of bytes read=338474
FILE: Number of bytes written=838322
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=220
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=122
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=36
Total committed heap usage (bytes)=173445120
File Input Format Counters
Bytes Read=8260
2021-01-06 00:28:02,316 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000001_0
2021-01-06 00:28:02,316 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000002_0
2021-01-06 00:28:02,333 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,333 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,334 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,341 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/kms-acls.xml:0+3518
2021-01-06 00:28:02,436 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,436 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,436 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,436 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,436 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,441 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,446 INFO mapred.LocalJobRunner:
2021-01-06 00:28:02,446 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,459 INFO mapred.Task: Task:attempt_local575103824_0001_m_000002_0 is done. And is in the process of committing
2021-01-06 00:28:02,475 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,476 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000002_0' done.
2021-01-06 00:28:02,477 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000002_0: Counters: 18
File System Counters
FILE: Number of bytes read=343051
FILE: Number of bytes written=838360
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=135
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=112
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=61
Total committed heap usage (bytes)=223019008
File Input Format Counters
Bytes Read=3518
2021-01-06 00:28:02,477 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000002_0
2021-01-06 00:28:02,477 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000003_0
2021-01-06 00:28:02,489 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,490 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,491 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,494 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/hdfs-site.xml:0+775
2021-01-06 00:28:02,598 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,599 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,599 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,599 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,599 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,602 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,606 INFO mapred.LocalJobRunner:
2021-01-06 00:28:02,606 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,626 INFO mapred.Task: Task:attempt_local575103824_0001_m_000003_0 is done. And is in the process of committing
2021-01-06 00:28:02,643 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,643 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000003_0' done.
2021-01-06 00:28:02,644 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000003_0: Counters: 18
File System Counters
FILE: Number of bytes read=344885
FILE: Number of bytes written=838398
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=21
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=113
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=59
Total committed heap usage (bytes)=165588992
File Input Format Counters
Bytes Read=775
2021-01-06 00:28:02,644 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000003_0
2021-01-06 00:28:02,644 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000004_0
2021-01-06 00:28:02,653 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,653 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,653 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,655 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/core-site.xml:0+774
2021-01-06 00:28:02,724 INFO mapreduce.Job: map 100% reduce 0%
2021-01-06 00:28:02,786 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,786 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,786 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,786 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,786 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,796 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,799 INFO mapred.LocalJobRunner:
2021-01-06 00:28:02,799 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,812 INFO mapred.Task: Task:attempt_local575103824_0001_m_000004_0 is done. And is in the process of committing
2021-01-06 00:28:02,817 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,817 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000004_0' done.
2021-01-06 00:28:02,818 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000004_0: Counters: 18
File System Counters
FILE: Number of bytes read=346718
FILE: Number of bytes written=838436
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=20
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=113
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=46
Total committed heap usage (bytes)=141631488
File Input Format Counters
Bytes Read=774
2021-01-06 00:28:02,819 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000004_0
2021-01-06 00:28:02,819 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000005_0
2021-01-06 00:28:02,831 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,831 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,832 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,836 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/mapred-site.xml:0+758
2021-01-06 00:28:02,951 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,951 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,951 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,951 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,951 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,954 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,957 INFO mapred.LocalJobRunner:
2021-01-06 00:28:02,957 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,963 INFO mapred.Task: Task:attempt_local575103824_0001_m_000005_0 is done. And is in the process of committing
2021-01-06 00:28:02,977 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,978 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000005_0' done.
2021-01-06 00:28:02,978 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000005_0: Counters: 18
File System Counters
FILE: Number of bytes read=348023
FILE: Number of bytes written=838474
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=21
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=115
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=49
Total committed heap usage (bytes)=141697024
File Input Format Counters
Bytes Read=758
2021-01-06 00:28:02,979 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000005_0
2021-01-06 00:28:02,979 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000006_0
2021-01-06 00:28:02,994 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,994 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,995 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,998 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/yarn-site.xml:0+690
2021-01-06 00:28:03,104 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:03,104 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:03,104 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:03,104 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:03,104 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:03,107 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:03,109 INFO mapred.LocalJobRunner:
2021-01-06 00:28:03,109 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:03,114 INFO mapred.Task: Task:attempt_local575103824_0001_m_000006_0 is done. And is in the process of committing
2021-01-06 00:28:03,124 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:03,125 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000006_0' done.
2021-01-06 00:28:03,125 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000006_0: Counters: 18
File System Counters
FILE: Number of bytes read=349260
FILE: Number of bytes written=838512
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=19
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=113
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=47
Total committed heap usage (bytes)=141635584
File Input Format Counters
Bytes Read=690
2021-01-06 00:28:03,126 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000006_0
2021-01-06 00:28:03,126 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000007_0
2021-01-06 00:28:03,131 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:03,132 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:03,132 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:03,134 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/kms-site.xml:0+682
2021-01-06 00:28:03,247 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:03,247 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:03,247 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:03,247 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:03,247 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:03,250 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:03,252 INFO mapred.LocalJobRunner:
2021-01-06 00:28:03,252 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:03,256 INFO mapred.Task: Task:attempt_local575103824_0001_m_000007_0 is done. And is in the process of committing
2021-01-06 00:28:03,276 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:03,276 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000007_0' done.
2021-01-06 00:28:03,277 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000007_0: Counters: 18
File System Counters
FILE: Number of bytes read=350489
FILE: Number of bytes written=838550
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=20
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=112
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=41
Total committed heap usage (bytes)=141688832
File Input Format Counters
Bytes Read=682
2021-01-06 00:28:03,277 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000007_0
2021-01-06 00:28:03,277 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000008_0
2021-01-06 00:28:03,286 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:03,287 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:03,287 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:03,295 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/httpfs-site.xml:0+620
2021-01-06 00:28:03,423 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:03,423 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:03,423 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:03,423 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:03,423 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:03,443 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:03,445 INFO mapred.LocalJobRunner:
2021-01-06 00:28:03,445 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:03,459 INFO mapred.Task: Task:attempt_local575103824_0001_m_000008_0 is done. And is in the process of committing
2021-01-06 00:28:03,474 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:03,474 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000008_0' done.
2021-01-06 00:28:03,475 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000008_0: Counters: 18
File System Counters
FILE: Number of bytes read=351656
FILE: Number of bytes written=838588
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=17
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=115
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=56
Total committed heap usage (bytes)=141807616
File Input Format Counters
Bytes Read=620
2021-01-06 00:28:03,475 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000008_0
2021-01-06 00:28:03,476 INFO mapred.LocalJobRunner: map task executor complete.
2021-01-06 00:28:03,563 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2021-01-06 00:28:03,564 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_r_000000_0
2021-01-06 00:28:03,622 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:03,623 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:03,624 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:03,642 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5e20bb90
2021-01-06 00:28:03,646 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2021-01-06 00:28:03,725 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=175977264, maxSingleShuffleLimit=43994316, mergeThreshold=116145000, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2021-01-06 00:28:03,754 INFO reduce.EventFetcher: attempt_local575103824_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2021-01-06 00:28:03,913 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:03,933 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000007_0
2021-01-06 00:28:03,948 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
2021-01-06 00:28:03,952 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
2021-01-06 00:28:03,955 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local575103824_0001_m_000000_0
2021-01-06 00:28:03,955 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->23
2021-01-06 00:28:03,962 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:03,992 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000001_0
2021-01-06 00:28:03,992 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 23, usedMemory ->25
2021-01-06 00:28:03,996 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,009 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000004_0
2021-01-06 00:28:04,010 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 25, usedMemory ->27
2021-01-06 00:28:04,011 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,011 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000002_0
2021-01-06 00:28:04,011 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 27, usedMemory ->29
2021-01-06 00:28:04,013 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,013 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000005_0
2021-01-06 00:28:04,013 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 29, usedMemory ->31
2021-01-06 00:28:04,018 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000008_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,036 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000008_0
2021-01-06 00:28:04,036 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 31, usedMemory ->33
2021-01-06 00:28:04,061 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,062 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000003_0
2021-01-06 00:28:04,062 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->35
2021-01-06 00:28:04,065 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,065 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000006_0
2021-01-06 00:28:04,065 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 35, usedMemory ->37
2021-01-06 00:28:04,068 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2021-01-06 00:28:04,076 INFO mapred.LocalJobRunner: 9 / 9 copied.
2021-01-06 00:28:04,076 INFO reduce.MergeManagerImpl: finalMerge called with 9 in-memory map-outputs and 0 on-disk map-outputs
2021-01-06 00:28:04,085 INFO mapred.Merger: Merging 9 sorted segments
2021-01-06 00:28:04,099 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
2021-01-06 00:28:04,100 INFO reduce.MergeManagerImpl: Merged 9 segments, 37 bytes to disk to satisfy reduce memory limit
2021-01-06 00:28:04,101 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
2021-01-06 00:28:04,103 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2021-01-06 00:28:04,103 INFO mapred.Merger: Merging 1 sorted segments
2021-01-06 00:28:04,104 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
2021-01-06 00:28:04,104 INFO mapred.LocalJobRunner: 9 / 9 copied.
2021-01-06 00:28:04,180 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2021-01-06 00:28:04,183 INFO mapred.Task: Task:attempt_local575103824_0001_r_000000_0 is done. And is in the process of committing
2021-01-06 00:28:04,184 INFO mapred.LocalJobRunner: 9 / 9 copied.
2021-01-06 00:28:04,184 INFO mapred.Task: Task attempt_local575103824_0001_r_000000_0 is allowed to commit now
2021-01-06 00:28:04,186 INFO output.FileOutputCommitter: Saved output of task 'attempt_local575103824_0001_r_000000_0' to file:/home/bdkl/hadoop-3.2.1/grep-temp-213726027
2021-01-06 00:28:04,191 INFO mapred.LocalJobRunner: reduce > reduce
2021-01-06 00:28:04,191 INFO mapred.Task: Task 'attempt_local575103824_0001_r_000000_0' done.
2021-01-06 00:28:04,192 INFO mapred.Task: Final Counters for attempt_local575103824_0001_r_000000_0: Counters: 24
File System Counters
FILE: Number of bytes read=352042
FILE: Number of bytes written=838736
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=73
Reduce input records=1
Reduce output records=1
Spilled Records=1
Shuffled Maps =9
Failed Shuffles=0
Merged Map outputs=9
GC time elapsed (ms)=4
Total committed heap usage (bytes)=141807616
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=123
2021-01-06 00:28:04,192 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_r_000000_0
2021-01-06 00:28:04,192 INFO mapred.LocalJobRunner: reduce task executor complete.
2021-01-06 00:28:04,730 INFO mapreduce.Job: map 100% reduce 100%
2021-01-06 00:28:04,732 INFO mapreduce.Job: Job job_local575103824_0001 completed successfully
2021-01-06 00:28:04,788 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=3453753
FILE: Number of bytes written=8384660
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=738
Map output records=1
Map output bytes=17
Map output materialized bytes=73
Input split bytes=1032
Combine input records=1
Combine output records=1
Reduce input groups=1
Reduce shuffle bytes=73
Reduce input records=1
Reduce output records=1
Spilled Records=2
Shuffled Maps =9
Failed Shuffles=0
Merged Map outputs=9
GC time elapsed (ms)=506
Total committed heap usage (bytes)=1538580480
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=27469
File Output Format Counters
Bytes Written=123
2021-01-06 00:28:04,852 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2021-01-06 00:28:04,906 INFO input.FileInputFormat: Total input files to process : 1
2021-01-06 00:28:04,911 INFO mapreduce.JobSubmitter: number of splits:1
2021-01-06 00:28:04,983 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1010195886_0002
2021-01-06 00:28:04,984 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-01-06 00:28:05,182 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2021-01-06 00:28:05,182 INFO mapreduce.Job: Running job: job_local1010195886_0002
2021-01-06 00:28:05,183 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2021-01-06 00:28:05,183 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:05,184 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:05,184 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2021-01-06 00:28:05,213 INFO mapred.LocalJobRunner: Waiting for map tasks
2021-01-06 00:28:05,216 INFO mapred.LocalJobRunner: Starting task: attempt_local1010195886_0002_m_000000_0
2021-01-06 00:28:05,238 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:05,238 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:05,239 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:05,244 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/grep-temp-213726027/part-r-00000:0+111
2021-01-06 00:28:05,395 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:05,395 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:05,395 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:05,395 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:05,395 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:05,400 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:05,411 INFO mapred.LocalJobRunner:
2021-01-06 00:28:05,411 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:05,411 INFO mapred.MapTask: Spilling map output
2021-01-06 00:28:05,411 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2021-01-06 00:28:05,411 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2021-01-06 00:28:05,420 INFO mapred.MapTask: Finished spill 0
2021-01-06 00:28:05,435 INFO mapred.Task: Task:attempt_local1010195886_0002_m_000000_0 is done. And is in the process of committing
2021-01-06 00:28:05,439 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:05,439 INFO mapred.Task: Task 'attempt_local1010195886_0002_m_000000_0' done.
2021-01-06 00:28:05,439 INFO mapred.Task: Final Counters for attempt_local1010195886_0002_m_000000_0: Counters: 17
File System Counters
FILE: Number of bytes read=668879
FILE: Number of bytes written=1677194
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=1
Map output records=1
Map output bytes=17
Map output materialized bytes=25
Input split bytes=126
Combine input records=0
Spilled Records=1
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=53
Total committed heap usage (bytes)=146010112
File Input Format Counters
Bytes Read=123
2021-01-06 00:28:05,439 INFO mapred.LocalJobRunner: Finishing task: attempt_local1010195886_0002_m_000000_0
2021-01-06 00:28:05,445 INFO mapred.LocalJobRunner: map task executor complete.
2021-01-06 00:28:05,460 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2021-01-06 00:28:05,463 INFO mapred.LocalJobRunner: Starting task: attempt_local1010195886_0002_r_000000_0
2021-01-06 00:28:05,471 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:05,471 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:05,471 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:05,471 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@66e68dd5
2021-01-06 00:28:05,471 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2021-01-06 00:28:05,479 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=175977264, maxSingleShuffleLimit=43994316, mergeThreshold=116145000, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2021-01-06 00:28:05,482 INFO reduce.EventFetcher: attempt_local1010195886_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2021-01-06 00:28:05,485 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local1010195886_0002_m_000000_0 decomp: 21 len: 25 to MEMORY
2021-01-06 00:28:05,485 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1010195886_0002_m_000000_0
2021-01-06 00:28:05,485 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21
2021-01-06 00:28:05,488 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2021-01-06 00:28:05,493 INFO mapred.LocalJobRunner: 1 / 1 copied.
2021-01-06 00:28:05,493 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2021-01-06 00:28:05,495 INFO mapred.Merger: Merging 1 sorted segments
2021-01-06 00:28:05,495 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
2021-01-06 00:28:05,511 INFO reduce.MergeManagerImpl: Merged 1 segments, 21 bytes to disk to satisfy reduce memory limit
2021-01-06 00:28:05,512 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
2021-01-06 00:28:05,512 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2021-01-06 00:28:05,512 INFO mapred.Merger: Merging 1 sorted segments
2021-01-06 00:28:05,512 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
2021-01-06 00:28:05,513 INFO mapred.LocalJobRunner: 1 / 1 copied.
2021-01-06 00:28:05,534 INFO mapred.Task: Task:attempt_local1010195886_0002_r_000000_0 is done. And is in the process of committing
2021-01-06 00:28:05,535 INFO mapred.LocalJobRunner: 1 / 1 copied.
2021-01-06 00:28:05,536 INFO mapred.Task: Task attempt_local1010195886_0002_r_000000_0 is allowed to commit now
2021-01-06 00:28:05,537 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1010195886_0002_r_000000_0' to file:/home/bdkl/hadoop-3.2.1/output
2021-01-06 00:28:05,554 INFO mapred.LocalJobRunner: reduce > reduce
2021-01-06 00:28:05,554 INFO mapred.Task: Task 'attempt_local1010195886_0002_r_000000_0' done.
2021-01-06 00:28:05,554 INFO mapred.Task: Final Counters for attempt_local1010195886_0002_r_000000_0: Counters: 24
File System Counters
FILE: Number of bytes read=668961
FILE: Number of bytes written=1677242
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=25
Reduce input records=1
Reduce output records=1
Spilled Records=1
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
Total committed heap usage (bytes)=146010112
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=23
2021-01-06 00:28:05,555 INFO mapred.LocalJobRunner: Finishing task: attempt_local1010195886_0002_r_000000_0
2021-01-06 00:28:05,557 INFO mapred.LocalJobRunner: reduce task executor complete.
2021-01-06 00:28:06,183 INFO mapreduce.Job: Job job_local1010195886_0002 running in uber mode : false
2021-01-06 00:28:06,184 INFO mapreduce.Job: map 100% reduce 100%
2021-01-06 00:28:06,186 INFO mapreduce.Job: Job job_local1010195886_0002 completed successfully
2021-01-06 00:28:06,231 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=1337840
FILE: Number of bytes written=3354436
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=1
Map output records=1
Map output bytes=17
Map output materialized bytes=25
Input split bytes=126
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=25
Reduce input records=1
Reduce output records=1
Spilled Records=2
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=53
Total committed heap usage (bytes)=292020224
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=123
File Output Format Counters
Bytes Written=23
查看一下数据的大小
伪分布式的部署
- 伪分布式是因为在一台机器实现的分布式,不是真正意义上的分布式。
步骤一:定义 hdfs的master,定义整个分布式文件系统的副本数
vim core-site.xml
vim hdfs-site.xml
- 步骤二:给本机192.169.0.102做免密登录
注意:因为hadoop的工作方式的主从的,它是通过localhost ssh 启动相应的进程
- 步骤三:对分布式文件系统做格式化,启动dfs
注意:这个master主要存储元数据的镜像和对分布式文件系统的修改
- 步骤四:启动java
- 步骤五:查看节点信息
使用命令行也可以查询信息
[bdkl@server2 ~]$ cd hadoop
[bdkl@server2 hadoop]$ hdfs dfsadmin -report
Configured Capacity: 8575254528 (7.99 GB)
Present Capacity: 5603237888 (5.22 GB)
DFS Remaining: 5603225600 (5.22 GB)
DFS Used: 12288 (12 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (1):
Name: 192.168.0.102:9866 (server2)
Hostname: server2
Decommission Status : Normal
Configured Capacity: 8575254528 (7.99 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 2972016640 (2.77 GB)
DFS Remaining: 5603225600 (5.22 GB)
DFS Used%: 0.00%
DFS Remaining%: 65.34%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Jan 10 00:19:51 CST 2021
Last Block Report: Sun Jan 10 00:08:43 CST 2021
Num of Blocks: 0
创建目录
此时默认查看当前用户的路径
上传数据
将原来在主机上的input/和output/删除,然后再进行运算测试
rm -fr input/ output/
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar grep input output 'dfs[a-z.]+'
注意:我们再次进行运算测试时,在本地没有生成output/,这是因为我们已经将input/中的数据放置在分布式存储中,在进行运算时,直接通过调用hadoop的api接口来取的数据,而生成的数据output/也直接放置在分布式存储中
我们可以通过在本地执行命令来查看放置在分布式存储中的数据
我们也可以通过命令将分布式文件系统的数据get 到本地
注意:删除本地的数据,不影响分布式文件系统中已经存储的数据
如何用命令来删除分布式文件系统中的数据
再用新的jar包命令进行词频统计
jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar wordcount input output
hdfs dfs -cat output/*
2021-01-10 00:54:19,299 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
"*" 21
"AS 9
"License"); 9
"alice,bob 21
"clumping" 1
(ASF) 1
(root 1
(the 9
--> 18
-1 1
-1, 1
0.0 1
1-MAX_INT. 1
1. 1
1.0. 1
2.0 9
40 2
40+20=60 1
: 2
<!-- 18
</configuration> 9
</description> 33
</name> 2
</property> 57
<?xml 8
<?xml-stylesheet 4
<configuration> 9
<description> 31
<description>ACL 25
<description>Default 1
<name>default.key.acl.DECRYPT_EEK</name> 1
<name>default.key.acl.GENERATE_EEK</name> 1
<name>default.key.acl.MANAGEMENT</name> 1
<name>default.key.acl.READ</name> 1
<name>hadoop.kms.acl.CREATE</name> 1
<name>hadoop.kms.acl.DECRYPT_EEK</name> 1
<name>hadoop.kms.acl.DELETE</name> 1
<name>hadoop.kms.acl.GENERATE_EEK</name> 1
<name>hadoop.kms.acl.GET</name> 1
<name>hadoop.kms.acl.GET_KEYS</name> 1
<name>hadoop.kms.acl.GET_METADATA</name> 1
<name>hadoop.kms.acl.ROLLOVER</name> 1
<name>hadoop.kms.acl.SET_KEY_MATERIAL</name> 1
<name>security.admin.operations.protocol.acl</name> 1
<name>security.applicationclient.protocol.acl</name> 1
<name>security.applicationhistory.protocol.acl</name> 1
<name>security.applicationmaster-nodemanager.applicationmaster.protocol.acl</name> 1
<name>security.applicationmaster.protocol.acl</name> 1
<name>security.client.datanode.protocol.acl</name> 1
<name>security.client.protocol.acl</name> 1
<name>security.collector-nodemanager.protocol.acl</name> 1
<name>security.containermanagement.protocol.acl</name> 1
<name>security.datanode.protocol.acl</name> 1
<name>security.distributedscheduling.protocol.acl</name> 1
<name>security.ha.service.protocol.acl</name> 1
<name>security.inter.datanode.protocol.acl</name> 1
<name>security.interqjournal.service.protocol.acl</name> 1
<name>security.job.client.protocol.acl</name> 1
<name>security.job.task.protocol.acl</name> 1
<name>security.mrhs.client.protocol.acl</name> 1
<name>security.namenode.protocol.acl</name> 1
<name>security.qjournal.service.protocol.acl</name> 1
<name>security.refresh.policy.protocol.acl</name> 1
<name>security.refresh.user.mappings.protocol.acl</name> 1
<name>security.resourcelocalizer.protocol.acl</name> 1
<name>security.resourcemanager-administration.protocol.acl</name> 1
<name>security.resourcetracker.protocol.acl</name> 1
<name>security.zkfc.protocol.acl</name> 1
<name>yarn.scheduler.capacity.application.fail-fast</name> 1
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name> 1
<name>yarn.scheduler.capacity.maximum-applications</name> 1
<name>yarn.scheduler.capacity.node-locality-delay</name> 1
<name>yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments</name> 1
<name>yarn.scheduler.capacity.queue-mappings-override.enable</name> 1
<name>yarn.scheduler.capacity.queue-mappings</name> 1
<name>yarn.scheduler.capacity.rack-locality-additional-delay</name> 1
<name>yarn.scheduler.capacity.resource-calculator</name> 1
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name> 1
<name>yarn.scheduler.capacity.root.default.acl_application_max_priority</name> 1
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name> 1
<name>yarn.scheduler.capacity.root.default.capacity</name> 1
<name>yarn.scheduler.capacity.root.default.default-application-lifetime 1
<name>yarn.scheduler.capacity.root.default.maximum-application-lifetime 1
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name> 1
<name>yarn.scheduler.capacity.root.default.state</name> 1
<name>yarn.scheduler.capacity.root.default.user-limit-factor</name> 1
<name>yarn.scheduler.capacity.root.queues</name> 1
<property> 57
<value>*</value> 41
<value>-1</value> 3
<value>0.1</value> 1
<value>10000</value> 1
<value>100</value> 2
<value>1</value> 2
<value>40</value> 1
<value></value> 1
<value>RUNNING</value> 1
<value>default</value> 1
<value>false</value> 2
<value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value> 1
A 22
ACL 37
ACL, 2
ACLs 1
ANY 9
ASF 1
AdminOperationsProtocol. 1
Any 2
Apache 10
ApplicationClientProtocol, 1
ApplicationHistoryProtocol, 1
ApplicationMaster 1
ApplicationMasterProtocol, 2
ApplicationMasters 3
BASIS, 9
But 1
CONDITIONS 9
CPU 1
CREATE 1
CapacityScheduler 2
ClientDatanodeProtocol, 1
ClientProtocol, 1
CollectorNodemanagerProtocol, 1
Complementary 1
Configuring 1
ContainerManagementProtocol 1
Controller 1
Controls 1
CryptoExtension 2
DECRYPT_EEK 1
DatanodeProtocol, 1
Default 3
DefaultResourceCalculator 1
DistributedFileSystem. 1
DistributedSchedulingAMProtocol, 1
DominantResourceCalculator 1
Example: 1
Failover 1
For 22
Foundation 1
GENERATE_EEK 1
GET 2
HAAdmin 1
HAService 1
HSClientProtocol, 1
History 1
IS" 9
If 5
In 1
Increasing 1
InterDatanodeProtocol, 1
InterQJournalProtocol, 1
It 2
JN 2
JNs 1
KIND, 9
KMS 1
LICENSE 5
Legal 1
License 27
License, 9
License. 18
Licensed 9
Lower 1
MANAGEMENT 1
MR 2
MRClientProtocol, 1
Maximum 3
Memory 1
Memory, 1
NN 1
NOTICE 1
NamenodeProtocol, 1
NodeManager 3
Nodemanager 2
Note 2
Note, 1
Number 2
OF 9
OFF_SWITCH 2
OR 9
Protocols 1
Put 5
QJournalProtocol, 1
QuorumJournalManager 1
READ 1
RM 1
ROLLOVER 1
RUNNING 1
RefreshAuthorizationPolicyProtocol, 1
RefreshUserMappingsProtocol. 1
ResourceCalculator 1
ResourceLocalizer 2
ResourceManager 3
ResourceManagerAdministrationProtocol, 1
ResourceTrackerProtocol, 1
Resourcemanager 1
Resources 1
STOPPED. 1
See 15
Server 1
Site 1
Software 1
State 1
TaskUmbilicalProtocol, 1
The 54
This 6
Typically 1
Unless 9
Used 2
User 1
Version 9
WARRANTIES 9
WITHOUT 9
We 2
When 2
Whether 1
YARN 2
You 9
ZK 1
[user={name} 1
[u|g]:[name]:[queue_name][,next 1
a 58
access 1
accompanying 5
account. 2
acls 4
active 1
additional 2
admin 2
administer 1
administrators 1
after 5
agreed 9
agreements. 1
all 27
allow 1
allowed 1
allowed.</description> 21
also 1
an 11
and 70
any 1
applicable 11
application 8
applications 5
applications' 1
applications. 1
approximately 1
are 27
as 10
assign 1
assigning 1
assignments 3
at 10
attempt 1
attempts 2
based 1
basis 1
be 16
blank. 21
block 1
by 52
calculated 1
can 9
can't 1
capacity 1
capacity.</description> 1
case, 1
changes 1
client 2
client-to-datanode 1
clients 3
cluster 3
cluster. 1
code 1
collector 1
comma-separated 21
commands 1
commands. 2
communciate 2
communicate 12
communicate. 2
compare 2
compliance 9
concurrent 1
config 1
configuration 1
configuration. 2
configured 3
considered 2
constraint 1
container 1
containers 2
containers, 1
containers. 2
context. 1
contributor 1
controls 1
copy 9
copyright 1
create-key 1
creating 1
datanodes 1
decryptEncryptedKey 1
default 13
default_priority={priority}] 1
defined. 4
delay 1
delete-key 1
dfsadmin 1
different 1
disabled. 2
disables 2
distributed 19
dominant-resource 1
during 2
e.g, 1
e.g. 21
each 7
edit 1
either 9
enabled, 1
encoding="UTF-8"?> 5
equal 2
etc. 3
example, 1
exceed 1
exceeds 2
except 9
explicitly 4
express 9
fail 1
false. 1
feature 2
feature. 1
file 12
file. 10
for 65
from 1
generateEncryptedKey 1
generation 1
generic 1
get-current-key 1
get-key-metadata 1
get-key-version 1
get-keys 1
get-keys-metadata 1
governing 9
group 42
group={name} 1
hard 1
has 1
heartbeat. 1
history 1
hot-reloaded 1
href="configuration.xsl"?> 4
http://www.apache.org/licenses/LICENSE-2.0 9
i.e. 2
if 4
ignored, 1
implementation 1
implied. 9
improve 1
in 36
in-effect. 1
information 1
instead 1
inter-datanode 1
into 2
is 76
it 3
job 4
jobs 4
key 7
key. 1
killed 1
killing 1
language 9
law 9
leaf 2
less 2
level 1
license 1
licenses 1
lifetime 6
lifetime. 3
limit 2
limitations 9
list 45
locality 1
locations 1
logs.</description> 1
longer 1
low 1
manage 1
map 2
mapping 1
mapping]* 1
mappings 1
mappings. 1
maps 1
masters 1
material 3
max_priority={priority} 1
maximum 3
may 18
means 21
missed 5
more 1
mradmin 1
multi-dimensional 1
name 1
namenode 1
namenode. 2
namenode.</description> 1
names. 21
no 1
node's 1
node-locality-delay 1
node-locality-delay=40 1
nodemanager 2
nodes 1
nodes. 1
not 18
number 6
obtain 9
of 77
off-switch 3
on 12
one 4
ones, 1
ones. 1
only 3
operations 5
operations. 8
opportunities 3
opportunities, 1
opportunities. 1
or 23
other 1
other. 7
over 1
overridden 1
override 1
overrides 5
ownership. 1
parameter, 2
parent 1
part 2
particular 1
pending 1
per 1
percent 1
percentage 1
permissions 9
place 1
point-in-time 2
policy 1
positive 1
present, 1
previous 1
priority. 1
properties 1
property 5
protocol 4
protocol, 2
provide 1
query 1
queue 8
queue). 1
queue. 7
queues 4
queues, 1
rack-local 3
rack-locality-delay=20, 1
rack. 1
rate 1
recovery 1
recovery. 1
reduce 2
refresh 2
refreshable. 1
regarding 1
request 1
request, 1
required 9
resource 1
resources 2
response. 2
result 1
returned 2
rolling 1
rollover-key 1
root 1
run 1
running 1
running. 1
same 1
schedule 2
scheduler 1
scheduler. 1
scheduling 3
scheduling. 1
secondary 1
seconds. 2
security 1
separated 21
server 1
service 2
setting 2
should 3
site-specific 5
size 3
software 9
sooner. 1
special 21
specific 10
specified 3
specify 1
stand-by 1
state 1
states 1
status 2
submission 2
submit 2
submitted 4
such 1
syntax 1
taken 2
taken. 1
target 1
tasks 1
tasktracker. 1
than 3
that 7
the 139
then 2
this 28
time 1
timeline 3
timestamp. 1
to 53
too 1
type="text/xsl" 4
u:%user:%user 1
under 28
unique 1
updating 1
use 11
used 24
user 48
user. 2
user? 1
users 24
users,wheel". 21
uses 2
using 1
v2 1
valid. 1
value 30
value, 2
values 2
version="1.0" 5
version="1.0"?> 3
via 1
well 1
when 3
which 10
while 1
who 3
will 12
with 28
work 1
writing, 9
you 10
zero 2