Linux企业实战（六十）——Hadoop（二）

最新推荐文章于 2024-03-01 11:10:44 发布
最新推荐文章于 2024-03-01 11:10:44 发布
阅读量331 收藏
点赞数
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/bdkl9998/article/details/112298749
版权

单机版部署Hadoop

  • 实验环境：
    在server2上部署
  • 步骤一：创建普通用户，解压软件包，创建软链接，编辑hadoop环境变量文件
    在这里插入图片描述
    在这里插入图片描述
    在这里插入图片描述
  • 步骤二：运行hadoop，进行测试
    在这里插入图片描述
[bdkl@server2 bin]$ ./hadoop
Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
 or    hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]
  where CLASSNAME is a user-provided Java class

  OPTIONS is none or any of:

buildpaths                       attempt to add class files from build tree
--config dir                     Hadoop config directory
--debug                          turn on shell script debug mode
--help                           usage information
hostnames list[,of,host,names]   hosts to use in slave mode
hosts filename                   list of hosts to use in slave mode
loglevel level                   set the log4j level for this command
workers                          turn on worker mode

  SUBCOMMAND is one of:


    Admin Commands:

daemonlog     get/set the log level for each daemon

    Client Commands:

archive       create a Hadoop archive
checknative   check native Hadoop and compression libraries availability
classpath     prints the class path needed to get the Hadoop jar and the required libraries
conftest      validate configuration XML files
credential    interact with credential providers
distch        distributed metadata changer
distcp        copy file or directories recursively
dtutil        operations related to delegation tokens
envvars       display computed Hadoop environment variables
fs            run a generic filesystem user client
gridmix       submit a mix of synthetic job, modeling a profiled from production load
jar <jar>     run a jar file. NOTE: please use "yarn jar" to launch YARN applications, not this command.
jnipath       prints the java.library.path
kdiag         Diagnose Kerberos Problems
kerbname      show auth_to_local principal conversion
key           manage keys via the KeyProvider
rumenfolder   scale a rumen input trace
rumentrace    convert logs into a rumen trace
s3guard       manage metadata on S3
trace         view and modify Hadoop tracing settings
version       print the version

    Daemon Commands:

kms           run KMS, the Key Management Server

SUBCOMMAND may print help when invoked w/o parameters or with -h.
  • 步骤三：进行测试
    在这里插入图片描述
#进行运算,output目录必须不存在
[bdkl@server2 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar grep input output 'dfs[a-z.]+'
2021-01-06 00:27:56,455 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2021-01-06 00:27:56,721 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2021-01-06 00:27:56,722 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2021-01-06 00:27:58,870 INFO input.FileInputFormat: Total input files to process : 9
2021-01-06 00:27:59,100 INFO mapreduce.JobSubmitter: number of splits:9
2021-01-06 00:27:59,701 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local575103824_0001
2021-01-06 00:27:59,701 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-01-06 00:28:00,624 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2021-01-06 00:28:00,629 INFO mapreduce.Job: Running job: job_local575103824_0001
2021-01-06 00:28:00,652 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2021-01-06 00:28:00,754 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:00,754 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:00,764 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2021-01-06 00:28:01,042 INFO mapred.LocalJobRunner: Waiting for map tasks
2021-01-06 00:28:01,046 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000000_0
2021-01-06 00:28:01,201 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:01,201 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:01,333 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:01,346 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/hadoop-policy.xml:0+11392
2021-01-06 00:28:01,714 INFO mapreduce.Job: Job job_local575103824_0001 running in uber mode : false
2021-01-06 00:28:01,715 INFO mapreduce.Job:  map 0% reduce 0%
2021-01-06 00:28:01,919 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:01,920 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:01,920 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:01,920 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:01,920 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:01,935 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,005 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:02,005 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,005 INFO mapred.MapTask: Spilling map output
2021-01-06 00:28:02,005 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2021-01-06 00:28:02,005 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2021-01-06 00:28:02,078 INFO mapred.MapTask: Finished spill 0
2021-01-06 00:28:02,131 INFO mapred.Task: Task:attempt_local575103824_0001_m_000000_0 is done. And is in the process of committing
2021-01-06 00:28:02,166 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,166 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000000_0' done.
2021-01-06 00:28:02,191 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000000_0: Counters: 18
	File System Counters
		FILE: Number of bytes read=329155
		FILE: Number of bytes written=838284
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=265
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=25
		Input split bytes=117
		Combine input records=1
		Combine output records=1
		Spilled Records=1
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=107
		Total committed heap usage (bytes)=126259200
	File Input Format Counters 
		Bytes Read=11392
2021-01-06 00:28:02,191 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000000_0
2021-01-06 00:28:02,193 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000001_0
2021-01-06 00:28:02,199 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,199 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,200 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,201 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/capacity-scheduler.xml:0+8260
2021-01-06 00:28:02,270 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,270 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,270 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,270 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,270 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,273 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,278 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:02,278 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,298 INFO mapred.Task: Task:attempt_local575103824_0001_m_000001_0 is done. And is in the process of committing
2021-01-06 00:28:02,314 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,314 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000001_0' done.
2021-01-06 00:28:02,316 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000001_0: Counters: 18
	File System Counters
		FILE: Number of bytes read=338474
		FILE: Number of bytes written=838322
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=220
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=122
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=36
		Total committed heap usage (bytes)=173445120
	File Input Format Counters 
		Bytes Read=8260
2021-01-06 00:28:02,316 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000001_0
2021-01-06 00:28:02,316 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000002_0
2021-01-06 00:28:02,333 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,333 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,334 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,341 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/kms-acls.xml:0+3518
2021-01-06 00:28:02,436 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,436 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,436 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,436 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,436 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,441 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,446 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:02,446 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,459 INFO mapred.Task: Task:attempt_local575103824_0001_m_000002_0 is done. And is in the process of committing
2021-01-06 00:28:02,475 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,476 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000002_0' done.
2021-01-06 00:28:02,477 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000002_0: Counters: 18
	File System Counters
		FILE: Number of bytes read=343051
		FILE: Number of bytes written=838360
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=135
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=112
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=61
		Total committed heap usage (bytes)=223019008
	File Input Format Counters 
		Bytes Read=3518
2021-01-06 00:28:02,477 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000002_0
2021-01-06 00:28:02,477 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000003_0
2021-01-06 00:28:02,489 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,490 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,491 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,494 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/hdfs-site.xml:0+775
2021-01-06 00:28:02,598 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,599 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,599 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,599 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,599 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,602 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,606 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:02,606 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,626 INFO mapred.Task: Task:attempt_local575103824_0001_m_000003_0 is done. And is in the process of committing
2021-01-06 00:28:02,643 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,643 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000003_0' done.
2021-01-06 00:28:02,644 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000003_0: Counters: 18
	File System Counters
		FILE: Number of bytes read=344885
		FILE: Number of bytes written=838398
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=21
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=113
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=59
		Total committed heap usage (bytes)=165588992
	File Input Format Counters 
		Bytes Read=775
2021-01-06 00:28:02,644 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000003_0
2021-01-06 00:28:02,644 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000004_0
2021-01-06 00:28:02,653 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,653 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,653 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,655 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/core-site.xml:0+774
2021-01-06 00:28:02,724 INFO mapreduce.Job:  map 100% reduce 0%
2021-01-06 00:28:02,786 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,786 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,786 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,786 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,786 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,796 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,799 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:02,799 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,812 INFO mapred.Task: Task:attempt_local575103824_0001_m_000004_0 is done. And is in the process of committing
2021-01-06 00:28:02,817 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,817 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000004_0' done.
2021-01-06 00:28:02,818 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000004_0: Counters: 18
	File System Counters
		FILE: Number of bytes read=346718
		FILE: Number of bytes written=838436
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=20
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=113
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=46
		Total committed heap usage (bytes)=141631488
	File Input Format Counters 
		Bytes Read=774
2021-01-06 00:28:02,819 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000004_0
2021-01-06 00:28:02,819 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000005_0
2021-01-06 00:28:02,831 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,831 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,832 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,836 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/mapred-site.xml:0+758
2021-01-06 00:28:02,951 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:02,951 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:02,951 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:02,951 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:02,951 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:02,954 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:02,957 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:02,957 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:02,963 INFO mapred.Task: Task:attempt_local575103824_0001_m_000005_0 is done. And is in the process of committing
2021-01-06 00:28:02,977 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:02,978 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000005_0' done.
2021-01-06 00:28:02,978 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000005_0: Counters: 18
	File System Counters
		FILE: Number of bytes read=348023
		FILE: Number of bytes written=838474
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=21
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=115
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=49
		Total committed heap usage (bytes)=141697024
	File Input Format Counters 
		Bytes Read=758
2021-01-06 00:28:02,979 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000005_0
2021-01-06 00:28:02,979 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000006_0
2021-01-06 00:28:02,994 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:02,994 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:02,995 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:02,998 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/yarn-site.xml:0+690
2021-01-06 00:28:03,104 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:03,104 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:03,104 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:03,104 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:03,104 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:03,107 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:03,109 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:03,109 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:03,114 INFO mapred.Task: Task:attempt_local575103824_0001_m_000006_0 is done. And is in the process of committing
2021-01-06 00:28:03,124 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:03,125 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000006_0' done.
2021-01-06 00:28:03,125 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000006_0: Counters: 18
	File System Counters
		FILE: Number of bytes read=349260
		FILE: Number of bytes written=838512
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=19
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=113
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=47
		Total committed heap usage (bytes)=141635584
	File Input Format Counters 
		Bytes Read=690
2021-01-06 00:28:03,126 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000006_0
2021-01-06 00:28:03,126 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000007_0
2021-01-06 00:28:03,131 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:03,132 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:03,132 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:03,134 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/kms-site.xml:0+682
2021-01-06 00:28:03,247 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:03,247 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:03,247 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:03,247 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:03,247 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:03,250 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:03,252 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:03,252 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:03,256 INFO mapred.Task: Task:attempt_local575103824_0001_m_000007_0 is done. And is in the process of committing
2021-01-06 00:28:03,276 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:03,276 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000007_0' done.
2021-01-06 00:28:03,277 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000007_0: Counters: 18
	File System Counters
		FILE: Number of bytes read=350489
		FILE: Number of bytes written=838550
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=20
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=112
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=41
		Total committed heap usage (bytes)=141688832
	File Input Format Counters 
		Bytes Read=682
2021-01-06 00:28:03,277 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000007_0
2021-01-06 00:28:03,277 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_m_000008_0
2021-01-06 00:28:03,286 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:03,287 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:03,287 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:03,295 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/input/httpfs-site.xml:0+620
2021-01-06 00:28:03,423 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:03,423 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:03,423 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:03,423 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:03,423 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:03,443 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:03,445 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:03,445 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:03,459 INFO mapred.Task: Task:attempt_local575103824_0001_m_000008_0 is done. And is in the process of committing
2021-01-06 00:28:03,474 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:03,474 INFO mapred.Task: Task 'attempt_local575103824_0001_m_000008_0' done.
2021-01-06 00:28:03,475 INFO mapred.Task: Final Counters for attempt_local575103824_0001_m_000008_0: Counters: 18
	File System Counters
		FILE: Number of bytes read=351656
		FILE: Number of bytes written=838588
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=17
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=115
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=56
		Total committed heap usage (bytes)=141807616
	File Input Format Counters 
		Bytes Read=620
2021-01-06 00:28:03,475 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_m_000008_0
2021-01-06 00:28:03,476 INFO mapred.LocalJobRunner: map task executor complete.
2021-01-06 00:28:03,563 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2021-01-06 00:28:03,564 INFO mapred.LocalJobRunner: Starting task: attempt_local575103824_0001_r_000000_0
2021-01-06 00:28:03,622 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:03,623 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:03,624 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:03,642 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5e20bb90
2021-01-06 00:28:03,646 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2021-01-06 00:28:03,725 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=175977264, maxSingleShuffleLimit=43994316, mergeThreshold=116145000, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2021-01-06 00:28:03,754 INFO reduce.EventFetcher: attempt_local575103824_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2021-01-06 00:28:03,913 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:03,933 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000007_0
2021-01-06 00:28:03,948 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
2021-01-06 00:28:03,952 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
2021-01-06 00:28:03,955 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local575103824_0001_m_000000_0
2021-01-06 00:28:03,955 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->23
2021-01-06 00:28:03,962 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:03,992 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000001_0
2021-01-06 00:28:03,992 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 23, usedMemory ->25
2021-01-06 00:28:03,996 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,009 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000004_0
2021-01-06 00:28:04,010 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 25, usedMemory ->27
2021-01-06 00:28:04,011 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,011 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000002_0
2021-01-06 00:28:04,011 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 27, usedMemory ->29
2021-01-06 00:28:04,013 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,013 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000005_0
2021-01-06 00:28:04,013 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 29, usedMemory ->31
2021-01-06 00:28:04,018 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000008_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,036 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000008_0
2021-01-06 00:28:04,036 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 31, usedMemory ->33
2021-01-06 00:28:04,061 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,062 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000003_0
2021-01-06 00:28:04,062 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->35
2021-01-06 00:28:04,065 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local575103824_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
2021-01-06 00:28:04,065 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local575103824_0001_m_000006_0
2021-01-06 00:28:04,065 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 35, usedMemory ->37
2021-01-06 00:28:04,068 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2021-01-06 00:28:04,076 INFO mapred.LocalJobRunner: 9 / 9 copied.
2021-01-06 00:28:04,076 INFO reduce.MergeManagerImpl: finalMerge called with 9 in-memory map-outputs and 0 on-disk map-outputs
2021-01-06 00:28:04,085 INFO mapred.Merger: Merging 9 sorted segments
2021-01-06 00:28:04,099 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
2021-01-06 00:28:04,100 INFO reduce.MergeManagerImpl: Merged 9 segments, 37 bytes to disk to satisfy reduce memory limit
2021-01-06 00:28:04,101 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
2021-01-06 00:28:04,103 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2021-01-06 00:28:04,103 INFO mapred.Merger: Merging 1 sorted segments
2021-01-06 00:28:04,104 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
2021-01-06 00:28:04,104 INFO mapred.LocalJobRunner: 9 / 9 copied.
2021-01-06 00:28:04,180 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2021-01-06 00:28:04,183 INFO mapred.Task: Task:attempt_local575103824_0001_r_000000_0 is done. And is in the process of committing
2021-01-06 00:28:04,184 INFO mapred.LocalJobRunner: 9 / 9 copied.
2021-01-06 00:28:04,184 INFO mapred.Task: Task attempt_local575103824_0001_r_000000_0 is allowed to commit now
2021-01-06 00:28:04,186 INFO output.FileOutputCommitter: Saved output of task 'attempt_local575103824_0001_r_000000_0' to file:/home/bdkl/hadoop-3.2.1/grep-temp-213726027
2021-01-06 00:28:04,191 INFO mapred.LocalJobRunner: reduce > reduce
2021-01-06 00:28:04,191 INFO mapred.Task: Task 'attempt_local575103824_0001_r_000000_0' done.
2021-01-06 00:28:04,192 INFO mapred.Task: Final Counters for attempt_local575103824_0001_r_000000_0: Counters: 24
	File System Counters
		FILE: Number of bytes read=352042
		FILE: Number of bytes written=838736
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=73
		Reduce input records=1
		Reduce output records=1
		Spilled Records=1
		Shuffled Maps =9
		Failed Shuffles=0
		Merged Map outputs=9
		GC time elapsed (ms)=4
		Total committed heap usage (bytes)=141807616
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Output Format Counters 
		Bytes Written=123
2021-01-06 00:28:04,192 INFO mapred.LocalJobRunner: Finishing task: attempt_local575103824_0001_r_000000_0
2021-01-06 00:28:04,192 INFO mapred.LocalJobRunner: reduce task executor complete.
2021-01-06 00:28:04,730 INFO mapreduce.Job:  map 100% reduce 100%
2021-01-06 00:28:04,732 INFO mapreduce.Job: Job job_local575103824_0001 completed successfully
2021-01-06 00:28:04,788 INFO mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=3453753
		FILE: Number of bytes written=8384660
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=738
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=73
		Input split bytes=1032
		Combine input records=1
		Combine output records=1
		Reduce input groups=1
		Reduce shuffle bytes=73
		Reduce input records=1
		Reduce output records=1
		Spilled Records=2
		Shuffled Maps =9
		Failed Shuffles=0
		Merged Map outputs=9
		GC time elapsed (ms)=506
		Total committed heap usage (bytes)=1538580480
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=27469
	File Output Format Counters 
		Bytes Written=123
2021-01-06 00:28:04,852 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2021-01-06 00:28:04,906 INFO input.FileInputFormat: Total input files to process : 1
2021-01-06 00:28:04,911 INFO mapreduce.JobSubmitter: number of splits:1
2021-01-06 00:28:04,983 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1010195886_0002
2021-01-06 00:28:04,984 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-01-06 00:28:05,182 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2021-01-06 00:28:05,182 INFO mapreduce.Job: Running job: job_local1010195886_0002
2021-01-06 00:28:05,183 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2021-01-06 00:28:05,183 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:05,184 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:05,184 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2021-01-06 00:28:05,213 INFO mapred.LocalJobRunner: Waiting for map tasks
2021-01-06 00:28:05,216 INFO mapred.LocalJobRunner: Starting task: attempt_local1010195886_0002_m_000000_0
2021-01-06 00:28:05,238 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:05,238 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:05,239 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:05,244 INFO mapred.MapTask: Processing split: file:/home/bdkl/hadoop-3.2.1/grep-temp-213726027/part-r-00000:0+111
2021-01-06 00:28:05,395 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2021-01-06 00:28:05,395 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2021-01-06 00:28:05,395 INFO mapred.MapTask: soft limit at 83886080
2021-01-06 00:28:05,395 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2021-01-06 00:28:05,395 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2021-01-06 00:28:05,400 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-01-06 00:28:05,411 INFO mapred.LocalJobRunner: 
2021-01-06 00:28:05,411 INFO mapred.MapTask: Starting flush of map output
2021-01-06 00:28:05,411 INFO mapred.MapTask: Spilling map output
2021-01-06 00:28:05,411 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2021-01-06 00:28:05,411 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2021-01-06 00:28:05,420 INFO mapred.MapTask: Finished spill 0
2021-01-06 00:28:05,435 INFO mapred.Task: Task:attempt_local1010195886_0002_m_000000_0 is done. And is in the process of committing
2021-01-06 00:28:05,439 INFO mapred.LocalJobRunner: map
2021-01-06 00:28:05,439 INFO mapred.Task: Task 'attempt_local1010195886_0002_m_000000_0' done.
2021-01-06 00:28:05,439 INFO mapred.Task: Final Counters for attempt_local1010195886_0002_m_000000_0: Counters: 17
	File System Counters
		FILE: Number of bytes read=668879
		FILE: Number of bytes written=1677194
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=1
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=25
		Input split bytes=126
		Combine input records=0
		Spilled Records=1
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=53
		Total committed heap usage (bytes)=146010112
	File Input Format Counters 
		Bytes Read=123
2021-01-06 00:28:05,439 INFO mapred.LocalJobRunner: Finishing task: attempt_local1010195886_0002_m_000000_0
2021-01-06 00:28:05,445 INFO mapred.LocalJobRunner: map task executor complete.
2021-01-06 00:28:05,460 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2021-01-06 00:28:05,463 INFO mapred.LocalJobRunner: Starting task: attempt_local1010195886_0002_r_000000_0
2021-01-06 00:28:05,471 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-01-06 00:28:05,471 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-01-06 00:28:05,471 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-01-06 00:28:05,471 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@66e68dd5
2021-01-06 00:28:05,471 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2021-01-06 00:28:05,479 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=175977264, maxSingleShuffleLimit=43994316, mergeThreshold=116145000, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2021-01-06 00:28:05,482 INFO reduce.EventFetcher: attempt_local1010195886_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2021-01-06 00:28:05,485 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local1010195886_0002_m_000000_0 decomp: 21 len: 25 to MEMORY
2021-01-06 00:28:05,485 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1010195886_0002_m_000000_0
2021-01-06 00:28:05,485 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21
2021-01-06 00:28:05,488 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2021-01-06 00:28:05,493 INFO mapred.LocalJobRunner: 1 / 1 copied.
2021-01-06 00:28:05,493 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2021-01-06 00:28:05,495 INFO mapred.Merger: Merging 1 sorted segments
2021-01-06 00:28:05,495 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
2021-01-06 00:28:05,511 INFO reduce.MergeManagerImpl: Merged 1 segments, 21 bytes to disk to satisfy reduce memory limit
2021-01-06 00:28:05,512 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
2021-01-06 00:28:05,512 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2021-01-06 00:28:05,512 INFO mapred.Merger: Merging 1 sorted segments
2021-01-06 00:28:05,512 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
2021-01-06 00:28:05,513 INFO mapred.LocalJobRunner: 1 / 1 copied.
2021-01-06 00:28:05,534 INFO mapred.Task: Task:attempt_local1010195886_0002_r_000000_0 is done. And is in the process of committing
2021-01-06 00:28:05,535 INFO mapred.LocalJobRunner: 1 / 1 copied.
2021-01-06 00:28:05,536 INFO mapred.Task: Task attempt_local1010195886_0002_r_000000_0 is allowed to commit now
2021-01-06 00:28:05,537 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1010195886_0002_r_000000_0' to file:/home/bdkl/hadoop-3.2.1/output
2021-01-06 00:28:05,554 INFO mapred.LocalJobRunner: reduce > reduce
2021-01-06 00:28:05,554 INFO mapred.Task: Task 'attempt_local1010195886_0002_r_000000_0' done.
2021-01-06 00:28:05,554 INFO mapred.Task: Final Counters for attempt_local1010195886_0002_r_000000_0: Counters: 24
	File System Counters
		FILE: Number of bytes read=668961
		FILE: Number of bytes written=1677242
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=25
		Reduce input records=1
		Reduce output records=1
		Spilled Records=1
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=146010112
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Output Format Counters 
		Bytes Written=23
2021-01-06 00:28:05,555 INFO mapred.LocalJobRunner: Finishing task: attempt_local1010195886_0002_r_000000_0
2021-01-06 00:28:05,557 INFO mapred.LocalJobRunner: reduce task executor complete.
2021-01-06 00:28:06,183 INFO mapreduce.Job: Job job_local1010195886_0002 running in uber mode : false
2021-01-06 00:28:06,184 INFO mapreduce.Job:  map 100% reduce 100%
2021-01-06 00:28:06,186 INFO mapreduce.Job: Job job_local1010195886_0002 completed successfully
2021-01-06 00:28:06,231 INFO mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=1337840
		FILE: Number of bytes written=3354436
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=1
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=25
		Input split bytes=126
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=25
		Reduce input records=1
		Reduce output records=1
		Spilled Records=2
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=53
		Total committed heap usage (bytes)=292020224
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=123
	File Output Format Counters 
		Bytes Written=23

查看一下数据的大小
在这里插入图片描述

伪分布式的部署

  • 伪分布式是因为在一台机器实现的分布式，不是真正意义上的分布式。

步骤一：定义 hdfs的master，定义整个分布式文件系统的副本数
在这里插入图片描述

vim core-site.xml

在这里插入图片描述

vim hdfs-site.xml

在这里插入图片描述

  • 步骤二：给本机192.169.0.102做免密登录
    在这里插入图片描述
    在这里插入图片描述
    在这里插入图片描述
    注意：因为hadoop的工作方式的主从的,它是通过localhost ssh 启动相应的进程
    在这里插入图片描述
    在这里插入图片描述
  • 步骤三：对分布式文件系统做格式化，启动dfs
    在这里插入图片描述
    注意：这个master主要存储元数据的镜像和对分布式文件系统的修改
    在这里插入图片描述
    在这里插入图片描述
  • 步骤四：启动java
    在这里插入图片描述

在这里插入图片描述

  • 步骤五：查看节点信息
    在这里插入图片描述
    使用命令行也可以查询信息
[bdkl@server2 ~]$ cd hadoop
[bdkl@server2 hadoop]$ hdfs dfsadmin  -report
Configured Capacity: 8575254528 (7.99 GB)
Present Capacity: 5603237888 (5.22 GB)
DFS Remaining: 5603225600 (5.22 GB)
DFS Used: 12288 (12 KB)
DFS Used%: 0.00%
Replicated Blocks:
	Under replicated blocks: 0
	Blocks with corrupt replicas: 0
	Missing blocks: 0
	Missing blocks (with replication factor 1): 0
	Low redundancy blocks with highest priority to recover: 0
	Pending deletion blocks: 0
Erasure Coded Block Groups: 
	Low redundancy block groups: 0
	Block groups with corrupt internal blocks: 0
	Missing block groups: 0
	Low redundancy blocks with highest priority to recover: 0
	Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (1):

Name: 192.168.0.102:9866 (server2)
Hostname: server2
Decommission Status : Normal
Configured Capacity: 8575254528 (7.99 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 2972016640 (2.77 GB)
DFS Remaining: 5603225600 (5.22 GB)
DFS Used%: 0.00%
DFS Remaining%: 65.34%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Jan 10 00:19:51 CST 2021
Last Block Report: Sun Jan 10 00:08:43 CST 2021
Num of Blocks: 0

在这里插入图片描述
创建目录
在这里插入图片描述
此时默认查看当前用户的路径

上传数据
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
将原来在主机上的input/和output/删除，然后再进行运算测试

rm -fr input/ output/
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar grep input output 'dfs[a-z.]+'

在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
注意：我们再次进行运算测试时，在本地没有生成output/,这是因为我们已经将input/中的数据放置在分布式存储中，在进行运算时，直接通过调用hadoop的api接口来取的数据，而生成的数据output/也直接放置在分布式存储中

我们可以通过在本地执行命令来查看放置在分布式存储中的数据
在这里插入图片描述
我们也可以通过命令将分布式文件系统的数据get 到本地
在这里插入图片描述

注意：删除本地的数据，不影响分布式文件系统中已经存储的数据
在这里插入图片描述
在这里插入图片描述

如何用命令来删除分布式文件系统中的数据
在这里插入图片描述
在这里插入图片描述
再用新的jar包命令进行词频统计

jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar  wordcount input output
hdfs dfs -cat output/*

2021-01-10 00:54:19,299 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
"*"	21
"AS	9
"License");	9
"alice,bob	21
"clumping"	1
(ASF)	1
(root	1
(the	9
-->	18
-1	1
-1,	1
0.0	1
1-MAX_INT.	1
1.	1
1.0.	1
2.0	9
40	2
40+20=60	1
:	2
<!--	18
</configuration>	9
</description>	33
</name>	2
</property>	57
<?xml	8
<?xml-stylesheet	4
<configuration>	9
<description>	31
<description>ACL	25
<description>Default	1
<name>default.key.acl.DECRYPT_EEK</name>	1
<name>default.key.acl.GENERATE_EEK</name>	1
<name>default.key.acl.MANAGEMENT</name>	1
<name>default.key.acl.READ</name>	1
<name>hadoop.kms.acl.CREATE</name>	1
<name>hadoop.kms.acl.DECRYPT_EEK</name>	1
<name>hadoop.kms.acl.DELETE</name>	1
<name>hadoop.kms.acl.GENERATE_EEK</name>	1
<name>hadoop.kms.acl.GET</name>	1
<name>hadoop.kms.acl.GET_KEYS</name>	1
<name>hadoop.kms.acl.GET_METADATA</name>	1
<name>hadoop.kms.acl.ROLLOVER</name>	1
<name>hadoop.kms.acl.SET_KEY_MATERIAL</name>	1
<name>security.admin.operations.protocol.acl</name>	1
<name>security.applicationclient.protocol.acl</name>	1
<name>security.applicationhistory.protocol.acl</name>	1
<name>security.applicationmaster-nodemanager.applicationmaster.protocol.acl</name>	1
<name>security.applicationmaster.protocol.acl</name>	1
<name>security.client.datanode.protocol.acl</name>	1
<name>security.client.protocol.acl</name>	1
<name>security.collector-nodemanager.protocol.acl</name>	1
<name>security.containermanagement.protocol.acl</name>	1
<name>security.datanode.protocol.acl</name>	1
<name>security.distributedscheduling.protocol.acl</name>	1
<name>security.ha.service.protocol.acl</name>	1
<name>security.inter.datanode.protocol.acl</name>	1
<name>security.interqjournal.service.protocol.acl</name>	1
<name>security.job.client.protocol.acl</name>	1
<name>security.job.task.protocol.acl</name>	1
<name>security.mrhs.client.protocol.acl</name>	1
<name>security.namenode.protocol.acl</name>	1
<name>security.qjournal.service.protocol.acl</name>	1
<name>security.refresh.policy.protocol.acl</name>	1
<name>security.refresh.user.mappings.protocol.acl</name>	1
<name>security.resourcelocalizer.protocol.acl</name>	1
<name>security.resourcemanager-administration.protocol.acl</name>	1
<name>security.resourcetracker.protocol.acl</name>	1
<name>security.zkfc.protocol.acl</name>	1
<name>yarn.scheduler.capacity.application.fail-fast</name>	1
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>	1
<name>yarn.scheduler.capacity.maximum-applications</name>	1
<name>yarn.scheduler.capacity.node-locality-delay</name>	1
<name>yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments</name>	1
<name>yarn.scheduler.capacity.queue-mappings-override.enable</name>	1
<name>yarn.scheduler.capacity.queue-mappings</name>	1
<name>yarn.scheduler.capacity.rack-locality-additional-delay</name>	1
<name>yarn.scheduler.capacity.resource-calculator</name>	1
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>	1
<name>yarn.scheduler.capacity.root.default.acl_application_max_priority</name>	1
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>	1
<name>yarn.scheduler.capacity.root.default.capacity</name>	1
<name>yarn.scheduler.capacity.root.default.default-application-lifetime	1
<name>yarn.scheduler.capacity.root.default.maximum-application-lifetime	1
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>	1
<name>yarn.scheduler.capacity.root.default.state</name>	1
<name>yarn.scheduler.capacity.root.default.user-limit-factor</name>	1
<name>yarn.scheduler.capacity.root.queues</name>	1
<property>	57
<value>*</value>	41
<value>-1</value>	3
<value>0.1</value>	1
<value>10000</value>	1
<value>100</value>	2
<value>1</value>	2
<value>40</value>	1
<value></value>	1
<value>RUNNING</value>	1
<value>default</value>	1
<value>false</value>	2
<value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>	1
A	22
ACL	37
ACL,	2
ACLs	1
ANY	9
ASF	1
AdminOperationsProtocol.	1
Any	2
Apache	10
ApplicationClientProtocol,	1
ApplicationHistoryProtocol,	1
ApplicationMaster	1
ApplicationMasterProtocol,	2
ApplicationMasters	3
BASIS,	9
But	1
CONDITIONS	9
CPU	1
CREATE	1
CapacityScheduler	2
ClientDatanodeProtocol,	1
ClientProtocol,	1
CollectorNodemanagerProtocol,	1
Complementary	1
Configuring	1
ContainerManagementProtocol	1
Controller	1
Controls	1
CryptoExtension	2
DECRYPT_EEK	1
DatanodeProtocol,	1
Default	3
DefaultResourceCalculator	1
DistributedFileSystem.	1
DistributedSchedulingAMProtocol,	1
DominantResourceCalculator	1
Example:	1
Failover	1
For	22
Foundation	1
GENERATE_EEK	1
GET	2
HAAdmin	1
HAService	1
HSClientProtocol,	1
History	1
IS"	9
If	5
In	1
Increasing	1
InterDatanodeProtocol,	1
InterQJournalProtocol,	1
It	2
JN	2
JNs	1
KIND,	9
KMS	1
LICENSE	5
Legal	1
License	27
License,	9
License.	18
Licensed	9
Lower	1
MANAGEMENT	1
MR	2
MRClientProtocol,	1
Maximum	3
Memory	1
Memory,	1
NN	1
NOTICE	1
NamenodeProtocol,	1
NodeManager	3
Nodemanager	2
Note	2
Note,	1
Number	2
OF	9
OFF_SWITCH	2
OR	9
Protocols	1
Put	5
QJournalProtocol,	1
QuorumJournalManager	1
READ	1
RM	1
ROLLOVER	1
RUNNING	1
RefreshAuthorizationPolicyProtocol,	1
RefreshUserMappingsProtocol.	1
ResourceCalculator	1
ResourceLocalizer	2
ResourceManager	3
ResourceManagerAdministrationProtocol,	1
ResourceTrackerProtocol,	1
Resourcemanager	1
Resources	1
STOPPED.	1
See	15
Server	1
Site	1
Software	1
State	1
TaskUmbilicalProtocol,	1
The	54
This	6
Typically	1
Unless	9
Used	2
User	1
Version	9
WARRANTIES	9
WITHOUT	9
We	2
When	2
Whether	1
YARN	2
You	9
ZK	1
[user={name}	1
[u|g]:[name]:[queue_name][,next	1
a	58
access	1
accompanying	5
account.	2
acls	4
active	1
additional	2
admin	2
administer	1
administrators	1
after	5
agreed	9
agreements.	1
all	27
allow	1
allowed	1
allowed.</description>	21
also	1
an	11
and	70
any	1
applicable	11
application	8
applications	5
applications'	1
applications.	1
approximately	1
are	27
as	10
assign	1
assigning	1
assignments	3
at	10
attempt	1
attempts	2
based	1
basis	1
be	16
blank.	21
block	1
by	52
calculated	1
can	9
can't	1
capacity	1
capacity.</description>	1
case,	1
changes	1
client	2
client-to-datanode	1
clients	3
cluster	3
cluster.	1
code	1
collector	1
comma-separated	21
commands	1
commands.	2
communciate	2
communicate	12
communicate.	2
compare	2
compliance	9
concurrent	1
config	1
configuration	1
configuration.	2
configured	3
considered	2
constraint	1
container	1
containers	2
containers,	1
containers.	2
context.	1
contributor	1
controls	1
copy	9
copyright	1
create-key	1
creating	1
datanodes	1
decryptEncryptedKey	1
default	13
default_priority={priority}]	1
defined.	4
delay	1
delete-key	1
dfsadmin	1
different	1
disabled.	2
disables	2
distributed	19
dominant-resource	1
during	2
e.g,	1
e.g.	21
each	7
edit	1
either	9
enabled,	1
encoding="UTF-8"?>	5
equal	2
etc.	3
example,	1
exceed	1
exceeds	2
except	9
explicitly	4
express	9
fail	1
false.	1
feature	2
feature.	1
file	12
file.	10
for	65
from	1
generateEncryptedKey	1
generation	1
generic	1
get-current-key	1
get-key-metadata	1
get-key-version	1
get-keys	1
get-keys-metadata	1
governing	9
group	42
group={name}	1
hard	1
has	1
heartbeat.	1
history	1
hot-reloaded	1
href="configuration.xsl"?>	4
http://www.apache.org/licenses/LICENSE-2.0	9
i.e.	2
if	4
ignored,	1
implementation	1
implied.	9
improve	1
in	36
in-effect.	1
information	1
instead	1
inter-datanode	1
into	2
is	76
it	3
job	4
jobs	4
key	7
key.	1
killed	1
killing	1
language	9
law	9
leaf	2
less	2
level	1
license	1
licenses	1
lifetime	6
lifetime.	3
limit	2
limitations	9
list	45
locality	1
locations	1
logs.</description>	1
longer	1
low	1
manage	1
map	2
mapping	1
mapping]*	1
mappings	1
mappings.	1
maps	1
masters	1
material	3
max_priority={priority}	1
maximum	3
may	18
means	21
missed	5
more	1
mradmin	1
multi-dimensional	1
name	1
namenode	1
namenode.	2
namenode.</description>	1
names.	21
no	1
node's	1
node-locality-delay	1
node-locality-delay=40	1
nodemanager	2
nodes	1
nodes.	1
not	18
number	6
obtain	9
of	77
off-switch	3
on	12
one	4
ones,	1
ones.	1
only	3
operations	5
operations.	8
opportunities	3
opportunities,	1
opportunities.	1
or	23
other	1
other.	7
over	1
overridden	1
override	1
overrides	5
ownership.	1
parameter,	2
parent	1
part	2
particular	1
pending	1
per	1
percent	1
percentage	1
permissions	9
place	1
point-in-time	2
policy	1
positive	1
present,	1
previous	1
priority.	1
properties	1
property	5
protocol	4
protocol,	2
provide	1
query	1
queue	8
queue).	1
queue.	7
queues	4
queues,	1
rack-local	3
rack-locality-delay=20,	1
rack.	1
rate	1
recovery	1
recovery.	1
reduce	2
refresh	2
refreshable.	1
regarding	1
request	1
request,	1
required	9
resource	1
resources	2
response.	2
result	1
returned	2
rolling	1
rollover-key	1
root	1
run	1
running	1
running.	1
same	1
schedule	2
scheduler	1
scheduler.	1
scheduling	3
scheduling.	1
secondary	1
seconds.	2
security	1
separated	21
server	1
service	2
setting	2
should	3
site-specific	5
size	3
software	9
sooner.	1
special	21
specific	10
specified	3
specify	1
stand-by	1
state	1
states	1
status	2
submission	2
submit	2
submitted	4
such	1
syntax	1
taken	2
taken.	1
target	1
tasks	1
tasktracker.	1
than	3
that	7
the	139
then	2
this	28
time	1
timeline	3
timestamp.	1
to	53
too	1
type="text/xsl"	4
u:%user:%user	1
under	28
unique	1
updating	1
use	11
used	24
user	48
user.	2
user?	1
users	24
users,wheel".	21
uses	2
using	1
v2	1
valid.	1
value	30
value,	2
values	2
version="1.0"	5
version="1.0"?>	3
via	1
well	1
when	3
which	10
while	1
who	3
will	12
with	28
work	1
writing,	9
you	10
zero	2
bdkl9998
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
hadoop3自学入门笔记(1)——虚拟机安装和网络配置
01-20
hadoop的视频和书籍在15年的时候就看过，但是一直没动手实践过，要知道技术不经过实战，一点提升也没有。因此下定决心边学边做，希望能有所收获。 软件版本介绍 virtualbox 6.1 centos7 hadoop-3.2.1 jdk-8u241-...
Hadoop简介和集群搭建测试(一)
yanghuadong的博客
03-20 749
Hadoop的优势有四高： (1) 高可用:Hadoop底层维护多个数据副本，所以即使Hadoop某个计算元 素或存储出现故障，也不会导致数据的丢失 (2) 高扩展: 在集群间分配任务数据，可方便的扩展数以千计的节点 (3) 高效性:在MapReduce的思想下，Hadoop是并行工作的，以加快任务处 理速度 (4) 高容错性:能够自动将失败的任务重新分配。 Hadoop组成： HDFS 架构概述: Hadoop Distributed File System，简称 HDFS，是...
大数据技术之Hadoop
我从不打没有准备的仗！
02-27 3925
大数据技术之Hadoop
Chapter2 大数据处理架构Hadoop
weixin_45832571的博客
09-15 1150
2.1 Hadoop简介和版本演变 2.1.1 Hadoop简介 Hadoop是Apache软件基金会旗下开源软件，为用户提供高层接口，为用户提供了底层细节透明的分布式基础架构。 Hadoop是基于java语言开发的，具有很好的跨平台性，但是它支持多种语言，包括C、java、Python等。 Hadoop并不是单一的技术，而是多种大数据解决方案的继承体。 Hadoop的两大核心为：分布式文件系统HDFS、MapReduce。 这两大核心共同解决了大数据的两大问题：海量数据的分布式存储、海量数据的分布式处理
一文让你学明白Hadoop《大数据技术之Hadoop》详细知识总结
m0_72418042的博客
01-23 1550
一文让你学明白Hadoop！《大数据技术之Hadoop》详细知识总结
Hadoop入门
weixin_58202386的博客
04-20 2834
一. 概念 二.环境准备 三.Hadoop生产集群搭建 四. 常见问题的解决方法
细细品味Hadoop
07-25
【标题】"细细品味Hadoop"揭示了我们即将探讨的核心主题——Hadoop在Linux环境，特别是CentOS操作系统中的应用和管理。Hadoop是Apache软件基金会开发的一个开源分布式计算框架，它设计用于处理和存储海量数据，是大...
hadoop史上最详尽的实例
06-24
四、Hadoop实战——《Hadoop实战》 《Hadoop实战》这本书提供了丰富的实践教程，涵盖从基础概念到高级技术，包括集群搭建、数据处理、优化策略和故障排查等。书中不仅有详细的步骤指导，还有大量案例分析，帮助读者...
Hadoop 学习笔记·五 —— Hadoop 之 HDFS
un357951的博客
03-26 435
华为云ID：hw68610019 第四章笔记 HDFS HDFS 概述 随着数据量越来越大，在一个操作系统中存不下所有的数据，那么就需要分配到更多的操作系统管理的磁盘中，但是不方便管理和维护，因此需要一种系统来管理多台机器上的文件，这就是 分布式文件管理系统，HDFS 只是分布式文件管理系统中的一种。 HDFS 概述 HDFS（Hadoop Distributed File System），是一个文件系统，用来存储文件，通过目录树来定位文件；其次，它是分布式的，由很多服务器联合起来实现其功能，集群中的服务器
大数据技术之 Hadoop（MapReduce）
HelloWowofei的博客
07-22 772
大数据技术之 Hadoop（MapReduce） —————————————————————————— 第 1 章 MapReduce 概述 1.1 MapReduce 定义 MapReduce 是一个分布式运算程序的编程框架，是用户开发“基于 Hadoop 的数据分析 应用”的核心框架。 MapReduce 核心功能是将用户编写的业务逻辑代码和自带默认组件整合成一个完整的 分布式运算程序，并发运行在一个 Hadoop 集群上。 1.2 MapReduce 优缺点
idea开发过程spark运行报权限错误
weixin_37796929的博客
05-20 1088
19/05/20 10:11:44 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 19/05/20 10:11:44 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders u...
hadoop的介绍
zhou920786312的博客
01-01 861
1HADOOP简介   什么是HADOOP HADOOP可以对海量数据进行分布式处理 HADOOP的核心组件有 HDFS（分布式文件系统） YARN（运算资源调度系统） MAPREDUCE（分布式运算编程框架） 广义上来说，HADOOP通常是指一个更广泛的概念——HADOOP生态圈 HADOOP在大数据、云计算中的位置和关系 云计算是分布式计算、并行计算、网格计算、多核计...
Sqoop 使用详解
zhouzhiwengang的专栏
03-02 2552
Sqoop 是Apache 旗下的一款开源工具，用于Hadoop与关系型数据库之间传送数据，其核心功能有两个：导入数据和导出数据。导入数据是指将MySQL、Oracle等关系型数据库导入Hadoop的HDFS、Hive、HBase等数据存储系统；导出数据是指将Hadoop文件系统中的数据导出到MySQL、Oracle等关系型数据库。Sqoop 本质是一个命令行工具，与HDFS、Hive、MySQL经常一起使用。
Hadoop大数据简介
热门推荐
只是甲的博客
12-01 1万+
一.Hadoop概述 1.1 Hadoop简要概述 我们生活在这个数据大爆炸的时代 ，很难估算全球电子设备中存储的数据总共有多少 。当前一个中小型公司的数据量也达到数十TB，甚至更多。 有句话说得好 ：“ 大数据胜于好算法 。” 意思是说对于某些应用 （譬如根据以往的偏好来推荐电影和音乐），不论算法有多牛 ，基于小数据的推荐效果往往都不如基于大量可用数据的 一般算法的推荐效果 。 我们遇到的问题很简单：在硬盘存储容量多年来不断提升的同时，访问速 度（硬盘数据读取速度）却没有与时俱进 。 第二个问题是大多数分
大数据技术之Hadoop-大数据分布式
最新发布
qq_33934592的博客
03-01 555
狭义上：大数据是一类技术栈，是一种用来处理海量数据的软件技术体系。广义上：大数据是数字化时代、信息化时代的基础（技术）支撑，以数据为生活赋能。
[雪峰磁针石博客]大数据Hadoop工具python教程3-MapReduce ...
weixin_34060299的博客
01-28 478
MapReduce是一种编程模型，通过将工作分成独立的任务并在一组机器上并行执行任务，可以处理和生成大量数据。 MapReduce编程风格的灵感来自函数式编程结构map和reduce，它们通常用于处理数据列表。在高层MapReduce程序将输入数据元素列表转换为输出数据元素列表两次，一次在映射阶段，一次在还原阶段。 本章首先介绍MapReduce编程模型...
hadoop简介
somilong的博客
06-03 1331
转： http://baike.baidu.com/link?url=HwhPVuqqWelWIr0TeSBGPZ5SjoaYb5_Givp9-rJN-PYbSTMlwpECSKvjzLBzUE7hn9VvmhDoKb5NNCPw1pCsTa Hadoop是一个由Apache基金会所开发的分布式系统基础架构。 用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分
Windows下搭建Linux虚拟机与Hadoop伪分布式实战
实验一：大数据实验——熟悉常用的Linux操作与Hadoop操作 在这个实验中，目标是帮助读者理解和掌握在大数据处理领域中至关重要的技能，即Linux操作系统的使用以及Hadoop框架的安装和基本操作。Linux作为大数据处理...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值