Hadoop执行MapReduce作业的过程-案例-计算pi的值

           在安装好了Hadoop集群后，我们是不是马上想去实践一下，做个例子来测试它是否符合我们的要求呢。这样，我们使用系统的例子来测试，也就最常见的求PI 的值。

          首先要启动好集群。hdfs 和yarn相关进程都启动，并且正常，才能执行mapreduce作业，因为执行作业需要调用资源，就必须启动yarn 服务。

      

    以上是正常的。 我们就可以执行mr 作业了。

2. 在hadoop 目录下有 share目录，下面有写好的例子，打包为jar 文件，我们可以直接拿来测试。

 命令:  hadoop  jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar 10 100        

执行结果如下: 日志输出:  

Number of Maps  = 10
Samples per Map = 100
19/01/07 09:45:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0

Starting Job
19/01/07 09:45:52 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
19/01/07 09:45:52 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
19/01/07 09:45:53 INFO input.FileInputFormat: Total input paths to process : 10
19/01/07 09:45:53 INFO mapreduce.JobSubmitter: number of splits:10
19/01/07 09:45:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local127055197_0001
19/01/07 09:45:55 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
19/01/07 09:45:55 INFO mapreduce.Job: Running job: job_local127055197_0001
19/01/07 09:45:55 INFO mapred.LocalJobRunner: OutputCommitter set in config null
19/01/07 09:45:55 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/01/07 09:45:55 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
19/01/07 09:45:55 INFO mapred.LocalJobRunner: Waiting for map tasks
19/01/07 09:45:55 INFO mapred.LocalJobRunner: Starting task: attempt_local127055197_0001_m_000000_0
19/01/07 09:45:55 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/01/07 09:45:55 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/01/07 09:45:55 INFO mapred.MapTask: Processing split: hdfs://slave4:8020/user/root/QuasiMonteCarlo_1546825548256_803866814/in/part0:0+118
19/01/07 09:45:56 INFO mapreduce.Job: Job job_local127055197_0001 running in uber mode : false
19/01/07 09:45:57 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/01/07 09:45:57 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/01/07 09:45:57 INFO mapred.MapTask: soft limit at 83886080
19/01/07 09:45:57 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/01/07 09:45:57 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/01/07 09:45:57 INFO mapreduce.Job:  map 0% reduce 0%
19/01/07 09:45:57 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/01/07 09:45:58 INFO mapred.LocalJobRunner: 
19/01/07 09:45:58 INFO mapred.MapTask: Starting flush of map output
19/01/07 09:45:58 INFO mapred.MapTask: Spilling map output
19/01/07 09:45:58 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
19/01/07 09:45:58 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
19/01/07 09:45:58 INFO mapred.MapTask: Finished spill 0
19/01/07 09:45:58 INFO mapred.Task: Task:attempt_local127055197_0001_m_000000_0 is done. And is in the process of committing
19/01/07 09:45:59 INFO mapred.LocalJobRunner: map
19/01/07 09:45:59 INFO mapred.Task: Task 'attempt_local127055197_0001_m_000000_0' done.
19/01/07 09:45:59 INFO mapred.LocalJobRunner: Finishing task: attempt_local127055197_0001_m_000000_0
19/01/07 09:45:59 INFO mapred.LocalJobRunner: Starting task: attempt_local127055197_0001_m_000001_0
19/01/07 09:45:59 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
19/01/07 09:45:59 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
19/01/07 09:45:59 INFO mapred.MapTask: Processing split: hdfs://slave4:8020/user/root/QuasiMonteCarlo_1546825548256_803866814/in/part1:0+118
19/01/07 09:45:59 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/01/07 09:45:59 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/01/07 09:45:59 INFO mapred.MapTask: soft limit at 83886080
19/01/07 09:45:59 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/01/07 09:45:59 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/01/07 09:45:59 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
19/01/07 09:45:59 INFO mapred.LocalJobRunner: 
19/01/07 09:45:59 INFO mapred.MapTask: Starting flush of map output<