MapReduce工作原理

今天体验了一下mapreduce的运算过程，写出来总结一下。

首先我们要了解两个节点一个是ResourceManager:整个运算过程资源调度（整个）；另一个是NodeManager：每一个节点运算上面资源的管理（个体）

我们通过通过命令sbin/start-yarn.sh 开启集群的资源节点  输入jps查看

hadoop@master:/mysoftware/hadoop-2.7.3/share/hadoop/mapreduce$ jps
1218 SecondaryNameNode
1048 NameNode
1802 Jps
1435 ResourceManager

进入到/mysoftware/hadoop-2.7.3/share/hadoop/mapreduce下有许多jar包  如下：

hadoop@master:/mysoftware/hadoop-2.7.3/share/hadoop/mapreduce$ ls
hadoop-mapreduce-client-app-2.7.3.jar         hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
hadoop-mapreduce-client-common-2.7.3.jar      hadoop-mapreduce-client-shuffle-2.7.3.jar
hadoop-mapreduce-client-core-2.7.3.jar        hadoop-mapreduce-examples-2.7.3.jar
hadoop-mapreduce-client-hs-2.7.3.jar          lib
hadoop-mapreduce-client-hs-plugins-2.7.3.jar  lib-examples
hadoop-mapreduce-client-jobclient-2.7.3.jar   sources

我们输入 ： hadoop jar hadoop-mapreduce-examples-2.7.3.jar  可以查看这个jar包下面有有哪些命令 如下：

An example program must be given as the first argument.
Valid program names are:
  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
  bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
  dbcount: An example job that count the pageview counts from a database.
  distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
  grep: A map/reduce program that counts the matches of a regex in the input.
  join: A job that effects a join over sorted, equally partitioned datasets
  multifilewc: A