hadoop wordcount

1. hadoop自带的wordcount

1.1位置

[root@hadoop001 ~]# ls $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar
/export/servers/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar

1.2 使用

[root@hadoop001 ~]# hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar
An example program must be given as the first argument.
Valid program names are:
  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
  bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
  dbcount: An example job that count the pageview counts from a database.
  distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
  grep: A map/reduce program that counts the matches of a regex in the input.
  join: A job that effects a join over sorted, equally partitioned datasets
  multifilewc: A job that counts words from several files.
  pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
  pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
  randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
  randomwriter: A map/reduce program that writes 10GB of random data per node.
  secondarysort: An example defining a secondary sort to the reduce.
  sort: A map/reduce program that sorts the data written by the random writer.
  sudoku: A sudoku solver.
  teragen: Generate data for the terasort
  terasort: Run the terasort
  teravalidate: Checking results of terasort
  wordcount: A map/reduce program that counts the words in the input files.
  wordmean: A map/reduce program that counts the average length of the words in the input files.
  wordmedian: A map/reduce program that counts the median length of the words in the input files.
  wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.

[root@hadoop001 ~]# hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount
Usage: wordcount <in> [<in>...] <out>

2

2.1. MapReduce的核心思想

MapReduce是一种并行编程模型，是Hadoop生态系统的核心组件之一，“分而治之”是MapReduce的核心思想，它表示把一个大规模的数据集切分成很多小的单独的数据集，然后放在多个机器上同时处理。

我们用一个通俗易懂的例子来体现“分而治之”的思想。

2.2 MapReduce的核心函数

MapReduce把整个并行运算过程高度抽象到两个函数上，一个是map另一个是reduce。Map函数就是分而治之中的“分”，reduce函数就是分而治之中的“治”。

MapReduce把一个存储在分布式文件系统中的大规模数据集切分成许多独立的小的数据集，然后分配给多个map任务处理。然后map任务的输出结果会进一步处理成reduce任务的输入，最后由reduce任务进行汇总，然后上传到分布式文件系统中。

Map函数：map函数会将小的数据集转换为适合输入的<key,value>键值对的形式，然后处理成一系列具有相同key的<key,value>作为输出，我们可以把输出看做list(<key,value>)

Reduce函数：reduce函数会把map函数的输出作为输入，然后提取具有相同key的元素，并进行操作，最后的输出结果也是<key,value>键值对的形式，并合并成一个文件。

2.3 MapReduce的工作过程

如图展示的就是MapReduce的工作过程，一共分为input、split、map、shuffle、reduce、output六个阶段。下面用三明治的例子来演示一下MapReduce的工作流程：

√ input阶段相当于准备食材的步骤；