linux进程map,运行一个真正的MapReduce程序

上次的程序只是操作文件系统，本次运行一个真正的MapReduce程序。

运行的是官方提供的例子程序wordcount，这个例子类似其他程序的hello world。

1. 首先确认启动的正常：运行 start-all.sh

2. 执行jps命令检查：NameNode，DateNode，SecondaryNameNode，ResourceManager，NodeManager是否已经启动正常。这里我遇到了一个问题，NodeManager没有正常启动。错误信息如下：

2014-01-07 13:46:21,442FATAL org.apache.Hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed to initialize mapreduce.shuffle

java.lang.IllegalArgumentException: The ServiceName: mapreduce.shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers

at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:98)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:218)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:188)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:338)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:386)

经过检查，是配置文件中有点错误，请修改yarn-site.xml文件，更改为如下内容(原因不明)

yarn.nodemanager.aux-services

mapreduce_shuffle

3. 准备数据：在hadoop文件系统中增加input/file1.txt和input/file2.txt

[root@dbserver mapreduce]# hadoop fs -ls /input

Found2items-rw-r--r-- 1 root supergroup 12 2013-12-06 16:22 /input/file1.txt-rw-r--r-- 1 root supergroup 13 2013-12-06 16:22 /input/file2.txt

[root@dbserver mapreduce]# hadoop fs -cat /input/file1.txt

Hello World

[root@dbserver mapreduce]# hadoop fs -cat /input/file2.txt

Hello Hadoop

4. 例子程序的位置在：/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar

hadoop jar ./hadoop-mapreduce-examples-2.2.0.jar wordcount /input /output

画面输出内容

14/01/07 14:00:37 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032

14/01/07 14:00:38 INFO input.FileInputFormat: Total input paths to process : 2

14/01/07 14:00:38 INFO mapreduce.JobSubmitter: number of splits:2

14/01/07 14:00:38INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name14/01/07 14:00:38INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar14/01/07 14:00:38INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class14/01/07 14:00:38INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class14/01/07 14:00:38INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class14/01/07 14:00:38INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name14/01/07 14:00:38INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class14/01/07 14:00:38 INFO Configuration.deprecation: mapred.input.diris deprecated. Instead, use mapreduce.input.fileinputformat.inputdir14/01/07 14:00:38 INFO Configuration.deprecation: mapred.output.diris deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir14/01/07 14:00:38INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps14/01/07 14:00:38INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class14/01/07 14:00:38 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir

14/01/07 14:00:38 INFO mapreduce.JobSubmitter: Submitting tokens forjob: job_1389074273046_000114/01/07 14:00:38 INFO impl.YarnClientImpl: Submitted application application_1389074273046_0001 to ResourceManager at localhost/127.0.0.1:8032

14/01/07 14:00:38 INFO mapreduce.Job: The url to track the job: http://dbserver:8088/proxy/application_1389074273046_0001/

14/01/07 14:00:38INFO mapreduce.Job: Running job: job_1389074273046_000114/01/07 14:00:48 INFO mapreduce.Job: Job job_1389074273046_0001 running in uber mode : false

14/01/07 14:00:48 INFO mapreduce.Job: map 0% reduce 0%

14/01/07 14:00:58 INFO mapreduce.Job: map 100% reduce 0%

14/01/07 14:01:04 INFO mapreduce.Job: map 100% reduce 100%

14/01/07 14:01:05INFO mapreduce.Job: Job job_1389074273046_0001 completed successfully14/01/07 14:01:05 INFO mapreduce.Job: Counters: 43File System Counters

FILE: Number of bytes read=55FILE: Number of bytes written=236870FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number ofwrite operations=0HDFS: Number of bytes read=229HDFS: Number of bytes written=25HDFS: Number of read operations=9HDFS: Number of large read operations=0HDFS: Number ofwrite operations=2Job Counters

Launched map tasks=2Launched reduce tasks=1Data-local map tasks=2Totaltime spent by all maps in occupied slots (ms)=15178Totaltime spent by all reduces in occupied slots (ms)=4384Map-Reduce Framework

Map input records=2Map output records=4Map output bytes=41Map output materialized bytes=61Inputsplit bytes=204Combine input records=4Combine output records=4Reduce inputgroups=3Reduce shuffle bytes=61Reduce input records=4Reduce output records=3Spilled Records=8Shuffled Maps=2Failed Shuffles=0Merged Map outputs=2GCtime elapsed (ms)=108CPUtime spent (ms)=2200Physical memory (bytes) snapshot=568229888Virtual memory (bytes) snapshot=2566582272Total committed heap usage (bytes)=392298496Shuffle Errors

BAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format Counters

Bytes Read=25File Output Format Counters

Bytes Written=25

5. 查看运行结果：

[root@dbserver mapreduce]# hadoop fs -ls /output

Found2items-rw-r--r-- 1 root supergroup 0 2014-01-07 14:01 /output/_SUCCESS-rw-r--r-- 1 root supergroup 25 2014-01-07 14:01 /output/part-r-00000[root@dbserver mapreduce]# hadoop fs-cat /output/part-r-00000Hadoop1Hello2World1