1、我们知道map的数量和文件数、文件大小、块大小、以及split大小有关,而reduce的数量跟哪些因素有关呢?
设置mapred.tasktracker.reduce.tasks.maximum的大小可以决定单个tasktracker一次性启动reduce的数目,但是不能决定总的reduce数目。
conf.setNumReduceTasks(4);JobConf对象的这个方法可以用来设定总的reduce的数目,看下Job Counters的统计:
Job Counters
Data-local map tasks=2
Total time spent by all maps waiting after reserving slots (ms)=0
Total time spent by all reduces waiting after reserving slots (ms)=0
SLOTS_MILLIS_MAPS=10695
SLOTS_MILLIS_REDUCES=29502
Launched map tasks=2
Launched reduce tasks=4
确实启动了4个reduce:看下输出:
diegoball@diegoball:~/IdeaProjects/test/build/classes$ hadoop fs -ls /user/diegoball/join_ou1123
11/03/25 15:28:45 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/03/25 15:28:45 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
Found 5 items
-rw-r--r-- 1 diegoball supergroup 0 2011-03-25 15:28 /user/diegoball/join_ou1123/_SUCCESS
-rw-r--r-- 1 diegoball supergroup 124 2011-03-25 15:27 /user/diegoball/join_ou1123/part-00000
-rw-r--r-- 1 diegoball supergroup 0 2011-03-25 15:27 /user/diegoball/join_ou1123/part-00001