《Hadoop The Definitive Guide》ch08 MapReduce Features

最新推荐文章于 2017-07-29 09:38:27 发布
最新推荐文章于 2017-07-29 09:38:27 发布
阅读量631 收藏
点赞数
分类专栏： Hadoop 文章标签： hadoop mapreduce features output input file
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/nomad2/article/details/7724751
版权

1. 计数器

1) 内置计数器

2) 用户自定义Java计数器

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop jar ch08.jar MaxTemperatureWithCounters input/ncdc/all max-temp
12/07/03 19:53:21 INFO mapred.FileInputFormat: Total input paths to process : 2
12/07/03 19:53:21 INFO mapred.JobClient: Running job: job_201207030133_0002
12/07/03 19:53:22 INFO mapred.JobClient:  map 0% reduce 0%
12/07/03 19:53:37 INFO mapred.JobClient:  map 100% reduce 0%
12/07/03 19:53:49 INFO mapred.JobClient:  map 100% reduce 100%
12/07/03 19:53:54 INFO mapred.JobClient: Job complete: job_201207030133_0002
12/07/03 19:53:54 INFO mapred.JobClient: Counters: 29
12/07/03 19:53:54 INFO mapred.JobClient:   Job Counters 
12/07/03 19:53:54 INFO mapred.JobClient:     Launched reduce tasks=1
12/07/03 19:53:54 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=16305
12/07/03 19:53:54 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/07/03 19:53:54 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/03 19:53:54 INFO mapred.JobClient:     Launched map tasks=2
12/07/03 19:53:54 INFO mapred.JobClient:     Data-local map tasks=2
12/07/03 19:53:54 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=10068
12/07/03 19:53:54 INFO mapred.JobClient:   File Input Format Counters 
12/07/03 19:53:54 INFO mapred.JobClient:     Bytes Read=147972
12/07/03 19:53:54 INFO mapred.JobClient:   File Output Format Counters 
12/07/03 19:53:54 INFO mapred.JobClient:     Bytes Written=18
12/07/03 19:53:54 INFO mapred.JobClient:   FileSystemCounters
12/07/03 19:53:54 INFO mapred.JobClient:     FILE_BYTES_READ=28
12/07/03 19:53:54 INFO mapred.JobClient:     HDFS_BYTES_READ=148184
12/07/03 19:53:54 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=62992
12/07/03 19:53:54 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=18
12/07/03 19:53:54 INFO mapred.JobClient:   TemperatureQuality
12/07/03 19:53:54 INFO mapred.JobClient:     1=13129
12/07/03 19:53:54 INFO mapred.JobClient:     9=1
12/07/03 19:53:54 INFO mapred.JobClient:   Air Temperature Records
12/07/03 19:53:54 INFO mapred.JobClient:     Missing=1
12/07/03 19:53:54 INFO mapred.JobClient:   Map-Reduce Framework
12/07/03 19:53:54 INFO mapred.JobClient:     Map output materialized bytes=34
12/07/03 19:53:54 INFO mapred.JobClient:     Map input records=13130
12/07/03 19:53:54 INFO mapred.JobClient:     Reduce shuffle bytes=34
12/07/03 19:53:54 INFO mapred.JobClient:     Spilled Records=4
12/07/03 19:53:54 INFO mapred.JobClient:     Map output bytes=118161
12/07/03 19:53:54 INFO mapred.JobClient:     Map input bytes=1777168
12/07/03 19:53:54 INFO mapred.JobClient:     Combine input records=13129
12/07/03 19:53:54 INFO mapred.JobClient:     SPLIT_RAW_BYTES=212
12/07/03 19:53:54 INFO mapred.JobClient:     Reduce input records=2
12/07/03 19:53:54 INFO mapred.JobClient:     Reduce input groups=2
12/07/03 19:53:54 INFO mapred.JobClient:     Combine output records=2
12/07/03 19:53:54 INFO mapred.JobClient:     Reduce output records=2
12/07/03 19:53:54 INFO mapred.JobClient:     Map output records=13129

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop jar ch08.jar MissingTemperatureFields job_201207030133_0002
Records with missing temperature fields: 0.01%

2. 排序

对数据进行排序是MapReduce的核心。

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop jar ch08.jar SortDataPreprocessor input/ncdc/all input/ncdc/all-seq
12/07/03 20:55:15 INFO mapred.FileInputFormat: Total input paths to process : 2
12/07/03 20:55:16 INFO mapred.JobClient: Running job: job_201207030133_0003
12/07/03 20:55:17 INFO mapred.JobClient:  map 0% reduce 0%
12/07/03 20:55:30 INFO mapred.JobClient:  map 100% reduce 0%
12/07/03 20:55:35 INFO mapred.JobClient: Job complete: job_201207030133_0003
12/07/03 20:55:35 INFO mapred.JobClient: Counters: 16
12/07/03 20:55:35 INFO mapred.JobClient:   Job Counters 
12/07/03 20:55:35 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=16560
12/07/03 20:55:35 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/07/03 20:55:35 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/03 20:55:35 INFO mapred.JobClient:     Launched map tasks=2
12/07/03 20:55:35 INFO mapred.JobClient:     Data-local map tasks=2
12/07/03 20:55:35 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
12/07/03 20:55:35 INFO mapred.JobClient:   File Input Format Counters 
12/07/03 20:55:35 INFO mapred.JobClient:     Bytes Read=147972
12/07/03 20:55:35 INFO mapred.JobClient:   File Output Format Counters 
12/07/03 20:55:35 INFO mapred.JobClient:     Bytes Written=163409
12/07/03 20:55:35 INFO mapred.JobClient:   FileSystemCounters
12/07/03 20:55:35 INFO mapred.JobClient:     HDFS_BYTES_READ=148184
12/07/03 20:55:35 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=41754
12/07/03 20:55:35 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=163409
12/07/03 20:55:35 INFO mapred.JobClient:   Map-Reduce Framework
12/07/03 20:55:35 INFO mapred.JobClient:     Map input records=13130
12/07/03 20:55:35 INFO mapred.JobClient:     Spilled Records=0
12/07/03 20:55:35 INFO mapred.JobClient:     Map input bytes=1777168
12/07/03 20:55:35 INFO mapred.JobClient:     Map output records=13129
12/07/03 20:55:35 INFO mapred.JobClient:     SPLIT_RAW_BYTES=212

部分排序 

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop jar ch08.jar SortByTemperatureUsingHashPartitioner -D mapred.reduce.tasks=30 input/ncdc/all-seq output-hashsort                                        <
12/07/03 22:28:32 INFO mapred.FileInputFormat: Total input paths to process : 2
12/07/03 22:28:33 INFO mapred.JobClient: Running job: job_201207030133_0004
12/07/03 22:28:34 INFO mapred.JobClient:  map 0% reduce 0%
12/07/03 22:28:47 INFO mapred.JobClient:  map 100% reduce 0%
12/07/03 22:28:59 INFO mapred.JobClient:  map 100% reduce 3%
12/07/03 22:29:02 INFO mapred.JobClient:  map 100% reduce 6%
12/07/03 22:29:08 INFO mapred.JobClient:  map 100% reduce 10%
12/07/03 22:29:11 INFO mapred.JobClient:  map 100% reduce 13%
12/07/03 22:29:23 INFO mapred.JobClient:  map 100% reduce 20%
12/07/03 22:29:32 INFO mapred.JobClient:  map 100% reduce 23%
12/07/03 22:29:38 INFO mapred.JobClient:  map 100% reduce 26%
12/07/03 22:29:41 INFO mapred.JobClient:  map 100% reduce 30%
12/07/03 22:29:47 INFO mapred.JobClient:  map 100% reduce 33%
12/07/03 22:29:56 INFO mapred.JobClient:  map 100% reduce 36%
12/07/03 22:30:02 INFO mapred.JobClient:  map 100% reduce 40%
12/07/03 22:30:05 INFO mapred.JobClient:  map 100% reduce 43%
12/07/03 22:30:11 INFO mapred.JobClient:  map 100% reduce 46%
12/07/03 22:30:14 INFO mapred.JobClient:  map 100% reduce 50%
12/07/03 22:30:23 INFO mapred.JobClient:  map 100% reduce 53%
12/07/03 22:30:29 INFO mapred.JobClient:  map 100% reduce 56%
12/07/03 22:30:35 INFO mapred.JobClient:  map 100% reduce 60%
12/07/03 22:30:38 INFO mapred.JobClient:  map 100% reduce 63%
12/07/03 22:30:44 INFO mapred.JobClient:  map 100% reduce 66%
12/07/03 22:30:47 INFO mapred.JobClient:  map 100% reduce 70%
12/07/03 22:30:59 INFO mapred.JobClient:  map 100% reduce 73%
12/07/03 22:31:02 INFO mapred.JobClient:  map 100% reduce 76%
12/07/03 22:31:08 INFO mapred.JobClient:  map 100% reduce 80%
12/07/03 22:31:11 INFO mapred.JobClient:  map 100% reduce 83%
12/07/03 22:31:17 INFO mapred.JobClient:  map 100% reduce 87%
12/07/03 22:31:23 INFO mapred.JobClient:  map 100% reduce 90%
12/07/03 22:31:32 INFO mapred.JobClient:  map 100% reduce 93%
12/07/03 22:31:35 INFO mapred.JobClient:  map 100% reduce 96%
12/07/03 22:31:41 INFO mapred.JobClient:  map 100% reduce 100%
12/07/03 22:31:46 INFO mapred.JobClient: Job complete: job_201207030133_0004
12/07/03 22:31:46 INFO mapred.JobClient: Counters: 26
12/07/03 22:31:46 INFO mapred.JobClient:   Job Counters 
12/07/03 22:31:46 INFO mapred.JobClient:     Launched reduce tasks=30
12/07/03 22:31:46 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=16282
12/07/03 22:31:46 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/07/03 22:31:46 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/03 22:31:46 INFO mapred.JobClient:     Launched map tasks=2
12/07/03 22:31:46 INFO mapred.JobClient:     Data-local map tasks=2
12/07/03 22:31:46 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=335658
12/07/03 22:31:46 INFO mapred.JobClient:   File Input Format Counters 
12/07/03 22:31:46 INFO mapred.JobClient:     Bytes Read=163409
12/07/03 22:31:46 INFO mapred.JobClient:   File Output Format Counters 
12/07/03 22:31:46 INFO mapred.JobClient:     Bytes Written=180399
12/07/03 22:31:46 INFO mapred.JobClient:   FileSystemCounters
12/07/03 22:31:46 INFO mapred.JobClient:     FILE_BYTES_READ=1882171
12/07/03 22:31:46 INFO mapred.JobClient:     HDFS_BYTES_READ=163635
12/07/03 22:31:46 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=4431596
12/07/03 22:31:46 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=180399
12/07/03 22:31:46 INFO mapred.JobClient:   Map-Reduce Framework
12/07/03 22:31:46 INFO mapred.JobClient:     Map output materialized bytes=1882351
12/07/03 22:31:46 INFO mapred.JobClient:     Map input records=13129
12/07/03 22:31:46 INFO mapred.JobClient:     Reduce shuffle bytes=1278651
12/07/03 22:31:46 INFO mapred.JobClient:     Spilled Records=26258
12/07/03 22:31:46 INFO mapred.JobClient:     Map output bytes=1842641
12/07/03 22:31:46 INFO mapred.JobClient:     Map input bytes=163159
12/07/03 22:31:46 INFO mapred.JobClient:     Combine input records=0
12/07/03 22:31:46 INFO mapred.JobClient:     SPLIT_RAW_BYTES=226
12/07/03 22:31:46 INFO mapred.JobClient:     Reduce input records=13129
12/07/03 22:31:46 INFO mapred.JobClient:     Reduce input groups=116
12/07/03 22:31:46 INFO mapred.JobClient:     Combine output records=0
12/07/03 22:31:46 INFO mapred.JobClient:     Reduce output records=13129
12/07/03 22:31:46 INFO mapred.JobClient:     Map output records=13129

已划分的MapFile文件查找 

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop jar ch08.jar SortByTemperatureToMapFile -D mapred.reduce.tasks=30 input/ncdc/all-seq output-hashmapso>
12/07/03 22:35:53 INFO mapred.FileInputFormat: Total input paths to process : 2
12/07/03 22:35:53 INFO mapred.JobClient: Running job: job_201207030133_0005
12/07/03 22:35:54 INFO mapred.JobClient:  map 0% reduce 0%
12/07/03 22:36:08 INFO mapred.JobClient:  map 100% reduce 0%
12/07/03 22:36:20 INFO mapred.JobClient:  map 100% reduce 3%
12/07/03 22:36:23 INFO mapred.JobClient:  map 100% reduce 6%
12/07/03 22:36:29 INFO mapred.JobClient:  map 100% reduce 10%
12/07/03 22:36:32 INFO mapred.JobClient:  map 100% reduce 13%
12/07/03 22:36:44 INFO mapred.JobClient:  map 100% reduce 20%
12/07/03 22:36:53 INFO mapred.JobClient:  map 100% reduce 23%
12/07/03 22:36:56 INFO mapred.JobClient:  map 100% reduce 26%
12/07/03 22:37:02 INFO mapred.JobClient:  map 100% reduce 30%
12/07/03 22:37:05 INFO mapred.JobClient:  map 100% reduce 33%
12/07/03 22:37:17 INFO mapred.JobClient:  map 100% reduce 40%
12/07/03 22:37:26 INFO mapred.JobClient:  map 100% reduce 43%
12/07/03 22:37:29 INFO mapred.JobClient:  map 100% reduce 46%
12/07/03 22:37:35 INFO mapred.JobClient:  map 100% reduce 50%
12/07/03 22:37:38 INFO mapred.JobClient:  map 100% reduce 53%
12/07/03 22:37:51 INFO mapred.JobClient:  map 100% reduce 60%
12/07/03 22:38:00 INFO mapred.JobClient:  map 100% reduce 63%
12/07/03 22:38:03 INFO mapred.JobClient:  map 100% reduce 66%
12/07/03 22:38:09 INFO mapred.JobClient:  map 100% reduce 70%
12/07/03 22:38:12 INFO mapred.JobClient:  map 100% reduce 73%
12/07/03 22:38:18 INFO mapred.JobClient:  map 100% reduce 74%
12/07/03 22:38:21 INFO mapred.JobClient:  map 100% reduce 77%
12/07/03 22:38:24 INFO mapred.JobClient:  map 100% reduce 80%
12/07/03 22:38:33 INFO mapred.JobClient:  map 100% reduce 83%
12/07/03 22:38:36 INFO mapred.JobClient:  map 100% reduce 86%
12/07/03 22:38:42 INFO mapred.JobClient:  map 100% reduce 90%
12/07/03 22:38:45 INFO mapred.JobClient:  map 100% reduce 93%
12/07/03 22:38:57 INFO mapred.JobClient:  map 100% reduce 100%
12/07/03 22:39:02 INFO mapred.JobClient: Job complete: job_201207030133_0005
12/07/03 22:39:02 INFO mapred.JobClient: Counters: 26
12/07/03 22:39:02 INFO mapred.JobClient:   Job Counters 
12/07/03 22:39:02 INFO mapred.JobClient:     Launched reduce tasks=30
12/07/03 22:39:02 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=16299
12/07/03 22:39:02 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/07/03 22:39:02 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/03 22:39:02 INFO mapred.JobClient:     Launched map tasks=2
12/07/03 22:39:02 INFO mapred.JobClient:     Data-local map tasks=2
12/07/03 22:39:02 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=330354
12/07/03 22:39:02 INFO mapred.JobClient:   File Input Format Counters 
12/07/03 22:39:02 INFO mapred.JobClient:     Bytes Read=163409
12/07/03 22:39:02 INFO mapred.JobClient:   File Output Format Counters 
12/07/03 22:39:02 INFO mapred.JobClient:     Bytes Written=186935
12/07/03 22:39:02 INFO mapred.JobClient:   FileSystemCounters
12/07/03 22:39:02 INFO mapred.JobClient:     FILE_BYTES_READ=1882171
12/07/03 22:39:02 INFO mapred.JobClient:     HDFS_BYTES_READ=163635
12/07/03 22:39:02 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=4431532
12/07/03 22:39:02 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=186935
12/07/03 22:39:02 INFO mapred.JobClient:   Map-Reduce Framework
12/07/03 22:39:02 INFO mapred.JobClient:     Map output materialized bytes=1882351
12/07/03 22:39:02 INFO mapred.JobClient:     Map input records=13129
12/07/03 22:39:02 INFO mapred.JobClient:     Reduce shuffle bytes=1054827
12/07/03 22:39:02 INFO mapred.JobClient:     Spilled Records=26258
12/07/03 22:39:02 INFO mapred.JobClient:     Map output bytes=1842641
12/07/03 22:39:02 INFO mapred.JobClient:     Map input bytes=163159
12/07/03 22:39:02 INFO mapred.JobClient:     Combine input records=0
12/07/03 22:39:02 INFO mapred.JobClient:     SPLIT_RAW_BYTES=226
12/07/03 22:39:02 INFO mapred.JobClient:     Reduce input records=13129
12/07/03 22:39:02 INFO mapred.JobClient:     Reduce input groups=116
12/07/03 22:39:02 INFO mapred.JobClient:     Combine output records=0
12/07/03 22:39:02 INFO mapred.JobClient:     Reduce output records=13129
12/07/03 22:39:02 INFO mapred.JobClient:     Map output records=13129

全局排序 

>> hadoop jar ch08.jar SortByTemperatureUsingTotalOrderPartitioner -D mapred.reduce.tasks=30 input/ncdc/all-seq output-totalsort                                     <
12/07/03 23:35:45 INFO mapred.FileInputFormat: Total input paths to process : 2
12/07/03 23:35:45 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/07/03 23:35:45 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
12/07/03 23:35:45 INFO compress.CodecPool: Got brand-new decompressor
12/07/03 23:35:45 INFO compress.CodecPool: Got brand-new decompressor
12/07/03 23:35:45 INFO compress.CodecPool: Got brand-new decompressor
12/07/03 23:35:45 INFO compress.CodecPool: Got brand-new decompressor
12/07/03 23:35:45 INFO lib.InputSampler: Using 1339 samples
12/07/03 23:35:45 INFO compress.CodecPool: Got brand-new compressor
12/07/03 23:35:45 INFO mapred.FileInputFormat: Total input paths to process : 2
12/07/03 23:35:45 INFO mapred.JobClient: Running job: job_201207030133_0006
12/07/03 23:35:46 INFO mapred.JobClient:  map 0% reduce 0%
12/07/03 23:36:01 INFO mapred.JobClient:  map 100% reduce 0%
12/07/03 23:36:13 INFO mapred.JobClient:  map 100% reduce 3%
12/07/03 23:36:16 INFO mapred.JobClient:  map 100% reduce 6%
12/07/03 23:36:25 INFO mapred.JobClient:  map 100% reduce 10%
12/07/03 23:36:28 INFO mapred.JobClient:  map 100% reduce 13%
12/07/03 23:36:37 INFO mapred.JobClient:  map 100% reduce 20%
12/07/03 23:36:49 INFO mapred.JobClient:  map 100% reduce 26%
12/07/03 23:36:58 INFO mapred.JobClient:  map 100% reduce 30%
12/07/03 23:37:01 INFO mapred.JobClient:  map 100% reduce 33%
12/07/03 23:37:10 INFO mapred.JobClient:  map 100% reduce 36%
12/07/03 23:37:16 INFO mapred.JobClient:  map 100% reduce 40%
12/07/03 23:37:19 INFO mapred.JobClient:  map 100% reduce 43%
12/07/03 23:37:25 INFO mapred.JobClient:  map 100% reduce 46%
12/07/03 23:37:31 INFO mapred.JobClient:  map 100% reduce 50%
12/07/03 23:37:40 INFO mapred.JobClient:  map 100% reduce 56%
12/07/03 23:37:49 INFO mapred.JobClient:  map 100% reduce 60%
12/07/03 23:37:52 INFO mapred.JobClient:  map 100% reduce 63%
12/07/03 23:38:01 INFO mapred.JobClient:  map 100% reduce 66%
12/07/03 23:38:04 INFO mapred.JobClient:  map 100% reduce 70%
12/07/03 23:38:13 INFO mapred.JobClient:  map 100% reduce 76%
12/07/03 23:38:22 INFO mapred.JobClient:  map 100% reduce 80%
12/07/03 23:38:25 INFO mapred.JobClient:  map 100% reduce 83%
12/07/03 23:38:34 INFO mapred.JobClient:  map 100% reduce 87%
12/07/03 23:38:37 INFO mapred.JobClient:  map 100% reduce 90%
12/07/03 23:38:40 INFO mapred.JobClient:  map 100% reduce 91%
12/07/03 23:38:46 INFO mapred.JobClient:  map 100% reduce 93%
12/07/03 23:38:49 INFO mapred.JobClient:  map 100% reduce 96%
12/07/03 23:38:58 INFO mapred.JobClient:  map 100% reduce 100%
12/07/03 23:39:03 INFO mapred.JobClient: Job complete: job_201207030133_0006
12/07/03 23:39:03 INFO mapred.JobClient: Counters: 26
12/07/03 23:39:03 INFO mapred.JobClient:   Job Counters 
12/07/03 23:39:03 INFO mapred.JobClient:     Launched reduce tasks=30
12/07/03 23:39:03 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=18040
12/07/03 23:39:03 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/07/03 23:39:03 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/03 23:39:03 INFO mapred.JobClient:     Launched map tasks=2
12/07/03 23:39:03 INFO mapred.JobClient:     Data-local map tasks=2
12/07/03 23:39:03 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=336193
12/07/03 23:39:03 INFO mapred.JobClient:   File Input Format Counters 
12/07/03 23:39:03 INFO mapred.JobClient:     Bytes Read=163409
12/07/03 23:39:03 INFO mapred.JobClient:   File Output Format Counters 
12/07/03 23:39:03 INFO mapred.JobClient:     Bytes Written=177339
12/07/03 23:39:03 INFO mapred.JobClient:   FileSystemCounters
12/07/03 23:39:03 INFO mapred.JobClient:     FILE_BYTES_READ=1882171
12/07/03 23:39:03 INFO mapred.JobClient:     HDFS_BYTES_READ=165067
12/07/03 23:39:03 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=4462828
12/07/03 23:39:03 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=177339
12/07/03 23:39:03 INFO mapred.JobClient:   Map-Reduce Framework
12/07/03 23:39:03 INFO mapred.JobClient:     Map output materialized bytes=1882351
12/07/03 23:39:03 INFO mapred.JobClient:     Map input records=13129
12/07/03 23:39:03 INFO mapred.JobClient:     Reduce shuffle bytes=1138806
12/07/03 23:39:03 INFO mapred.JobClient:     Spilled Records=26258
12/07/03 23:39:03 INFO mapred.JobClient:     Map output bytes=1842641
12/07/03 23:39:03 INFO mapred.JobClient:     Map input bytes=163159
12/07/03 23:39:03 INFO mapred.JobClient:     Combine input records=0
12/07/03 23:39:03 INFO mapred.JobClient:     SPLIT_RAW_BYTES=226
12/07/03 23:39:03 INFO mapred.JobClient:     Reduce input records=13129
12/07/03 23:39:03 INFO mapred.JobClient:     Reduce input groups=116
12/07/03 23:39:03 INFO mapred.JobClient:     Combine output records=0
12/07/03 23:39:03 INFO mapred.JobClient:     Reduce output records=13129
12/07/03 23:39:03 INFO mapred.JobClient:     Map output records=13129

二次排序 

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop jar ch08.jar MaxTemperatureUsingSecondarySort input/ncdc/all output-secondarysort
12/07/03 23:59:15 INFO mapred.FileInputFormat: Total input paths to process : 2
12/07/03 23:59:15 INFO mapred.JobClient: Running job: job_201207030133_0007
12/07/03 23:59:16 INFO mapred.JobClient:  map 0% reduce 0%
12/07/03 23:59:31 INFO mapred.JobClient:  map 100% reduce 0%
12/07/03 23:59:43 INFO mapred.JobClient:  map 100% reduce 100%
12/07/03 23:59:48 INFO mapred.JobClient: Job complete: job_201207030133_0007
12/07/03 23:59:48 INFO mapred.JobClient: Counters: 26
12/07/03 23:59:48 INFO mapred.JobClient:   Job Counters 
12/07/03 23:59:48 INFO mapred.JobClient:     Launched reduce tasks=1
12/07/03 23:59:48 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=16330
12/07/03 23:59:48 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/07/03 23:59:48 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/03 23:59:48 INFO mapred.JobClient:     Launched map tasks=2
12/07/03 23:59:48 INFO mapred.JobClient:     Data-local map tasks=2
12/07/03 23:59:48 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=9967
12/07/03 23:59:48 INFO mapred.JobClient:   File Input Format Counters 
12/07/03 23:59:48 INFO mapred.JobClient:     Bytes Read=147972
12/07/03 23:59:48 INFO mapred.JobClient:   File Output Format Counters 
12/07/03 23:59:48 INFO mapred.JobClient:     Bytes Written=18
12/07/03 23:59:48 INFO mapred.JobClient:   FileSystemCounters
12/07/03 23:59:48 INFO mapred.JobClient:     FILE_BYTES_READ=131296
12/07/03 23:59:48 INFO mapred.JobClient:     HDFS_BYTES_READ=148184
12/07/03 23:59:48 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=326482
12/07/03 23:59:48 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=18
12/07/03 23:59:48 INFO mapred.JobClient:   Map-Reduce Framework
12/07/03 23:59:48 INFO mapred.JobClient:     Map output materialized bytes=131302
12/07/03 23:59:48 INFO mapred.JobClient:     Map input records=13130
12/07/03 23:59:48 INFO mapred.JobClient:     Reduce shuffle bytes=131302
12/07/03 23:59:48 INFO mapred.JobClient:     Spilled Records=26258
12/07/03 23:59:48 INFO mapred.JobClient:     Map output bytes=105032
12/07/03 23:59:48 INFO mapred.JobClient:     Map input bytes=1777168
12/07/03 23:59:48 INFO mapred.JobClient:     Combine input records=0
12/07/03 23:59:48 INFO mapred.JobClient:     SPLIT_RAW_BYTES=212
12/07/03 23:59:48 INFO mapred.JobClient:     Reduce input records=0
12/07/03 23:59:48 INFO mapred.JobClient:     Reduce input groups=2
12/07/03 23:59:48 INFO mapred.JobClient:     Combine output records=0
12/07/03 23:59:48 INFO mapred.JobClient:     Reduce output records=2
12/07/03 23:59:48 INFO mapred.JobClient:     Map output records=13129

3. 连接

4. 次要数据的分布 Side Data Distribution

分布式缓存

相对于在作业配置中对次要数据进行序列化，更好的方法是使用Hadoop的分布式缓存机制来分布数据集。它提供了为该任务及时复制文件和存档文件到任务节点的服务以便在运行时使用它们。为了节省网络带宽，每个作业文件通常复制到任何特定的节点一次。

>> hadoop jar ch08.jar MaxTemperatureByStationNameUsingDistributedCacheFile -files input/ncdc/metadata/stations-fixed-width.txt input/ncdc/all output                               <
12/07/04 00:18:14 INFO mapred.FileInputFormat: Total input paths to process : 2
12/07/04 00:18:14 INFO mapred.JobClient: Running job: job_201207030133_0008
12/07/04 00:18:15 INFO mapred.JobClient:  map 0% reduce 0%
12/07/04 00:18:29 INFO mapred.JobClient:  map 100% reduce 0%
12/07/04 00:18:41 INFO mapred.JobClient:  map 100% reduce 100%
12/07/04 00:18:46 INFO mapred.JobClient: Job complete: job_201207030133_0008
12/07/04 00:18:46 INFO mapred.JobClient: Counters: 26
12/07/04 00:18:46 INFO mapred.JobClient:   Job Counters 
12/07/04 00:18:46 INFO mapred.JobClient:     Launched reduce tasks=1
12/07/04 00:18:46 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=17800
12/07/04 00:18:46 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/07/04 00:18:46 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/04 00:18:46 INFO mapred.JobClient:     Launched map tasks=2
12/07/04 00:18:46 INFO mapred.JobClient:     Data-local map tasks=2
12/07/04 00:18:46 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=10372
12/07/04 00:18:46 INFO mapred.JobClient:   File Input Format Counters 
12/07/04 00:18:46 INFO mapred.JobClient:     Bytes Read=147972
12/07/04 00:18:46 INFO mapred.JobClient:   File Output Format Counters 
12/07/04 00:18:46 INFO mapred.JobClient:     Bytes Written=170
12/07/04 00:18:46 INFO mapred.JobClient:   FileSystemCounters
12/07/04 00:18:46 INFO mapred.JobClient:     FILE_BYTES_READ=234
12/07/04 00:18:46 INFO mapred.JobClient:     HDFS_BYTES_READ=148184
12/07/04 00:18:46 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=66722
12/07/04 00:18:46 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=170
12/07/04 00:18:46 INFO mapred.JobClient:   Map-Reduce Framework
12/07/04 00:18:46 INFO mapred.JobClient:     Map output materialized bytes=240
12/07/04 00:18:46 INFO mapred.JobClient:     Map input records=13130
12/07/04 00:18:46 INFO mapred.JobClient:     Reduce shuffle bytes=120
12/07/04 00:18:46 INFO mapred.JobClient:     Spilled Records=24
12/07/04 00:18:46 INFO mapred.JobClient:     Map output bytes=223193
12/07/04 00:18:46 INFO mapred.JobClient:     Map input bytes=1777168
12/07/04 00:18:46 INFO mapred.JobClient:     Combine input records=13129
12/07/04 00:18:46 INFO mapred.JobClient:     SPLIT_RAW_BYTES=212
12/07/04 00:18:46 INFO mapred.JobClient:     Reduce input records=12
12/07/04 00:18:46 INFO mapred.JobClient:     Reduce input groups=6
12/07/04 00:18:46 INFO mapred.JobClient:     Combine output records=12
12/07/04 00:18:46 INFO mapred.JobClient:     Reduce output records=6
12/07/04 00:18:46 INFO mapred.JobClient:     Map output records=13129


nomad2
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Hadoop The Definitive Guide, 4th Edition.pdf
01-31
The fourth edition covers Hadoop 2 exclusively. The Hadoop 2 release series is the current active release series and contains the most stable versions of Hadoop. There are new chapters covering YARN ...
Hadoop The Definitive Guide
11-27
Haddop The Definitive Guide，O'REILLY出版，英文原版，非扫描
《Hadoop The Definitive Guide》ch11 Pig
行风
07-07 1120
1.  Pig Pig是一种用于探索大型数据集的脚本语言，专门用于数据的批处理。 2. 安装和启动 export HADOOP_INSTALL=/local/nomad2/hadoop/hadoop-0.20.203.0 export PATH=$PATH:$HADOOP_INSTALL/bin export JAVA_HOME=/usr/lib/jvm/java-6-sun
急性子的开源​​大数据，第 1 部分: Hadoop 教程：Hello World 与 Java、Pig、Hive、Flume、Fuse、Oozie，以及 Sqoop 与 Informix、DB2 和
GarfieldEr007的专栏
11-25 2910
如何开始使用 Hadoop 和自己喜欢的数据库 本文的重点是解释大数据，然后在 Hadoop 中提供简单的工作示例，Hadoop 是在大数据领域的主要开源选手。您会很高兴地听到，Hadoop 并不是 Informix® 或 DB2® 的替代品，但事实上与现有的基础架构配合得很好。Hadoop 家族有多个组件，本文将深入探讨演示各功能的具体代码示例。如果您在自己的 PC 上尝试
Open Source Big Data for the Impatient, Part 1
shuhuai007的专栏
12-13 4168
转载：http://www.ibm.com/developerworks/data/library/techarticle/dm-1209hadoopbigdata/ There is a lot of excitement about Big Data and a lot of confusion to go with it. This article will provide a wor
急性子的开源​​大数据，第 1 部分: Hadoop 教程
shuhuai007的专栏
12-13 4154
转载：http://www.ibm.com/developerworks/cn/data/library/techarticle/dm-1209hadoopbigdata/index.html 关于大数据，有很多令人兴奋的事情，但使用它也带来了很多困惑。本文将提供一个可行的大数据定义，然后完成一系列示例，让您可以对在大数据领域领先的开源技术 Hadoop 的一些功能有直接的了解。具体来说，我
Hbase官方文档中文版
热门推荐
blue_jjw的专栏
12-15 2万+
http://abloz.com/hbase/book.html#quickstart 目录 序1. 入门 1.1. 介绍1.2. 快速开始 2. Apache HBase (TM)配置 2.1. 基础条件2.2. HBase 运行模式: 独立和分布式2.3. 配置文件2.4. 配置示例2.5. 重要配置  3. 升级 3.1. 从 0.94.x 升级到 0.
HBase 官方文档中文版
lbyyy的专栏
11-17 4304
HBase 官方文档中文版
hbase0.95 官方文档
cheng911215的专栏
07-31 3040
 Apache HBase™ 参考指南  HBase 官方文档中文版 Copyright © 2012 Apache Software Foundation。保留所有权利。 Apache Hadoop, Hadoop, MapReduce, HDFS, Zookeeper, HBase 及 HBase项目 logo 是Apache Software Fo
hbase官方文档中文版0.97
克终的博客
07-29 2035
hbase官方文档中文版0.97
《大数据》第1期“研究”——大数据管理系统评测基准的挑战与研究进展（下）...
weixin_45585364的博客
06-16 1279
4 BSMA：面向社交媒体数据分析型查询的基准评测4.1 BSMA框架BSMA是一个社交媒体数据分析型查询评测基准[8,9]，它包含了社交媒体数据的形式化描述规范和一...
Hadoop The Definitive Guide, 4th Edition（英文版）.pdf
04-20
《Hadoop The Definitive Guide, 4th Edition》是Hadoop领域的经典之作，由O'Reilly Media出版，为读者提供了全面深入的Hadoop知识体系。这本书是第四版，相较于之前的版本，它更新了Hadoop生态系统的最新发展，并对...
Hadoop The Definitive Guide PDF
12-03
书名：Hadoop The Definitive Guide 语言：英文 The rest of this book is organized as follows. Chapter 2 provides an introduction to MapReduce. Chapter 3 looks at Hadoop filesystems, and in particular ...
《Hadoop The Definitive Guide》ch13 ZooKeeper
行风
07-07 1426
1. 关于ZooKeeper的介绍，http://www.searchtb.com/2011/01/zookeeper-research.html zookeeper是一个开源分布式的服务，它提供了分布式协作、分布式同步、配置管理等功能。 2. 安装和运行 export ZOOKEEPER_INSTALL=/local/honghaos/zookeeper/zookeeper-3.4
《Hadoop The Definitive Guide》ch12 HBase
行风
07-07 1266
1. 关于HBase的介绍，http://www.searchtb.com/2011/01/understanding-hbase.html 很不错。 HBase – Hadoop Database，是一个高可靠性、高性能、面向列、可伸缩的分布式存储系统，利用HBase技术可在廉价PC Server上搭建起大规模结构化存储集群。 HBase是Google Bigtable的开源实现，类似G
《Hadoop The Definitive Guide》ch04 Hadoop I/O
行风
07-07 1150
1. Hadoop comes with a set of primitives for data I/O. Some of these are techniques that are more general than Hadoop, such as data integrity and compression, but deserve special consideration when d
《Hadoop The Definitive Guide》ch02 MapReduce
行风
07-07 1072
1. MapReduce data flow with multiple reduce tasks 2.  Hadoop安装 选择pseudo模式，配置文件如下， >> cat core-site.xml fs.default.name hdfs://localhost/
《Hadoop The Definitive Guide》ch06 How MapReduce Works
行风
07-07 771
1. MapReduce的工作原理 1) 客户端 提交MapReduce作业。 2) jobtracker 协调作业的运行。 jobtracker是一个Java应用程序，它的主类是JobTracker。 3) tasktracker 运行作业划分后的任务。tasktracker是一个Java应用程序，它的主类是TaskTracker。 4) 分布式文件系统（一般为HDFS），用来在其他实体
hadoop the definitive guide epub
最新发布
04-29
"Hadoop: The Definitive Guide"是一本对Hadoop分布式计算框架的相关介绍和实践指南的书籍，也是学习Hadoop的必备工具。这本书从Hadoop最基础的架构和原理开始，详细解析了Hadoop的分布式存储和计算框架，包括HDFS、...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值