《Hadoop The Definitive Guide》ch04 Hadoop I/O

最新推荐文章于 2018-01-02 15:38:00 发布
最新推荐文章于 2018-01-02 15:38:00 发布
阅读量1.1k 收藏
点赞数
分类专栏： Hadoop 文章标签： hadoop library output up input file
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/nomad2/article/details/7724522
版权
1. Hadoop comes with a set of primitives for data I/O. Some of these are techniques that are more general than Hadoop, such as data integrity and compression, but deserve
special consideration when dealing with multiterabyte datasets. Others are Hadoop tools or APIs that form the building blocks for developing distributed systems, such as

serialization frameworks and on-disk data structures.

2. 压缩

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> echo "text" | hadoop StreamCompressor org.apache.hadoop.io.compress.GzipCodec | gunzip -
12/07/02 00:21:12 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/07/02 00:21:12 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
text

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> echo "text" | hadoop PooledStreamCompressor org.apache.hadoop.io.compress.GzipCodec | gunzip -
12/07/02 00:24:45 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/07/02 00:24:45 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
12/07/02 00:24:45 INFO compress.CodecPool: Got brand-new compressor
text

在MapReduce中使用压缩

3. 序列化

序列化指的是将结构化对象转为字节流以便于通过网络进行传输或写入持久存储的过程。反序列化指的是将字节流转为一系列结构化对象的过程。

序列化用于分布式数据处理中两个截然不同的领域：进程间通信和持久存储。

Hadoop中，节点之间的进程间通信是通过RPC来实现的。

几个序列化框架 Apache Thrift和Google的 Protocol Buffers，Avro。

4. 基于文件的数据结构

4.1 SequenceFileDemo

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop SequenceFileWriteDemo numbers.seq
12/07/02 01:11:00 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/07/02 01:11:00 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
12/07/02 01:11:00 INFO compress.CodecPool: Got brand-new compressor
[128]   100     One, two, buckle my shoe
[173]   99      Three, four, shut the door
[220]   98      Five, six, pick up sticks
[264]   97      Seven, eight, lay them straight
[314]   96      Nine, ten, a big fat hen
[359]   95      One, two, buckle my shoe
[404]   94      Three, four, shut the door
[451]   93      Five, six, pick up sticks
[495]   92      Seven, eight, lay them straight
[545]   91      Nine, ten, a big fat hen
[590]   90      One, two, buckle my shoe
[635]   89      Three, four, shut the door
[682]   88      Five, six, pick up sticks
[726]   87      Seven, eight, lay them straight
[776]   86      Nine, ten, a big fat hen
[821]   85      One, two, buckle my shoe
[866]   84      Three, four, shut the door
[913]   83      Five, six, pick up sticks
[957]   82      Seven, eight, lay them straight
[1007]  81      Nine, ten, a big fat hen
[1052]  80      One, two, buckle my shoe
[1097]  79      Three, four, shut the door
[1144]  78      Five, six, pick up sticks
[1188]  77      Seven, eight, lay them straight
[1238]  76      Nine, ten, a big fat hen
[1283]  75      One, two, buckle my shoe
[1328]  74      Three, four, shut the door
[1375]  73      Five, six, pick up sticks
[1419]  72      Seven, eight, lay them straight
[1469]  71      Nine, ten, a big fat hen
[1514]  70      One, two, buckle my shoe
[1559]  69      Three, four, shut the door
[1606]  68      Five, six, pick up sticks
[1650]  67      Seven, eight, lay them straight
[1700]  66      Nine, ten, a big fat hen
[1745]  65      One, two, buckle my shoe
[1790]  64      Three, four, shut the door
[1837]  63      Five, six, pick up sticks
[1881]  62      Seven, eight, lay them straight
[1931]  61      Nine, ten, a big fat hen
[1976]  60      One, two, buckle my shoe
[2021]  59      Three, four, shut the door
[2088]  58      Five, six, pick up sticks
[2132]  57      Seven, eight, lay them straight
[2182]  56      Nine, ten, a big fat hen
[2227]  55      One, two, buckle my shoe
[2272]  54      Three, four, shut the door
[2319]  53      Five, six, pick up sticks
[2363]  52      Seven, eight, lay them straight
[2413]  51      Nine, ten, a big fat hen
[2458]  50      One, two, buckle my shoe
[2503]  49      Three, four, shut the door
[2550]  48      Five, six, pick up sticks
[2594]  47      Seven, eight, lay them straight
[2644]  46      Nine, ten, a big fat hen
[2689]  45      One, two, buckle my shoe
[2734]  44      Three, four, shut the door
[2781]  43      Five, six, pick up sticks
[2825]  42      Seven, eight, lay them straight
[2875]  41      Nine, ten, a big fat hen
[2920]  40      One, two, buckle my shoe
[2965]  39      Three, four, shut the door
[3012]  38      Five, six, pick up sticks
[3056]  37      Seven, eight, lay them straight
[3106]  36      Nine, ten, a big fat hen
[3151]  35      One, two, buckle my shoe
[3196]  34      Three, four, shut the door
[3243]  33      Five, six, pick up sticks
[3287]  32      Seven, eight, lay them straight
[3337]  31      Nine, ten, a big fat hen
[3382]  30      One, two, buckle my shoe
[3427]  29      Three, four, shut the door
[3474]  28      Five, six, pick up sticks
[3518]  27      Seven, eight, lay them straight
[3568]  26      Nine, ten, a big fat hen
[3613]  25      One, two, buckle my shoe
[3658]  24      Three, four, shut the door
[3705]  23      Five, six, pick up sticks
[3749]  22      Seven, eight, lay them straight
[3799]  21      Nine, ten, a big fat hen
[3844]  20      One, two, buckle my shoe
[3889]  19      Three, four, shut the door
[3936]  18      Five, six, pick up sticks
[3980]  17      Seven, eight, lay them straight
[4030]  16      Nine, ten, a big fat hen
[4075]  15      One, two, buckle my shoe
[4140]  14      Three, four, shut the door
[4187]  13      Five, six, pick up sticks
[4231]  12      Seven, eight, lay them straight
[4281]  11      Nine, ten, a big fat hen
[4326]  10      One, two, buckle my shoe
[4371]  9       Three, four, shut the door
[4418]  8       Five, six, pick up sticks
[4462]  7       Seven, eight, lay them straight
[4512]  6       Nine, ten, a big fat hen
[4557]  5       One, two, buckle my shoe
[4602]  4       Three, four, shut the door
[4649]  3       Five, six, pick up sticks
[4693]  2       Seven, eight, lay them straight
[4743]  1       Nine, ten, a big fat hen

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop SequenceFileReadDemo numbers.seq
12/07/02 01:15:49 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/07/02 01:15:49 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
12/07/02 01:15:49 INFO compress.CodecPool: Got brand-new decompressor
[128]   100     One, two, buckle my shoe
[173]   99      Three, four, shut the door
[220]   98      Five, six, pick up sticks
[264]   97      Seven, eight, lay them straight
[314]   96      Nine, ten, a big fat hen
[359]   95      One, two, buckle my shoe
[404]   94      Three, four, shut the door
[451]   93      Five, six, pick up sticks
[495]   92      Seven, eight, lay them straight
[545]   91      Nine, ten, a big fat hen
[590]   90      One, two, buckle my shoe
[635]   89      Three, four, shut the door
[682]   88      Five, six, pick up sticks
[726]   87      Seven, eight, lay them straight
[776]   86      Nine, ten, a big fat hen
[821]   85      One, two, buckle my shoe
[866]   84      Three, four, shut the door
[913]   83      Five, six, pick up sticks
[957]   82      Seven, eight, lay them straight
[1007]  81      Nine, ten, a big fat hen
[1052]  80      One, two, buckle my shoe
[1097]  79      Three, four, shut the door
[1144]  78      Five, six, pick up sticks
[1188]  77      Seven, eight, lay them straight
[1238]  76      Nine, ten, a big fat hen
[1283]  75      One, two, buckle my shoe
[1328]  74      Three, four, shut the door
[1375]  73      Five, six, pick up sticks
[1419]  72      Seven, eight, lay them straight
[1469]  71      Nine, ten, a big fat hen
[1514]  70      One, two, buckle my shoe
[1559]  69      Three, four, shut the door
[1606]  68      Five, six, pick up sticks
[1650]  67      Seven, eight, lay them straight
[1700]  66      Nine, ten, a big fat hen
[1745]  65      One, two, buckle my shoe
[1790]  64      Three, four, shut the door
[1837]  63      Five, six, pick up sticks
[1881]  62      Seven, eight, lay them straight
[1931]  61      Nine, ten, a big fat hen
[1976]  60      One, two, buckle my shoe
[2021*] 59      Three, four, shut the door
[2088]  58      Five, six, pick up sticks
[2132]  57      Seven, eight, lay them straight
[2182]  56      Nine, ten, a big fat hen
[2227]  55      One, two, buckle my shoe
[2272]  54      Three, four, shut the door
[2319]  53      Five, six, pick up sticks
[2363]  52      Seven, eight, lay them straight
[2413]  51      Nine, ten, a big fat hen
[2458]  50      One, two, buckle my shoe
[2503]  49      Three, four, shut the door
[2550]  48      Five, six, pick up sticks
[2594]  47      Seven, eight, lay them straight
[2644]  46      Nine, ten, a big fat hen
[2689]  45      One, two, buckle my shoe
[2734]  44      Three, four, shut the door
[2781]  43      Five, six, pick up sticks
[2825]  42      Seven, eight, lay them straight
[2875]  41      Nine, ten, a big fat hen
[2920]  40      One, two, buckle my shoe
[2965]  39      Three, four, shut the door
[3012]  38      Five, six, pick up sticks
[3056]  37      Seven, eight, lay them straight
[3106]  36      Nine, ten, a big fat hen
[3151]  35      One, two, buckle my shoe
[3196]  34      Three, four, shut the door
[3243]  33      Five, six, pick up sticks
[3287]  32      Seven, eight, lay them straight
[3337]  31      Nine, ten, a big fat hen
[3382]  30      One, two, buckle my shoe
[3427]  29      Three, four, shut the door
[3474]  28      Five, six, pick up sticks
[3518]  27      Seven, eight, lay them straight
[3568]  26      Nine, ten, a big fat hen
[3613]  25      One, two, buckle my shoe
[3658]  24      Three, four, shut the door
[3705]  23      Five, six, pick up sticks
[3749]  22      Seven, eight, lay them straight
[3799]  21      Nine, ten, a big fat hen
[3844]  20      One, two, buckle my shoe
[3889]  19      Three, four, shut the door
[3936]  18      Five, six, pick up sticks
[3980]  17      Seven, eight, lay them straight
[4030]  16      Nine, ten, a big fat hen
[4075*] 15      One, two, buckle my shoe
[4140]  14      Three, four, shut the door
[4187]  13      Five, six, pick up sticks
[4231]  12      Seven, eight, lay them straight
[4281]  11      Nine, ten, a big fat hen
[4326]  10      One, two, buckle my shoe
[4371]  9       Three, four, shut the door
[4418]  8       Five, six, pick up sticks
[4462]  7       Seven, eight, lay them straight
[4512]  6       Nine, ten, a big fat hen
[4557]  5       One, two, buckle my shoe
[4602]  4       Three, four, shut the door
[4649]  3       Five, six, pick up sticks
[4693]  2       Seven, eight, lay them straight
[4743]  1       Nine, ten, a big fat hen

查看写入的内容， 

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop fs -text numbers.seq |less

排序和合并序列文件 

>> hadoop jar /local/nomad2/hadoop/hadoop-0.20.203.0/hadoop-examples-0.20.203.0.jar sort -r 1 \
more?> -inFormat org.apache.hadoop.mapred.SequenceFileInputFormat \
more?> -outFormat org.apache.hadoop.mapred.SequenceFileOutputFormat \
more?> -outKey org.apache.hadoop.io.IntWritable \
more?> -outValue org.apache.hadoop.io.Text \
more?> numbers.seq sorted
Running on 1 nodes to sort from hdfs://localhost/user/nomad2/numbers.seq into hdfs://localhost/user/nomad2/sorted with 1 reduces.
Job started: Mon Jul 02 01:22:26 CST 2012
12/07/02 01:22:26 INFO mapred.FileInputFormat: Total input paths to process : 1
12/07/02 01:22:26 INFO mapred.JobClient: Running job: job_201207012246_0008
12/07/02 01:22:27 INFO mapred.JobClient:  map 0% reduce 0%
12/07/02 01:22:40 INFO mapred.JobClient:  map 100% reduce 0%
12/07/02 01:22:52 INFO mapred.JobClient:  map 100% reduce 100%
12/07/02 01:22:57 INFO mapred.JobClient: Job complete: job_201207012246_0008
12/07/02 01:22:57 INFO mapred.JobClient: Counters: 26
12/07/02 01:22:57 INFO mapred.JobClient:   Job Counters 
12/07/02 01:22:57 INFO mapred.JobClient:     Launched reduce tasks=1
12/07/02 01:22:57 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=16289
12/07/02 01:22:57 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/07/02 01:22:57 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/02 01:22:57 INFO mapred.JobClient:     Launched map tasks=2
12/07/02 01:22:57 INFO mapred.JobClient:     Data-local map tasks=2
12/07/02 01:22:57 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=10069
12/07/02 01:22:57 INFO mapred.JobClient:   File Input Format Counters 
12/07/02 01:22:57 INFO mapred.JobClient:     Bytes Read=6613
12/07/02 01:22:57 INFO mapred.JobClient:   File Output Format Counters 
12/07/02 01:22:57 INFO mapred.JobClient:     Bytes Written=4005
12/07/02 01:22:57 INFO mapred.JobClient:   FileSystemCounters
12/07/02 01:22:57 INFO mapred.JobClient:     FILE_BYTES_READ=3306
12/07/02 01:22:57 INFO mapred.JobClient:     HDFS_BYTES_READ=6868
12/07/02 01:22:57 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=70016
12/07/02 01:22:57 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=4005
12/07/02 01:22:57 INFO mapred.JobClient:   Map-Reduce Framework
12/07/02 01:22:57 INFO mapred.JobClient:     Map output materialized bytes=3312
12/07/02 01:22:57 INFO mapred.JobClient:     Map input records=100
12/07/02 01:22:57 INFO mapred.JobClient:     Reduce shuffle bytes=2811
12/07/02 01:22:57 INFO mapred.JobClient:     Spilled Records=200
12/07/02 01:22:57 INFO mapred.JobClient:     Map output bytes=3100
12/07/02 01:22:57 INFO mapred.JobClient:     Map input bytes=4660
12/07/02 01:22:57 INFO mapred.JobClient:     Combine input records=0
12/07/02 01:22:57 INFO mapred.JobClient:     SPLIT_RAW_BYTES=190
12/07/02 01:22:57 INFO mapred.JobClient:     Reduce input records=100
12/07/02 01:22:57 INFO mapred.JobClient:     Reduce input groups=100
12/07/02 01:22:57 INFO mapred.JobClient:     Combine output records=0
12/07/02 01:22:57 INFO mapred.JobClient:     Reduce output records=100
12/07/02 01:22:57 INFO mapred.JobClient:     Map output records=100
Job ended: Mon Jul 02 01:22:57 CST 2012
The job took 31 seconds.

4.2 MapFile 

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop MapFileWriteDemo numbers.map
12/07/02 01:27:49 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/07/02 01:27:49 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
12/07/02 01:27:49 INFO compress.CodecPool: Got brand-new compressor
12/07/02 01:27:49 INFO compress.CodecPool: Got brand-new compressor


将SequenceFile转化为MapFile 
[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop jar /local/nomad2/hadoop/hadoop-0.20.203.0/hadoop-examples-0.20.203.0.jar sort -r 1 \^J-inFormat>
Running on 1 nodes to sort from hdfs://localhost/user/nomad2/numbers.seq into hdfs://localhost/user/nomad2/numbers.map with 1 reduces.
Job started: Mon Jul 02 01:31:58 CST 2012
12/07/02 01:31:58 INFO mapred.FileInputFormat: Total input paths to process : 1
12/07/02 01:31:58 INFO mapred.JobClient: Running job: job_201207012246_0010
12/07/02 01:31:59 INFO mapred.JobClient:  map 0% reduce 0%
12/07/02 01:32:13 INFO mapred.JobClient:  map 100% reduce 0%
12/07/02 01:32:25 INFO mapred.JobClient:  map 100% reduce 100%
12/07/02 01:32:30 INFO mapred.JobClient: Job complete: job_201207012246_0010
12/07/02 01:32:30 INFO mapred.JobClient: Counters: 26
12/07/02 01:32:30 INFO mapred.JobClient:   Job Counters 
12/07/02 01:32:30 INFO mapred.JobClient:     Launched reduce tasks=1
12/07/02 01:32:30 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=16355
12/07/02 01:32:30 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/07/02 01:32:30 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/02 01:32:30 INFO mapred.JobClient:     Launched map tasks=2
12/07/02 01:32:30 INFO mapred.JobClient:     Data-local map tasks=2
12/07/02 01:32:30 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=10036
12/07/02 01:32:30 INFO mapred.JobClient:   File Input Format Counters 
12/07/02 01:32:30 INFO mapred.JobClient:     Bytes Read=6613
12/07/02 01:32:30 INFO mapred.JobClient:   File Output Format Counters 
12/07/02 01:32:30 INFO mapred.JobClient:     Bytes Written=4005
12/07/02 01:32:30 INFO mapred.JobClient:   FileSystemCounters
12/07/02 01:32:30 INFO mapred.JobClient:     FILE_BYTES_READ=3306
12/07/02 01:32:30 INFO mapred.JobClient:     HDFS_BYTES_READ=6868
12/07/02 01:32:30 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=70031
12/07/02 01:32:30 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=4005
12/07/02 01:32:30 INFO mapred.JobClient:   Map-Reduce Framework
12/07/02 01:32:30 INFO mapred.JobClient:     Map output materialized bytes=3312
12/07/02 01:32:30 INFO mapred.JobClient:     Map input records=100
12/07/02 01:32:30 INFO mapred.JobClient:     Reduce shuffle bytes=3312
12/07/02 01:32:30 INFO mapred.JobClient:     Spilled Records=200
12/07/02 01:32:30 INFO mapred.JobClient:     Map output bytes=3100
12/07/02 01:32:30 INFO mapred.JobClient:     Map input bytes=4660
12/07/02 01:32:30 INFO mapred.JobClient:     Combine input records=0
12/07/02 01:32:30 INFO mapred.JobClient:     SPLIT_RAW_BYTES=190
12/07/02 01:32:30 INFO mapred.JobClient:     Reduce input records=100
12/07/02 01:32:30 INFO mapred.JobClient:     Reduce input groups=100
12/07/02 01:32:30 INFO mapred.JobClient:     Combine output records=0
12/07/02 01:32:30 INFO mapred.JobClient:     Reduce output records=100
12/07/02 01:32:30 INFO mapred.JobClient:     Map output records=100
Job ended: Mon Jul 02 01:32:30 CST 2012
The job took 32 seconds.

[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop fs -mv numbers.map/part-00000 numbers.map/data


[ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ]
>> hadoop MapFileFixer numbers.map
12/07/02 01:33:31 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/07/02 01:33:31 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
12/07/02 01:33:31 INFO compress.CodecPool: Got brand-new compressor
Created MapFile numbers.map with 100 entries

nomad2
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
hadoop深入研究:(八)——codec
独自登高楼 望断天涯路
06-27 1万+
转载请注明出处：http://blog.csdn.net/lastsweetop/article/details/9173061所有源码在github上，https://github.com/lastsweetop/styhadoop简介codec其实就是coder和decoder两个单词的词头组成的缩略词。CompressionCodec定义了压缩和解压接口，我们这里讲的codec就是实现了Co
Spark的spark.sql.warehouse.dir问题
热门推荐
qq_14961401的博客
10-18 1万+
最近学习Spark的MLib，做到NaiveByesExample例子一直报一个错，说是Spark-warehouse路径有问题（都不知道Spark-warehouse这玩意儿是啥） 一开始main下的代码是这样的 def main(args: Array[String]): Unit = { System.setProperty("hadoop.home.dir","D:\\hado
Hadoop文件压缩放到远程centos调试
weixin_30839881的博客
01-02 132
文件压缩 1、准备代码 package com.cr.compress; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.io.compress.*; import org.apache.hado...
hadoop hbase lzo 安装笔记
zjumath的专栏
08-03 3324
hadoop版本 hadoop-0.20.2-cdh3u0 hbase版本 hbase-0.90.1-cdh3u0 1.首先编译安装lzo动态库,从http://www.oberhumer.com/opensource/lzo/download/上下载lz
Hadoop The Definitive Guide, 4th Edition.pdf
01-31
The fourth edition covers Hadoop 2 exclusively. The Hadoop 2 release series is the current active release series and contains the most stable versions of Hadoop. There are new chapters covering YARN ...
Hadoop The Definitive Guide
11-27
Haddop The Definitive Guide，O'REILLY出版，英文原版，非扫描
Hadoop The Definitive Guide, 4th Edition（英文版）.pdf
04-20
《Hadoop The Definitive Guide, 4th Edition》是Hadoop领域的经典之作，由O'Reilly Media出版，为读者提供了全面深入的Hadoop知识体系。这本书是第四版，相较于之前的版本，它更新了Hadoop生态系统的最新发展，并对...
Hadoop The Definitive Guide PDF
12-03
书名：Hadoop The Definitive Guide 语言：英文 The rest of this book is organized as follows. Chapter 2 provides an introduction to MapReduce. Chapter 3 looks at Hadoop filesystems, and in particular ...
hadoop2 （2017-7-21修改）对SequeneceFile 的（读写）操作
machen990519的博客
07-21 413
写操作 根据上一篇的介绍，在hadoop2.x之后，Hadoop中的SequenceFile.Writer将会逐渐摒弃大量的createWriter（）重载方法，而整合为更为简洁的createWriter()方法，除了配置参数外，其他的参数统统使用SequenceFile.Writer.Option来替代，具体有： 新的API里提供的option参数：
《Hadoop The Definitive Guide》ch13 ZooKeeper
行风
07-07 1426
1. 关于ZooKeeper的介绍，http://www.searchtb.com/2011/01/zookeeper-research.html zookeeper是一个开源分布式的服务，它提供了分布式协作、分布式同步、配置管理等功能。 2. 安装和运行 export ZOOKEEPER_INSTALL=/local/honghaos/zookeeper/zookeeper-3.4
《Hadoop The Definitive Guide》ch12 HBase
行风
07-07 1266
1. 关于HBase的介绍，http://www.searchtb.com/2011/01/understanding-hbase.html 很不错。 HBase – Hadoop Database，是一个高可靠性、高性能、面向列、可伸缩的分布式存储系统，利用HBase技术可在廉价PC Server上搭建起大规模结构化存储集群。 HBase是Google Bigtable的开源实现，类似G
《Hadoop The Definitive Guide》ch11 Pig
行风
07-07 1120
1.  Pig Pig是一种用于探索大型数据集的脚本语言，专门用于数据的批处理。 2. 安装和启动 export HADOOP_INSTALL=/local/nomad2/hadoop/hadoop-0.20.203.0 export PATH=$PATH:$HADOOP_INSTALL/bin export JAVA_HOME=/usr/lib/jvm/java-6-sun
《Hadoop The Definitive Guide》ch02 MapReduce
行风
07-07 1072
1. MapReduce data flow with multiple reduce tasks 2.  Hadoop安装 选择pseudo模式，配置文件如下， >> cat core-site.xml fs.default.name hdfs://localhost/
《Hadoop The Definitive Guide》ch06 How MapReduce Works
行风
07-07 771
1. MapReduce的工作原理 1) 客户端 提交MapReduce作业。 2) jobtracker 协调作业的运行。 jobtracker是一个Java应用程序，它的主类是JobTracker。 3) tasktracker 运行作业划分后的任务。tasktracker是一个Java应用程序，它的主类是TaskTracker。 4) 分布式文件系统（一般为HDFS），用来在其他实体
《Hadoop The Definitive Guide》ch05 Developing a MapReduce Application
行风
07-07 753
1. 介绍 MapReduce应用开发包含特定的流程。首先，编写map和reduce函数，最好能进行单元测试以保证它们能如期运行。然后写一个驱动程序来运行作业，可以使用数据集中的少量数据从IDE运行，看它是否能够正常运行。 2. GenericOptionsParser, Tool和ToolRunner [ate: /local/nomad2/hadoop/tomwhite-hadoop
《Hadoop The Definitive Guide》ch14 Case Studies
行风
07-08 656
1. Hive hive是基于Hadoop的一个数据仓库工具，可以将结构化的数据文件映射为一张数据库表，并提供完整的sql查询功能，可以将sql语句转换为MapReduce任务进行运行。 其优点是学习成本低，可以通过类SQL语句快速实现简单的MapReduce统计，不必开发专门的MapReduce应用，十分适合数据仓库的统计分析。 2. Nutch 参考：http://blog.csdn
《Hadoop The Definitive Guide》ch08 MapReduce Features
行风
07-07 631
1. 计数器 1) 内置计数器 2) 用户自定义Java计数器 [ate: /local/nomad2/hadoop/tomwhite-hadoop-book-32dae01 ] >> hadoop jar ch08.jar MaxTemperatureWithCounters input/ncdc/all max-temp 12/07/03 19:53:21 INFO mapred.Fil
hadoop the definitive guide epub
最新发布
04-29
"Hadoop: The Definitive Guide"是一本对Hadoop分布式计算框架的相关介绍和实践指南的书籍，也是学习Hadoop的必备工具。这本书从Hadoop最基础的架构和原理开始，详细解析了Hadoop的分布式存储和计算框架，包括HDFS、MapReduce等。此外，书中还介绍了Hadoop生态系统的其他重要组件，如Hive、Pig等。这本书着重介绍了Hadoop的实用性，提供了实际案例分析和实验。随着大数据时代的到来，Hadoop作为一个重要的分布式计算框架，正在不断逐渐成熟，为我们提供了更多的解决方案。因此，学习和掌握Hadoop框架的相关知识，就显得异常重要和必要。这本书为用户提供了一种较为系统和全面的学习方式，是学习Hadoop技术的一个非常好的参考资料。同时，由于该书存在epub格式的电子版，更易于用户查看和分享。总之，Hadoop: The Definitive Guide epub是一个实用性强的电子书，在Hadoop学习中起到了关键性的作用。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值