Chapter 3 The Hadoop distributed Filesystem
Design:
1. Very Large Files
2. Streaming Data Access (Write once, read-many-times)
3. Commodity Hardware
NOT GOOD FIT
1. Low-latency data access(HBase is currently a better choice for low-latency)
2. Lots of small files
3. Multiple writers, arbitrary file modification(No support for multiple writters? Why? Reliabilit?)
HDFS Concepts
1. Blocks
Unlike a filesystem for a single disk, a file in HDFS that is much smaller than a single block not occupy a full block's worth underlying storage.
Why HDFS block large? minize the time of seek
2. Namenodes and Datanodes
Namenode(master): manage filesystem namespace, maintain filesystem tree and its metadata(persistently), blocks location(in memory, reconstruct)
Datanode(slave): blocks
Namenode(master) Resilient Mechanism:
a) Back up filesystem and metadata
b) secondary namenode
3. Block Caching
4. HDFS Federation
Problem: files number limit to single node.
Solution: cluster of namenodes. ex: NodeA for /user, NodeB for /etc
5. HDFS High Availability
http://blog.cloudera.com/blog/2012/03/high-availability-for-the-hadoop-distributed-file-system-hdfs/
The Command Line interface
Hadoop Filesystems
Hadoop has an abstract notion of filesystem, of which HDFS is just one implementation.
其他的包括:S3、Azure、FTP等。WebHDFS可以使用Http的方式访问文件系统
Data Flow
写时的容错处理