Hadoop Distributed File System

HDFS 概述

1. 分布式
2. commodity hardware
3. fault-tolerant 容错
4. high throughput 高吞吐
5. large data sets

HDFS前提和设计目标 

      Hardware Failure 硬件错误

     每个机器只存储文件的部分数据，blocksize=128M，block存放在不同服务器，默认3副本机制

     Streaming Data Access 流式数据访问

     The emphasis is on high throughput of data access rather than low latency of data access.

     Large Data Sets

     数据大不怕，怕数据小

     Moving Computation is Cheaper than Moving Data 移动计算比移动数据更划算

 

HDFS架构    *****

1. NameNode（master） and DataNodes
2. master/slave的架构
3. NN: the file system namspace ;   regulates access to files by clients
4. DN: storage 
5. HDFS exposes a file system namespace and allows user data to be stored in files
6. a file is split into one or more blocks
7. blocks are stored in a set of DataNodes   （容错）
8. NameNode executes file system namespace operations: CRUD
9.  NameNode  determines the mapping of blocks to DataNodes (决定文件block的映射,用户不感知）