根据Maneesh Varshney的漫画改编,以简洁易懂的漫画形式讲解HDFS存储机制与运行原理。
一、角色出演
![角色](http://www.36dsj.com/wp-content/uploads/2016/02/765_1.png)
Client:客户端,系统使用者,调用HDFS API操作文件;与NN交互获取文件元数据;与DN交互进行数据读写。
Namenode:元数据节点,是系统唯一的管理者。负责元数据的管理;与client交互进行提供元数据查询;分配数据存储节点等。
Datanode:数据存储节点,负责数据块的存储与冗余备份;执行数据块的读写操作等。
二、写入数据
1、发送写数据请求
![发送写数据请求](http://www.36dsj.com/wp-content/uploads/2016/02/766_1.png)
2、文件切分
![文件切割](http://www.36dsj.com/wp-content/uploads/2016/02/767_1.png)
![DN分配1](http://www.36dsj.com/wp-content/uploads/2016/02/768_1.png)
![DN分配2](http://www.36dsj.com/wp-content/uploads/2016/02/769_1.png)
![数据写入1](http://www.36dsj.com/wp-content/uploads/2016/02/770_1.png)
![数据写入2](http://www.36dsj.com/wp-content/uploads/2016/02/771_1.png)
![完成写入1](http://www.36dsj.com/wp-content/uploads/2016/02/772_1.png)
![完成写入2](http://www.36dsj.com/wp-content/uploads/2016/02/773_1.png)
![完成写入3](http://www.36dsj.com/wp-content/uploads/2016/02/774_1.png)
![角色定位](http://www.36dsj.com/wp-content/uploads/2016/02/775_1.png)
三、HDFS读文件
1、用户需求
![用户需求](http://www.36dsj.com/wp-content/uploads/2016/02/776_1.png)
2、先联系元数据节点
![联系元数据节点1](http://www.36dsj.com/wp-content/uploads/2016/02/777_1.png)
![联系元数据节点2](http://www.36dsj.com/wp-content/uploads/2016/02/778_1.png)
![联系元数据节点2](http://www.36dsj.com/wp-content/uploads/2016/02/779_1.png)
![下载数据](http://www.36dsj.com/wp-content/uploads/2016/02/780_1.png)
4、思考
![思考](http://www.36dsj.com/wp-content/uploads/2016/02/781_1.png)
四、HDFS容错机制——第一部分:故障类型及监测方法
1、三类故障
(1)第一类:节点失败
![节点失败](http://www.36dsj.com/wp-content/uploads/2016/02/782_1.png)
![网络故障](http://www.36dsj.com/wp-content/uploads/2016/02/783_1.png)
![数据损坏](http://www.36dsj.com/wp-content/uploads/2016/02/784_1.png)
(1)节点失败监测机制
![节点失败检测机制1](http://www.36dsj.com/wp-content/uploads/2016/02/785_1.png)
![节点失败检测机制2](http://www.36dsj.com/wp-content/uploads/2016/02/786_1.png)
![节点失败检测机制3](http://www.36dsj.com/wp-content/uploads/2016/02/787_1.png)
![通信故障检测机制](http://www.36dsj.com/wp-content/uploads/2016/02/788_1.png)
![数据错误检测机制1](http://www.36dsj.com/wp-content/uploads/2016/02/789_1.png)
![数据错误检测机制2](http://www.36dsj.com/wp-content/uploads/2016/02/790_1.png)
![数据错误检测机制3](http://www.36dsj.com/wp-content/uploads/2016/02/791_1.png)
![回顾:心跳信息与数据块报告](http://www.36dsj.com/wp-content/uploads/2016/02/792_1.png)
五、容错第二部分:读写容错
1、写容错
![写容错1](http://www.36dsj.com/wp-content/uploads/2016/02/793_1.png)
![写容错2](http://www.36dsj.com/wp-content/uploads/2016/02/794_1.png)
![写容错3](http://www.36dsj.com/wp-content/uploads/2016/02/795_1.png)
![写容错4](http://www.36dsj.com/wp-content/uploads/2016/02/796_1.png)
![读容错](http://www.36dsj.com/wp-content/uploads/2016/02/797_1.png)
![读容错2](http://www.36dsj.com/wp-content/uploads/2016/02/798_1.png)
六、容错第三部分:数据节点(DN)失效
![数据节点(DN)失效1](http://www.36dsj.com/wp-content/uploads/2016/02/799_1.png)
![数据节点(DN)失效2](http://www.36dsj.com/wp-content/uploads/2016/02/800_1.png)
![数据节点(DN)失效3](http://www.36dsj.com/wp-content/uploads/2016/02/811_1.png)
![数据节点(DN)失效4](http://www.36dsj.com/wp-content/uploads/2016/02/812_1.png)
![数据节点(DN)失效5](http://www.36dsj.com/wp-content/uploads/2016/02/813_1.png)
![数据节点(DN)失效6](http://www.36dsj.com/wp-content/uploads/2016/02/814_1.png)
七、备份规则
![备份规则](http://www.36dsj.com/wp-content/uploads/2016/02/815_1.png)
![1、机架与数据节点](http://www.36dsj.com/wp-content/uploads/2016/02/816_1.png)
![2、副本放置策略](http://www.36dsj.com/wp-content/uploads/2016/02/817_1.png)
如果客户端上没有一个数据节点的话,则从整个集群中随机选择一个合适的数据节点作为此时这个数据块的本地节点。
![数字节点放置](http://www.36dsj.com/wp-content/uploads/2016/02/818_1.png)
这样集群可在完全失去某一机架的情况下还能存活。同时,这种策略减少了机架间的数据传输,提高了写操作的效率,因为数据块只存放在两个不同的机架上,减少了读取数据时需要的网络传输总带宽。这样在一定程度上兼顾了数据安全和网络传输的开销。
![DN节点选取](http://www.36dsj.com/wp-content/uploads/2016/02/819_1.png)
![细则](http://www.36dsj.com/wp-content/uploads/2016/02/820_1.png)