【Hadoop高级】Hadoop HA、hdfs安全模式

Hadoop HA

Safemode（安全模式）

During start up the NameNode loads the file system state from the fsimage and the edits log file. It then waits for DataNodes to report their blocks so that it does not prematurely start replicating the blocks though enough replicas already exist in the cluster. During this time NameNode stays in Safemode. Safemode for the NameNode is essentially a read-only mode for the HDFS cluster, where it does not allow any modifications to file system or blocks. Normally the NameNode leaves Safemode automatically after the DataNodes have reported that most file system blocks are available. If required, HDFS could be placed in Safemode explicitly using bin/hdfs dfsadmin -safemode command. NameNode front page shows whether Safemode is on or off. A more detailed description and configuration is maintained as JavaDoc for setSafeMode().

在启动期间，NameNode从fsimage和edits日志文件加载文件系统状态。然后，它等待DataNodes报告其块，以便尽管群集中已存在足够的副本，但它不会过早地开始复制块。在此期间，NameNode保持在Safemode中。NameNode的安全模式本质上是HDFS集群的只读模式，它不允许对文件系统或块进行任何修改。通常，在DataNode报告大多数文件系统块可用之后，NameNode会自动离开Safemode。如果需要，可以使用bin / hdfs dfsadmin -safemode命令显式地将HDFS置于Safemode中。NameNode首页显示Safemode是打开还是关闭。更详细的描述和配置保持为JavaDoc for setSafeMode（）。

namenode启动过程

1. 开启安全模式
   不允许对文件系统或块进行任何修改
2. 加载fsimage
3. 逐个执行所有Edits文件中的每一条操作将操作合并到fsimage，完成后生成一个空的edits文件
4. 接收datanode发送来的心跳消息和块信息
5. 根据以上信息确定文件系统状态
6. 退出安全模式

退出安全模式的条件

系统达到安全标准
• dfs.namenode.safemode.min.datanodes: 最小可用datanode数量
• dfs.namenode.safemode.threshold-pct: 副本数达到最小要求的block占系统总block数的百分比
• dfs.namenode.safemode.extension: 稳定时间

4.HDFS文件格式