HDFS 启动顺序

HDFS 启动顺序

1. Loading fsimage：加载镜像文件

   首次开启集群时，加载镜像文件：{hadoop.tmp.dir}/dfs/name/fsimage_0000000000000000000

   再次开启集群时，Loading fsimage /home/hadoop/tmp/dfs/name/current/fsimage_0000000000000000002

2. Loading edits：加载编辑日志

   首次开启开启集群时，无编辑日志

   再次开启集群时，/home/hadoop/tmp/dfs/name/current/edits_0000000000000000003-0000000000000000003

3. Saving checkpoint：保存检查点

   首次开启集群时，无保存检查点
   保存检查点机制防止因集群长时间运行导致edits_log文件过大，导致集群重启恢复过程过慢或损毁，所做的保护机制。
   检查点机制由secondarynamenode完成。

4. Safe mode：退出安全模式

   安全模式相当于写保护，进入安全模式，则禁止对数据做写入，修改，仅支持读取。

   退出安全模式，才可以对数据进行操作。

调整blocksize

1.调大：

​ [hdfs-site.xml]

<property>
		<name>dfs.blocksize</name>
		<value>256M</value>
</propery>

2.调小：【非要调10字节，怎么办，修改哪些配置文件】

1. [hdfs-size.xml]

   <property>
   		<name>dfs.blocksize</name>
   		<value>10</value>【字节单位】
   </property>
   

2. 集群可以正常启动，但是执行上传指令，报异常：

   》》[hadoop@master hadoop]$ start-dfs.sh
   Starting namenodes on [master]
   master: starting namenode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-namenode-master.out
   slave01: starting datanode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-datanode-slave01.out
   slave02: starting datanode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-datanode-slave02.out
   slave03: starting datanode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-datanode-slave03.out
   Starting secondary namenodes [0.0.0.0]
   0.0.0.0: starting secondarynamenode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-secondarynamenode-master.out

   【集群正常启动】

   》》[hadoop@master hadoop]$ hadoop fs -put ~/tree.txt /mumu/geek
   put: Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 10 < 1048576

   【报错：块大小的最小限制是1M，小于该限制】

3. 修改[hdfs-site.xml]文件中的${dfs.namenode.fs-limits.min-block-size}属性值

   dfs.namenode.fs-limits.min-block-size 1

4. 集群可以正常启动，但是执行上传指令，报异常：

   》》[hadoop@master hadoop]$ start-dfs.sh
   Starting namenodes on [master]
   master: starting namenode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-namenode-master.out
   slave01: starting datanode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-datanode-slave01.out
   slave02: starting datanode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-datanode-slave02.out
   slave03: starting datanode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-datanode-slave03.out
   Starting secondary namenodes [0.0.0.0]
   0.0.0.0: starting secondarynamenode, logging to /home/hadoop/soft/hadoop-2.7.3/logs/hadoop-hadoop-secondarynamenode-master.out

   【集群正常启动】

   》》[hadoop@master ~]$ hadoop fs -put tree.txt /mumu/geek
   -put: Invalid values: dfs.bytes-per-checksum (=512) must divide block size (=10).
   Usage: hadoop fs [generic options] -put [-f] [-p] [-l] …

   【报错：每512字节的块大小创建一个校验和，然而现在设置的块大小为10字节】

5. 修改[hdfs-site.xml]文件中的${dfs.bytes-per-checksum}属性值

   <property>
   		<name>dfs.bytes-per-checksum</name>
   		<value>2</value>
   </property>
   

checkpoint检查点

1.【P317】图

2.触发检查点阈值：

a.每小时触发一次

<property>
	<name>dfs.namenode.checkpoint.period</name>
	<value>3600</value>
	<description>The number of seconds between two periodic checkpoints.</description>
</property>

b.每100万次触发一次

<property>
	<name>dfs.namenode.checkpoint.txns</name>
	<value>1000000</value>
	<description>The Secondary NameNode or CheckpointNode will create a checkpoint of the namespace every 'dfs.namenode.checkpoint.txns' transactions, regardless of whether 'dfs.namenode.checkpoint.period' has expired.</description>
</property>

###　c.每一分钟检查事务id是否为100万次

<property>
	<name>dfs.namenode.checkpoint.check.period</name>
	<value>60</value>
	<description>The SecondaryNameNode and CheckpointNode will poll the NameNode every 'dfs.namenode.checkpoint.check.period' seconds to query the number of uncheckpointed transactions.</description>
</property>

安全模式

1.安全模式开启

$>hdfs dfsadmin -safemode get #获取当前安全模式状态
$>hdfs dfsadmin -safemode enter #进入安全模式状态
$>hdfs dfsadmin -safemode leave #离开安全模式状态
$>hdfs dfsadmin -safemode wait #等待当前安全模式状态

2.手动执行checkpoint,前提需进入到安全模式下

$>hdfs dfsadmin -safemode enter
$>hdfs dfsadmin -saveNamespace
$>hdfs dfsadmin -safemode leave

3.手动滚动日志

$>hdfs dfsadmin -rollEdits