Read note of HDFS User Guide

CheckPoint Node:

The Checkpoint node's memory requirements are on the same order as the NameNode. The Checkpoint node is started by (execute on checkpoint node)

bin/hdfs namenode -checkpoint 

two configuration parameters

  • dfs.namenode.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints
  • dfs.namenode.checkpoint.txns, set to 1 million by default, defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not been reached.

Backup Node:

As the Backup node maintains a copy of the namespace in memory,its RAM requirements are the same as the NameNode.

The NameNode supports one Backup node at a time.No Checkpoint nodes may be registered if a Backup node is in use

The Backup node is started by (execute on Backup node)

bin/hdfs namenode -backup

Import CheckPoint

The latest checkpoint can be imported to the NameNode if all other copies of the image and the edits files are lost. In order to do that one should:

  • Create an empty directory specified in the dfs.namenode.name.dir configuration variable;
  • Specify the location of the checkpoint directory in the configuration variable dfs.namenode.checkpoint.dir;
  • and start the NameNode with -importCheckpoint option.


Balancer all datanode

To start: 
       bin/hadoop-daemon.sh start balancer [-threshold <threshold>] 
       Example: 

bin/hadoop-daemon.sh start balancer
start the balancer with a default threshold of 10% 
bin/hadoop-daemon.sh start balancer -threshold 5
start the balancer with a threshold of 5% 

To stop: 

bin/hadoop-daemon.sh stop balancer

Recovery mode

However, what can you do if the only storage locations available are corrupt? In this case, there is a special NameNode startup mode called Recovery mode that may allow you to recover most of your data.

You can start the NameNode in recovery mode like so: namenode -recover

Recovery mode can cause you to lose data, you should always back up your edit log and fsimage before using it.


HDFS Upgrade and RollBack

Before upgrading, administrators need to remove existing backup using bin/hadoop dfsadmin -finalizeUpgrade command. The following briefly describes the typical upgrade procedure:

  • Before upgrading Hadoop software, finalize if there an existing backup. dfsadmin -upgradeProgress status can tell if the cluster needs to be finalized.
  • Stop the cluster and distribute new version of Hadoop.
  • Run the new version with -upgrade option (bin/start-dfs.sh -upgrade).
  • Most of the time, cluster works just fine. Once the new HDFS is considered working well (may be after a few days of operation), finalize the upgrade.Note that until the cluster is finalized, deleting the files that existed before the upgrade does not free up real disk space on the DataNodes.
  • If there is a need to move back to the old version,
    • stop the cluster and distribute earlier version of Hadoop.
    • start the cluster with rollback option. (bin/start-dfs.h -rollback).









 

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值