Hadoop管理

大部分内容源自hadoop-definitive guide， hadoop in action


（1） Removing data nodes

For a smoother and safer operation, particularly
when retiring large number of DataNodes, you should use Hadoop’s decommissioning
feature. Decommissioning ensures that all blocks will have the desired
replication factor among the remaining active nodes. In order to use this feature,
you must create an (initially empty) exclude file in the NameNode’s local file- system,
and the configuration parameter dfs.hosts.exclude must point to this file during
NameNode’s startup. When you want to retire DataNodes, list them in the exclude
file, one node per line. You have to specify the nodes using the full hostname, IP, or
IP:port format. Execute
bin/hadoop dfsadmin -refreshNodes
to force the NameNode to reread the exclude file and start the decommissioning process.

If you have started HDFS without setting dfs.hosts.exclude to point to an exclude
file, the proper way to decommission DataNodes is this: Shut down the NameNode.
Set dfs.hosts.exclude to point to an empty exclude file. Restart NameNode. After
NameNode has successfully restarted, follow the procedure above.

（2）Adding data nodes

Start the DataNode daemon manually (bin/hadoop
datanode). It will automatically contact the NameNode and join the cluster. You
should also add the new node to the conf/slaves file in the master server.

proactively start the HDFS balancer to balance the cluster for optimal
performance. Run the balancer script:
bin/start-balancer.sh

As balancing can be network intensive, we recommend doing it overnight or over a

weekend when your cluster may be less busy


（3）Managing Secondary NameNode 

By default, the Secondary NameNode3 and the NameNode run on the same
machine. For moderate size clusters (10 or more nodes), you should separate the
Secondary NameNode into its own machine

To configure HDFS to use a separate server as the SNN, first list that server’s host
name or IP address in the conf/masters file.
What’s listed in conf/masters is the SNN, not any of the masters.

You should also modify the conf/hdfs-site.xml file on the SNN such that the dfs.
http.address property points to port 50070 of the NameNode’s host address, like
<property>
<name>dfs.http.address</name>
<value>namenode.hadoop-host.com:50070</value>
</property>

（4）Recovering from a failed NameNode
1） multiple dfs metadata dir
We instruct the NameNode to always write to this mounted directory in addition to
writing to the NameNode’s local metadata directory. HDFS supports this ability to write the
metadata to multiple directories. You have to specify dfs.name.dir on the NameNode
with a comma separated list, like
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/dfs/name,/mnt/hadoop-backup</value>
<final>true</final>
</property>
This works, assuming the local dfs.name.dir directory at both the NameNode and the
Secondary NameNode are at /home/hadoop/dfs/name, and that the directory on the
SNN is mounted to the NameNode at /mnt/hadoop-backup. When HDFS sees a commaseparated
list in dfs.name.dir, it writes its metadata to every directory on the list.

2）switch IP
To have the backup node serve as the replacement NameNode, you’ll have to switch its
IP address to the original NameNode’s IP address.

3）start-dfs
You’ll also have to run the backup node as a NameNode by executing bin/start-dfs.sh on it

4）Set new SNN
To be safer, this new NameNode should also have a backup node set up before you
start it.


（5）Designing network layout and rack awareness

At this point you may wonder how Hadoop knows which rack a node is at. It requires
you to tell it.

To help Hadoop know the location of each node, you have to provide an executable
script that can map IP addresses into rack names. This network topology script must reside
on the master node and its location is specified in the topology.script.file.name
property in core-site.xml.


#!/bin/bash
ipaddr=$1
segments='echo $ipaddr | cut --delimiter=. --fields=4'
if [ "$segments" -lt 128 ]; then
echo /rack-1
else
echo /rack-2
fi

the result of the script:

rack1 192.168.137.1
rack1 192.168.137.2
rack1 192.168.137.16
......       ......
rack2 192.168.137.17


A table lookup may make more
sense for more complex cluster topologies. On the other hand, if there is no network
topology script, Hadoop assumes a flat topology where all nodes are assigned to
/default-rack.



（6）Scheduling jobs from mul-users
This file is typically named pools.xml and specifies
each pool’s name and capacity. The mapred.fairscheduler.poolnameproperty
defines the jobconf property the scheduler will use to determine which pool to use
for a job. A useful configuration pattern is to set this to a new property, say pool.name,
and assign pool.name to have a default value of ${user.name}. The Fair Scheduler
automatically gives each user her own individual pool. This particular pool.name
will by default assign each job to its owner’s pool. You can change the pool.name
property in a job’s jobconf to assign the job to a different pool.7 Finally, the mapred.
fairscheduler.assignmultiple property, when set to true, allows the scheduler to
assign both a map task and a reduce task on each heartbeat, which improves ramp-up
speed and throughput. To summarize, your mapred-site.xml will have the following
properties set: