hadoop集群动态扩展

Hadoop集群节点的动态增加与删除

Hadoop集群节点的动态增加
1. 安装配置节点
具体过程参考 《Hadoop集群实践 之 (1) Hadoop(HDFS)搭建》

2. 在配置过程中需要在所有的Hadoop服务器上更新以下三项配置
$ sudo vim /etc/hadoop/conf/slaves

hadoop-node-1
hadoop-node-2
hadoop-node-3

$ sudo vim /etc/hosts

10.6.1.150 hadoop-master
10.6.1.151 hadoop-node-1
10.6.1.152 hadoop-node-2
10.6.1.153 hadoop-node-3

$ sudo vim /etc/hadoop/conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
  <name>dfs.data.dir</name>
  <value>/data/hdfs</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
  <name>dfs.datanode.max.xcievers</name>
  <value>4096</value>
</property>
</configuration>

3. 启动datanode与tasktracker
dongguo@hadoop-node-3:~$ sudo /etc/init.d/hadoop-0.20-datanode start
dongguo@hadoop-node-3:~$ sudo /etc/init.d/hadoop-0.20-tasktracker start

4. 检查新增节点是否已经Live
通过WEB管理界面查看

http://10.6.1.150:50070/dfsnodelist.jsp?whatNodes=LIVE

可以看到hadoop-node-3已经被动态添加到了Hadoop集群中

5.应用新的备份系数dfs.replication

5.1 检查目前的备份系数
dongguo@hadoop-master:~$ sudo -u hdfs hadoop fs -lsr /dongguo
-rw-r--r-- 2 hdfs supergroup 33 2012-10-07 22:02 /dongguo/hello.txt

结果行中的第2列是备份系数（注：文件夹信息存储在namenode节点上，没有备份，故文件夹的备份系数是横杠-）
目前文件的备份系数仍是之前设置的参数2，Hadoop不会自动的按照新的备份系数进行调整。

dongguo@hadoop-master:~$ sudo -u hdfs hadoop fsck /

12/10/10 21:18:32 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
FSCK started by hdfs (auth:SIMPLE) from /10.6.1.150 for path / at Wed Oct 10 21:18:33 CST 2012
.................Status: HEALTHY
 Total size:	7786 B
 Total dirs:	46
 Total files:	17
 Total blocks (validated):	17 (avg. block size 458 B)
 Minimally replicated blocks:	17 (100.0 %)
 Over-replicated blocks:	0 (0.0 %)
 Under-replicated blocks