Hadoop分布式集群配置总结

 

假设用2台机器配置hadoop分布式集群，192.168.11.13为主服务器namenode，192.168.11.17为数据节点datanode 
1. 配置SSH的无密码公钥 
192.168.11.13 
用root登录 
创建用户linleran：adduser linleran 
设置密码：passwd linleran 
切换用户：su linleran 
到用户linleran的目录（/home/linleran）新建文件件.ssh：mkdir .ssh 
修改目录.ssh权限为：[linleran@centos ~]$ chmod 755 .ssh 
配置SSH的无密码公钥，一路回车。 
[linleran@centos ~]$ ssh-keygen -t rsa 
Generating public/private rsa key pair. 
Enter file in which to save the key (/home/linleran/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/linleran/.ssh/id_rsa. 
Your public key has been saved in /home/linleran/.ssh/id_rsa.pub. 
The key fingerprint is: 
df:99:37:84:a1:04:34:06:60:45:b9:ce:43:af:54:77 linleran@centos.test 
进入.ssh目录，将id_rsa.pub的内容复制到authorized_keys后面。 
修改authorized_keys的权限为 [linleran@centos .ssh]$ chmod 644 authorized_keys 

192.168.11.17同样执行以内容，且将id_rsa.pub内容添加到192.168.11.13的authorized_keys后面，同时将192.168.11.13的id_rsa.pub内容添加过来，以确保2台机器相互可以ssh连接。 

2.在主服务器安装hadoop 
解压 tar zxvf hadoop-0.18.2.tar.gz 
创建目录 
/home/linleran/hadoop-0.18.2/hadoopfs/name 
/home/linleran/hadoop-0.18.2/hadoopfs/data 
/home/linleran/hadoop-0.18.2/tmp/ 
配置hadoop-env.sh设置jdk路径 export JAVA_HOME=/home/linleran/jdk1.5.0_15 
配置conf/hadoop-site.xml 

<configuration> 
   <property>   
      <name>fs.default.name</name>  
      <value>hdfs://192.168.11.13:9000/</value> 
   </property> 
   <property> 
      <name>mapred.job.tracker</name> 
      <value>192.168.11.13:9001</value> 
   </property> 
   <property> 
      <name>dfs.name.dir</name> 
      <value>/home/linleran/hadoop-0.18.2/hadoopfs/name</value> 
   </property> 
   <property> 
     <name>dfs.data.dir</name> 
     <value>/home/linleran/hadoop-0.18.2/hadoopfs/data</value> 
   </property> 
   <property> 
      <name>dfs.replication</name> 
      <value>1</value> 
   </property> 
   <property> 
      <name>hadoop.tmp.dir</name> 
      <value>/home/linleran/hadoop-0.18.2/tmp/</value> 
   </property> 
</configuration> 

配置conf/masters 
将主机ip添加 192.168.11.13 

配置conf/slaves 
将datanodeIP添加 192.168.11.17 

将配置好的hadoop通过scp部署到节点。 
scp –r /home/linleran/hadoop-0.18.2 192.168.11.17: /home/linleran/hadoop-0.18.2 

3.在主服务器启动 
格式化一个新的分布式文件系统 
bin/hadoop namenode –format 
启动服务 
bin/start-all.sh 

4.测试 
测试分布式文件系统 
mkdir test-in 
echo “hello word” > file1.txt 
echo “hello hadoop” > file2.txt 
bin/hadoop dfs –put test-in input 
bin/hadoop jar hadoop-0.18.2-examples.jar wordcount input output 
bin/hadoop dfs –get output result 
cat result/* 

5.配置过程中遇到的问题 
SSH无密码访问不成功。.ssh目录需要755权限，authorized_keys需要644权限，否则一直提示输入密码。 
Linux防火墙开着，hadoop需要开的端口需要添加，或者关掉防火墙。否则会出现节点服务器一直无法连上主服务器。节点服务器的日志不断的retry： 
INFO org.apache.hadoop.ipc.Client: Retrying connect to server… 
数据节点连不上主服务器还有可能是使用了机器名的缘故，还是使用IP地址比较稳妥。 
分布式文件系统报错，java.lang.IllegalArgumentException: Wrong FS:，hadoop-site.xml要配置正确，fs.default.name的value是hdfs://IP:端口/，mapred.job.tracker的value是IP:端口 

参考： 
http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop1/index.html 
http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop2/index.html 
http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop3/index.html 
http://blog.csdn.net/cenwenchu79/archive/2008/08/29/2847529.aspx 
http://rdc.taobao.com/blog/dw/archives/206