[hadoop]hadoop2.4.0集群安装示例

软件环境:
linux:ubuntu1404
jdk:jdk-6u45-linux-x64.bin
hadoop:hadoop-2.4.0
主机名称和IP
192.168.233.129 master.hadoop master
192.168.233.130 node1.hadoop node1
192.168.233.131 node2.hadoop node2

一、设置主机名称，配置hostname文件
1.修改/etc/hostname文件，master增加所有node的IP，所有node增加master的IP
二、建立用户群（hadoop）和用户(hadoop)，配置sudo权限

attwice@master:~$ sudo su
root@master:~# groupadd hadoop 
root@master:~# useradd -s /bin/bash -d /home/hadoop -m hadoop -g hadoop
root@master:~# passwd hadoop
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
root@master:~# ls /home
attwice hadoop

配置sudo权限，修改文件/etc/sudoers，增加一行如下：

hadoop ALL=(ALL) ALL

三、配置SSH免密码登入
配置master无密码登录所有的node，配置所有node无密码登录master
安装ssh：sudo apt-get install ssh

hadoop@master:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
ec:dd:60:07:0b:f0:ea:a1:8f:d6:84:5e:f6:5c:0a:f0 hadoop@master
The key's randomart image is:
+--[ RSA 2048]----+
| . |
| o |
| o . |
| . o . o |
| +o S + . |
| .oEo o.+ |
| ..=.+.o. . |
| oo. + |
| .. . |
+-----------------+

进入.ssh目录，执行如下命令

[master上]
cat id_rsa.pub >> authorized_keys
scp id_rsa.pub hadoop@node1:~/.ssh/master.pub
scp id_rsa.pub hadoop@node2:~/.ssh/master.pub
scp hadoop@node1:~/.ssh/id_rsa.pub node1.pub
scp hadoop@node2:~/.ssh/id_rsa.pub node2.pub    
cat node1.pub >> authorized_keys
cat node2.pub >> authorized_keys
[node上]
cat id_rsa.pub >> authorized_keys
cat master.pub >> authorized_keys

用ssh命令测试各自的互通性：ssh localhost或ssh xxx
四、安装JDK，设置环境变量
1.将JDK安装到/usr/java下面
2.设置环境变量，修改/etc/profile文件，增加如下内容：

JAVA_HOME=/usr/java/jdk1.6.0_45
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME PATH

五、安装hadoop
1.解压hadoop-2.4.0.tar.gz文件到/home/hadoop/apps/目录下，进入目录/home/hadoop/apps/hadoop-2.4.0/etc/hadoop，修改配置文件
2.配置hadoop-env.sh（JAVA_HOME）

export JAVA_HOME=/usr/java/jdk1.6.0_45

3.配置core-site.xml，configuration增加如下内容：

  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/apps/temp/hadoop-${user.name}</value>
  </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://192.168.233.129:9000</value>
  </property>

master主机需创建temp文件夹
4.配置yarn-site.xml，configuration增加如下内容：

  <property>
    <name>yarn.resourcemanager.address</name>
    <value>192.168.233.129:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>192.168.233.129:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>192.168.233.129:8031</value>
  </property>

5.可选配置mapred-site.xml
6.配置slaves（仅仅在master主机上，执行完第7步再修改）

node1
node2

7.将配置好的hadoop复制到各个节点

scp -r hadoop-2.4.0 hadoop@node1:~/apps/
scp -r hadoop-2.4.0 hadoop@node2:~/apps/

六、测试是否配置成功
1.格式化及启动

bin/hadoop namenode -format
sbin/start-dfs.sh
sbin/start-yarn.sh

2.查看状态（后台和web方式）
bin/hadoop dfsadmin -report
http://192.168.233.129:50070/  查看信息
http://192.168.233.129:8088/  查看resource manager
七、辅助功能
1.关闭hadoop

sbin/stop-dfs.sh
sbin/stop-yarn.sh