一、基础环境
主机 | 虚拟机1 | 虚拟机2 | 虚拟机3 | |
名称 | hadoop.master | hadoop.slave01 | hadoop.slave02 | |
IP | 192.168.206.1 | 192.168.206.120 | 192.168.206.121 | 192.168.206.122 |
操作系统 | win7 64位 | centOS 6.4 64位 | centOS 6.4 64位 | centOS 6.4 64位 |
CPU核 | i5 4核 | 1 | 1 | 1 |
RAM | 8 GB | 2 GB | 2 GB | 2 GB |
硬盘 | 1T | 20 GB | 20 GB | 20 GB |
二、安装配置
1. 在三个虚拟机上创建hadoop组和hadoop用户, 创建完后用id hadoop看一下,我的机器上是uid=500(hadoop) gid=5000(hadoop) 组=500(hadoop)
su -
groupadd hadoop
useradd -g hadoop hadoop
2. 配置hadoop.master、hadoop.slave01、hadoop.slave02三个虚拟机免密码登录
1) 在/etc/hosts文件增加以下3行(3台机器上都要增加)
192.168.206.120 hadoop.master
192.168.206.121 hadoop.slave01
192.168.206.122 hadoop.slave02
2)生成authorized_keys文件
在hadoop.master的 /home/hadoop目录下执行 ssh-keygen -t rsa 遇提问,直接回车,在~/.ssh目录下得到id_rsa,id_rsa.pub两个文件
在hadoop.master的/home/hadoop目录下执行 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 在~/.ssh目录下得到authorized_keys文件
在hadoop.slave01、hadoop.slave02 的/home/hadoop目录下执行 ssh-keygen -t rsa 遇提问,直接回车,在~/.ssh目录下得到id_rsa,id_rsa.pub两个文件
在hadoop.master的/home/hadoop目录下执行 scp ~/.ssh/authorized_keys hadoop@hadoop.slave01:~/.ssh ,拷贝authorized_keys文件到hadoop.slave01/home/hadoop/.ssh/authorized_keys
在hadoop.slave01的/home/hadoop目录下执行 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
在hadoop.slave01的/home/hadoop目录下执行 scp ~/.ssh/authorized_keys hadoop@hadoop.slave02:~/.ssh ,拷贝authorized_keys文件到hadoop.slave02/home/hadoop/.ssh/authorized_keys
在hadoop.slave02的/home/hadoop目录下执行 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
在hadoop.slave02的/home/hadoop目录下执行 scp ~/.ssh/authorized_keys hadoop@hadoop.slave01:~/.ssh ,拷贝authorized_keys文件到hadoop.slave01/home/hadoop/.ssh/authorized_keys
在hadoop.slave02的/home/hadoop目录下执行 scp ~/.ssh/authorized_keys hadoop@hadoop.master:~/.ssh ,拷贝authorized_keys文件到hadoop.master/home/hadoop/.ssh/authorized_keys
在hadoop.master、hadoop.slave01、hadoop.slave02上执行chmod 700 ~/.ssh和chmod 600 ~/.ssh/authorized_keys/home/hadoop
3. 在hadoop.maste的主目录(/home/hadoop)下建目录cloud,将软件hadoop-2.6.0.tar.gz用tar命令解压缩到/home/hadoop/cloud目录,用mv命令对解压缩后的目录改名为hadoop,更改后的目录为/home/hadoop/cloud/hadoop
4. 配置core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop.master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/cloud/hdtmp</value>
</property>
</configuration>
5.配置hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/cloud/hdname</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/cloud/hddata</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/cloud/hdtmp/</value>
</property>
</configuration>
6.配置mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
7.配置yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop.master</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop.master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop.master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop.master:8032</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>${hadoop.tmp.dir}/nodemanager/local</value>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:8034</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>${hadoop.tmp.dir}/nodemanager/remote</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>${hadoop.tmp.dir}/nodemanager/logs</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
8.配置masters
hadoop.master
9.配置slaves
hadoop.slave01
hadoop.slave02
10.配置环境变量
export JAVA_HOME="/usr/lib/jvm/jre-1.7.0-openjdk.x86_64"
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$PATH:$HOME/bin
export HADOOP_HOME=/home/hadoop/cloud/hadoop
export PATH=.:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
三、启动、停止
1.格式化namenode
hadoop namenode -format
start-dfs.sh
start-yarn.sh
3.停止
stop-yarn.sh
stop-dfs.sh
4.用命令查看
hadoop.master
$ jps
3885 SecondaryNameNode
5497 Jps
4070 ResourceManager
3724 NameNode
hadoop.slave01
$jps
3716 NodeManager
4464 Jps
3594 DataNode
hadoop.slave02
$jps
3716 NodeManager
4700 Jps
3600 DataNode
<span style="font-family: Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255);">5. 用web ui 查看</span>
http://hadoop.master:8088
http://hadoop.master:50070