1.系统环境说明
192.168.100.71 Master.Hadoop CentOS7.1 x64版本 192.168.100.61 Salve1.Hadoop CentOS6.6 x64版本 192.168.100.72 Salve2.Hadoop CentOS7.1 x64版本
2.安装前的准备工作
2.1 关闭防火墙
7.1的命令 # systemctl status firewalld.service --查看防火墙状态 # systemctl stop firewalld.service --关闭防火墙 # systemctl disable firewalld.service --永久关闭防火墙 6.6的命令 # service iptables stop
2.2 检查ssh安装情况,如果没有则安装ssh
# systemctl status sshd.service --查看ssh状态 # yum install openssh-server openssh-clients
2.3 安装vim
# yum -y install vim
2.4 设置静态ip地址
# vim /etc/sysconfig/network-scripts/ifcfg-eno16777736BOOTPROTO="static"
ONBOOT="yes"
IPADDR0="192.168.100.71"
PREFIX0="255.255.255.0"
GATEWAY0="192.168.100.1"
DNS1="192.168.100.1"
2.5 修改host名称
# vim /etc/sysconfig/network
HOSTNAME=master
# vim /etc/hosts
192.168.100.71 Master.Hadoop 192.168.100.61 Slave1.Hadoop 192.168.100.72 slave2.Hadoop
# hostnamectl set-hostname Master.Hadoop (CentOS7 下原有的修改host方法无效了)
2.6 创建hadoop用户
# useradd hadoop --创建用户名为hadoop的用户# passwd hadoop --为用户hadoop设置密码
2.7 配置ssh无密钥登录
———–下面是在master上面的操作
# su hadoop --切换到hadoop用户$ cd ~ --打开用户文件夹 $ ssh-keygen -t rsa -P '' --生成密码对,/home/hadoop/.ssh/id_rsa和/home/hadoop/.ssh/id_rsa.pub $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys --把id_rsa.pub追加到授权的key里面去 $ chmod 600 ~/.ssh/authorized_keys --修改权限 $ su --切换到root用户# vim /etc/ssh/sshd_config --修改ssh配置文件 RSAAuthentication yes #启用RSA认证 PubkeyAuthentication yes #启用公钥私钥配对认证方式 AuthorizedKeysFile .ssh/authorized_keys #公钥文件路径 # su hadoop --切换到hadoop用户 $ scp ~/.ssh/id_rsa.pub hadoop@192.168.1.8:~/ --把公钥复制所有的Slave机器上
———-下面是在slave1上面的操作
# su hadoop --切换到hadoop用户 $ mkdir ~/.ssh $ chmod 700 ~/.ssh $ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys --追加到授权文件"authorized_keys" $ chmod 600 ~/.ssh/authorized_keys --修改权限 $ su --切换回root用户 # vim /etc/ssh/sshd_config --修改ssh配置文件 RSAAuthentication yes #启用RSA认证 PubkeyAuthentication yes #启用公钥私钥配对认证方式 AuthorizedKeysFile .ssh/authorized_keys #公钥文件路径 # su hadoop $ ssh Master.Hadoop -- 验证无密码登陆
3.安装必须的软件
3.1 安装JDK & 解压 Hadoop-2.6.0
# rpm -qa | grep jdk --查看原来安装的JDK,卸载之 java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.7.el7_1.x86_64 java-1.7.0-openjdk-1.7.0.75-2.5.4.7.el7_1.x86_64 # yum -y remove java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.7.el7_1.x86_64 # yum -y remove java-1.7.0-openjdk-1.7.0.75-2.5.4.7.el7_1.x86_64 # rpm -ivh jdk-7u75-linux-x64.rpm # cd /opt # tar zxvf hadoop-2.6.0.tar.gz # chown –R hadoop:hadoop hadoop-2.6.0 # vim /etc/profile export JAVA_HOME=/usr/java/jdk1.7.0_75 export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar: export HADOOP_HOME=/opt/hadoop-2.6.0 export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin # source profile --修改生效3.2 vim /etc/hosts文件 在最后面添加如下内容:
[root@master ]# cat /etc/hosts 192.168.100.71 Master.Hadoop 192.168.100.61 Salve1.Hadoop 192.168.100.72 Salve2.Hadoop5) [root@systdt hadoop]# vi core-site.xml
[root@systdt hadoop]# cat core-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/opt/hadoop/tmp</value> <description>Abase for other temporary directories.</description> </property> <property> <name>fs.defaultFS</name> <value>hdfs://Master.Hadoop:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>4096</value> </property> </configuration>6) vi hadoop-env.sh 和 yarn-env.sh 在开头添加如下环境变量(一定要添加切勿少了)
export JAVA_HOME=/usr/java/jdk1.7.0_71
7)vi hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.namenode.name.dir</name> <value>file:///usr/hadoop/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///usr/hadoop/dfs/data</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.nameservices</name> <value>hadoop-cluster1</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>Master.Hadoop:50090</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> </configuration>8) vi mapred-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> <final>true</final> </property> <property> <name>mapreduce.jobtracker.http.address</name> <value>Master.Hadoop:50030</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>Master.Hadoop:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>Master.Hadoop:19888</value> </property> <property> <name>mapred.job.tracker</name> <value>http://Master.Hadoop:9001</value> </property> </configuration>
9:vi yarn-site.xml
<?xml version="1.0"?> <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.resourcemanager.hostname</name> <value>Master.Hadoop</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>Master.Hadoop:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>Master.Hadoop:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>Master.Hadoop:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>Master.Hadoop:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>Master.Hadoop:8088</value> </property> </configuration>
第四步:配置Hadoop的集群;
1)先修改10.2.10.27(即:Master.Hadoop)
[root@systdt hadoop]# pwd /opt/www/hadoop-2.6.0/etc/hadoop [root@systdt hadoop]# vi slaves Salve1.Hadoop Salve2.Hadoop
2)登录:10.2.10.27 将hadoop-2.6.0整个的copy到另外两台机器上面
[root@systdt www]# scp -r /opt/hadoop-2.6.0 192.168.100.61:/opt/ [root@systdt www]# scp -r /opt/hadoop-2.6.0 192.168.100.72:/opt/3)配置两位两台机器上面的环境变量:重复第三步Master.Hadoop机器配置里面的(2,3,10步动作)
4)将三台机器的防火墙关闭掉:
[root@systdt www]# service iptables stop到此:整个三台Hadoop的集群的机器算是配置完成了;
第五步:启动整个集群及其验证:
1)登录到:10.2.10.27 (即:Master.Hadoop的机器上面)
[root@systdt sbin]# pwd /opt/hadoop-2.6.0/sbin [root@systdt sbin]# start-all.sh2)验证:
#Master上面的如下: [root@systdt sbin]# jps 22984 NameNode 23861 Jps 23239 ResourceManager 19766 NodeManager #Salve1和Salve2上面的如下: [root@systdt2 .ssh]# jps 609 Jps 30062 DataNode 2024 Bootstrap 30174 NodeManager第六步:就可以通过浏览器访问了哦
http://10.2.10.27:8088/cluster/nodes
10)执行"hadoop dfsadmin -report" 命令