使用shell脚本安装hadoop高可用集群

集群划分

192.168.56.120 hadoop01	192.168.56.121 hadoop02	192.168.56.122 hadoop03
QuorumPeerMain	QuorumPeerMain	QuorumPeerMain
JournalNode	JournalNode	JournalNode
NameNode	NameNode	–
ResourceManager	ResourceManager	–
NodeManager	NodeManager	NodeManager
DFSZKFailoverController	DFSZKFailoverController	–
DataNode	DataNode	DataNode

脚本和相关的文件如下,提取码: sweh
相关文件

一.创建一台虚拟机

• 1.准备一台纯净版的centos7系统,修改静态ip为192.168.56.120,主机名为hadoop01,关闭防火墙,重启网络,连接moba
• 2.创建安装包目录:mkdir /opt/software
• 3.创建安装目录: mkdir /opt/install
• 4.将脚本和相关配置文件安装包拖到安装包目录下
• 5.修改脚本的权限:chmod 777 install*
• 6.执行脚本安装jdk:/opt/software/installJdk.sh
• 7.执行脚本安装hadoop和zookeeper:/opt/software/installHadoop.sh
• 8.关闭该虚拟机

二.复制两台虚拟机

• 1.一台修改静态ip为192.168.56.121 ,主机名为hadoop02
• 2.另一台修改静态ip为192.168.56.122 ,主机名为hadoop03
• 3.将hadoop02机器zookeeper的myid修改为2,hadoop03修改为3
• 4.三台机器配置免登录
• 1)ssh-keygen -t rsa -P ""生成私钥
• 2)cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys添加到信任
• 3)远程免登录:ssh-copy-id -i ~/.ssh/id_rsa.pub -p22 root@192.168.56.120 ,注意一台向其他两台发送
• 4)三台机器分别执行:

ssh  -o StrictHostKeyChecking=no `hostname` 

三.启动集群

• 1.3台 zkServer.sh start,三台hadoop-daemon.sh start journalnode
• 2.hadoop01 hadoop namenode -format,hadoop-daemon.sh start namenode
• 3.hadoop02 hdfs namenode -bootstrapStandby,只在第一次启动需要同步以后不需要
• 3.hadoop01 hdfs zkfc -formatZK
• 4.三台停掉所有Journalnode进程: hadoop-daemon.sh stop journalnode
• 5.hadoop01hadoop-daemon.sh stop namenode
• 6.hadoop01 start-dfs.sh
• 7.hadoop01 start-yarn.sh
• 8.hadoop02 yarn-daemon.sh start resourcemanager
• 9启动后效果如下:
  
  
  

四.脚本内容如下

1.jdk

#!/bin/bash
#安装jdk1.8,需要先创建software目录和install目录,然后将jdk包拖进software目录中
tar -zxvf  /opt/software/jdk-8u221-linux-x64.tar.gz -C /opt/install/
`echo "export JAVA_HOME=/opt/install/jdk1.8.0_221" >> /etc/profile`
`echo 'export CLASSPATH=.:$JAVA_HOME/rt.jar:$JAVA_HOME/tools.jar:$JAVA_HOME/dt.jar' >> /etc/profile`
`echo 'export JRE_HOME=$JAVA_HOME/jre' >> /etc/profile`
`echo 'export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin' >> /etc/profile`
source /etc/profile
java -version

2.hadoop和zookeeper

#!/bin/bash
#将zookeeper和hadoop安装包挪到software下,将配置文件也挪到software目录下
tar -zxvf  /opt/software/zookeeper-3.4.6.tar.gz -C /opt/install/
tar -zxvf  /opt/software/hadoop-2.6.0-cdh5.14.2.tar.gz -C /opt/install/
mv /opt/install/hadoop-2.6.0-cdh5.14.2 /opt/install/hadoop
`echo "export HADOOP_HOME=/opt/install/hadoop" >> /etc/profile`
`echo 'export HADOOP_MAPRED_HOME=$HADOOP_HOME' >> /etc/profile`
`echo 'export HADOOP_COMMON_HOME=$HADOOP_HOME' >> /etc/profile`
`echo 'export HADOOP_HDFS_HOME=$HADOOP_HOME' >> /etc/profile`
`echo 'export YARN_HOME=$HADOOP_HOME' >> /etc/profile`
`echo 'export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native' >> /etc/profile`
`echo 'export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"' >> /etc/profile`
`echo 'export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin' >> /etc/profile`
`echo 'export ZK_HOME=/opt/install/zookeeper-3.4.6' >> /etc/profile`
`echo 'export PATH=$PATH:$ZK_HOME/bin' >> /etc/profile`
source /etc/profile
sed -i  '24,26s/\${JAVA_HOME}/\/opt\/install\/jdk1.8.0_221/gi' /opt/install/hadoop/etc/hadoop/hadoop-env.sh
cat $PWD/core-site.xml > /opt/install/hadoop/etc/hadoop/core-site.xml
cat $PWD/hdfs-site.xml > /opt/install/hadoop/etc/hadoop/hdfs-site.xml
cat $PWD/mapred-site.xml > /opt/install/hadoop/etc/hadoop/mapred-site.xml
cat $PWD/yarn-site.xml > /opt/install/hadoop/etc/hadoop/yarn-site.xml
cat $PWD/zoo.cfg > /opt/install/zookeeper-3.4.6/conf/zoo.cfg
cat $PWD/slaves > /opt/install/hadoop/etc/hadoop/slaves
mkdir /opt/install/zookeeper-3.4.6/zkData
mkdir   /opt/install/zookeeper-3.4.6/zkLog
touch /opt/install/zookeeper-3.4.6/zkData/myid
echo '1' >  /opt/install/zookeeper-3.4.6/zkData/myid
echo '192.168.56.120 hadoop01' >> /etc/hosts
echo '192.168.56.121 hadoop02' >> /etc/hosts
echo '192.168.56.122 hadoop03' >> /etc/hosts
source /etc/profile
tar -xvf $PWD/hadoop-native-64-2.6.0.tar -C $HADOOP_HOME/lib/native
tar -xvf $PWD/hadoop-native-64-2.6.0.tar -C $HADOOP_HOME/lib
echo 'hadoop 版本信息:' 
hadoop version

3.一键启动集群

#!/usr/bin/env bash
ssh hadoop01 "echo '1' > /opt/install/zookeeper-3.4.6/zkData/myid"
ssh hadoop02 "echo '2' > /opt/install/zookeeper-3.4.6/zkData/myid"
ssh hadoop03 "echo '3' > /opt/install/zookeeper-3.4.6/zkData/myid"
for s in hadoop01 hadoop02 hadoop03
do
	ssh $s "source /etc/profile; cd /opt/install/hadoop/; rm -rf hdfsTmp/"
done
sleep 2
for s in hadoop01 hadoop02 hadoop03
do
	ssh $s "source /etc/profile; zkServer.sh start; hadoop-daemon.sh start journalnode"
done
sleep 2
ssh hadoop01 "source /etc/profile; hadoop namenode -format; hadoop-daemon.sh start namenode"
sleep 2
ssh hadoop02 "source /etc/profile; hdfs namenode -bootstrapStandby"
sleep 2
ssh hadoop01 "source /etc/profile; hdfs zkfc -formatZK; hadoop-daemon.sh stop journalnode"
sleep 2
ssh hadoop02 "source /etc/profile; hadoop-daemon.sh stop journalnode"
sleep 2
ssh hadoop03 "source /etc/profile; hadoop-daemon.sh stop journalnode"
sleep 2
ssh hadoop01 "source /etc/profile; hadoop-daemon.sh stop namenode; start-dfs.sh; start-yarn.sh"
sleep 2
ssh hadoop02 "source /etc/profile; yarn-daemon.sh start resourcemanager"
sleep 2
for s in hadoop01 hadoop02 hadoop03
do
	echo "=================$s-jps================="
	ssh $s "source /etc/profile; jps"
done
sleep 2
for s in hadoop01 
do
	echo "=================$s -resourcemanager status================="
	ssh $s "source /etc/profile; yarn rmadmin -getServiceState rm1"
	echo "=================$s -namenode status================="
	ssh $s "source /etc/profile; hdfs haadmin -getServiceState nn1"
done
sleep 2
for s in hadoop02
do
	echo "=================$s -resourcemanager status================="
	ssh $s "source /etc/profile; yarn rmadmin -getServiceState rm2"
	echo "=================$s -namenode status================="
	ssh $s "source /etc/profile; hdfs haadmin -getServiceState nn2"
done
sleep 2
for s in hadoop01 hadoop02 hadoop03
do
	echo "=================$s- zkServer status================="
	ssh $s "source /etc/profile; zkServer.sh status"
done

使用脚本运行效果如下:

=================hadoop01-jps=================
22896 NameNode
22002 QuorumPeerMain
23430 DFSZKFailoverController
23016 DataNode
23674 NodeManager
24010 Jps
23549 ResourceManager
23231 JournalNode
=================hadoop02-jps=================
15138 NameNode
15218 DataNode
14900 QuorumPeerMain
15317 JournalNode
15445 DFSZKFailoverController
15787 Jps
15534 NodeManager
15710 ResourceManager
=================hadoop03-jps=================
12996 NodeManager
12794 DataNode
13147 Jps
12621 QuorumPeerMain
12893 JournalNode
=================hadoop01 -resourcemanager status=================
20/07/09 08:48:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-jav                                    a classes where applicable
active
=================hadoop01 -namenode status=================
active
=================hadoop02 -resourcemanager status=================
20/07/09 08:49:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-jav                                    a classes where applicable
standby
=================hadoop02 -namenode status=================
standby
=================hadoop01- zkServer status=================
JMX enabled by default
Using config: /opt/install/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: follower
=================hadoop02- zkServer status=================
JMX enabled by default
Using config: /opt/install/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: leader
=================hadoop03- zkServer status=================
JMX enabled by default
Using config: /opt/install/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: follower

注:需要下载psmisc依赖包,否则无法完成自动切换节点

yum install -y psmisc