amabri shell install 使用shell的方式一键式部署ambri HDP

amabri shell install 使用shell的方式一键式部署ambri HDP

背景&目的

从2016年第一个ambari相关项目开始，需要频繁的部署，不同的ambari版本,使用shell实现一键式部署ambari，后期不断完善，实际上还不够完美。

准备

1. ambari-2.5.0.3-centos7.tar.gz
2. HDP-2.6.5.0-centos7-rpm.tar.gz
3. HDP-UTILS-1.1.0.21-centos7.tar.gz
4. mysql-5.7.27-el7-x86_64.tar.gz
5. mysql-connector-java-5.1.47.jar
6. jdk-8u201-linux-x64.tar.gz
   jdk和mysql需要放在指定路径/opt

版本根据项目需求自己选择对应的版本，可在shell/yumInstall.sh、intsqllMysql.sh、installJDK.sh中替换对应的tar包

服务器准备

1. 操作系统：centos7
2. 网络配置已完成
3. 机房已配备ntp服务，集群使用ntpdate进行时钟同步

shell 脚本

参数设计

config.cnf

#主机名规范，公司-项目-部门或应用 类如jzy-amabri-hdp-后面加编号‘001’
export HOSTSPEC=j-z-y-
#网段信息 
export VLAN=136.192.61.
#用户
export USER=jzy
#密码
export PASSWORD=Jzy12#$
#jdkjar包路径
export JDKPATH=/opt/

主脚本

init.sh

for i in {001..26};
do
#create user
scp /home/createUser.sh pass-eda-hdp-$i:/home/;
ssh pass-eda-hdp-$i sh /home/createUser.sh;
#set hosts
scp /etc/hosts pass-eda-hdp-$i:/etc/;
#stop firewalld
systemctl disable firewalld
systemctl stop firewalld
#disable vm.swappiness
ssh pass-eda-hdp-$i sysctl vm.swappiness=0;
ssh pass-eda-hdp-$i echo vm.swappiness=0 >> /etc/sysctl.conf;
#close selinux and THP
scp /home/close* pass-eda-hdp-$i:/home/;
ssh pass-eda-hdp-$i sh /home/closeSelinux.sh;
ssh pass-eda-hdp-$i sh /home/closeTHP.sh;
ssh pass-eda-hdp-$i reboot;
#install jdk
scp ${path}jdk-8u201-linux-x64.tar.gz pass-eda-hdp-$i:/home/ocdp/;
scp /home/installJDK.sh pass-eda-hdp-$i:/home/;
ssh pass-eda-hdp-$i mv /home/ocdp/jdk-8u201-linux-x64.tar.gz ${path};
ssh pass-eda-hdp-$i sh /home/installJDK.sh;
ssh pass-eda-hdp-$i source /etc/profile;
#modifyLimits
scp /home/modifyLimits.sh pass-eda-hdp-$i:/home/;
ssh pass-eda-hdp-$i sh /home/modifyLimits.sh;
#yum ambari.repo
scp /etc/yum.repos.d/ambari.repo pass-eda-hdp-$i:/etc/yum.repos.d/;
ssh pass-eda-hdp-$i yum clean all && yum makecache;
done

安装jdk

installJDK.sh

#! /bin/bash
echo "******** JDK ********"
path=/opt/
userAndGroup=ocdp:ocdp
tar -xvf ${path}jdk-8u201-linux-x64.tar.gz -C ${path}
chown -R ${userAndGroup} ${path}jdk1.8.0_201
ln -s ${path}jdk1.8.0_201 ${path}jdk
chown -R ${userAndGroup} ${path}jdk
# 我这里是没有配置过环境变量的新机器，所以可以直接在/etc/profile 后追加
# 如果已经有环境变量可根据具体情况使用sed -i等命令配置
echo "export JAVA_HOME=${path}jdk" >> /etc/profile
echo "export PATH=\$PATH:\$JAVA_HOME/bin" >> /etc/profile
echo "export CLASSPATH=.:\$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar" >> /etc/profile
rm -rf /usr/bin/java
rm -rf /usr/bin/javac
#hostname=`"hostname"`
#ssh $hostname "source /etc/profile;java -version;exit"
source /etc/profile
echo "******* JDK installation completed ********"

修改主机参数

closeSelinux.sh

#! /bin/bash
echo "******** CLOSE SELINUX *********"
setenforce 0
file=/etc/selinux/config
sed -i "7s/^/#/" $file
sed '7 iSELINUX=disabled' -i $file
echo "******** END ********"

closeTHP.sh

#! /bin/bash
echo "******** CLOSE THP *********"
echo "if test -f /sys/kernel/mm/transparent_hugepage/enabled; then" >> /etc/rc.d/rc.local
echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.d/rc.local
echo "fi" >> /etc/rc.d/rc.local
echo "if test -f /sys/kernel/mm/transparent_hugepage/defrag; then" >> /etc/rc.d/rc.local
echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.d/rc.local
echo "fi" >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local
echo "******** END ********"

modifyLimits.sh

#! /bin/bash
echo "******** Modify system limits ********"
echo "* soft nofile 65535" >> /etc/security/limits.conf
echo "* hard nofile 65535" >> /etc/security/limits.conf
echo "ocdp   - nofile 65535" >> /etc/security/limits.conf
echo "ocdp   - nproc  65535" >> /etc/security/limits.conf
echo "******** END ********"

配置hosts文件

setHosts.sh

echo "0.0.0.0   0.0.0.0" >> /etc/hosts;
for i in {001..26};
do
echo "136.192.61.$[10#$i+190-1] pass-eda-hdp-$i" >> /etc/hosts;
done

ssh免密

sshScript.sh

#! /bin/bash
ssh-keygen -t rsa
# 192.168.111 改为集群所在网段（和hosts中配置对应）
ALL_CLIENTS=`cat /etc/hosts| grep "136.192.61" | awk '{print $2}'`
for client in $ALL_CLIENTS
do
        echo "=============copy-ssh-id $client============="
        ssh-copy-id $client
done                    

执行

sh init.sh
#install ambari
amabri-setup

#init ambari mysql
mysql -e"use mysql;GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION" -p123456 -h127.0.0.1
mysql -e"CREATE USER ambari IDENTIFIED BY '123456'" -p123456 -h127.0.0.1
mysql -e"CREATE DATABASE ambari" -p123456 -h127.0.0.1
# 执行ambari的sql脚本 路径默认是一样的
mysql  -uroot -p123456 -h127.0.0.1 ambari < /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
mysql -e"grant ALL on ambari.* to ambari;flush  privileges" -uroot -p123456 -h127.0.0.1

#start ambari
ambari-server start

注意

如果自行使用amabri-setup以下内容可以不管,在初始化mysql完成后
conf/ambari.properties 中配置需要使用sed 命令修改mysql，jdk相关内容。

cat conf/ambari.properties > /etc/ambari-server/conf/ambari.properties
#set amabri mysqlpassword
echo "123456" >  /etc/ambari-server/conf/password.dat

版本

当前版本为v0.1，未经过严格测试。

源码

源码已上传github

版权

目前是GUN3，后期趋于完美可能会改为apache2

后续

期望&缺陷

1. 加入ntp的安装
2. for循环1到26，用该提出来作为配置
3. tar 应该使用配置文件管理起来
4. mysql部分和amabri setup应该融入脚本
   实现从裸机直接到amabri页面安装的一键式脚本