本文章仅作为个人笔记
- CDH官网地址:https://www.cloudera.com/
- CDH镜像地址:
- cm5.11.1地址:
- http://archive.cloudera.com/cm5/repo-as-tarball/5.11.1/cm5.11.1-centos7.tar.gz
- http://archive.cloudera.com/cm5/repo-as-tarball/5.11.1/cm5.11.1-centos7.tar.gz.sha1
- cdh5.11.1地址:
- http://archive.cloudera.com/cdh5/parcels/5.11.1/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel
- http://archive.cloudera.com/cdh5/parcels/5.11.1/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha1
- http://archive.cloudera.com/cdh5/parcels/5.11.1/manifest.json
- jdk1.8.121官网下载地址:
- http://download.oracle.com/otn/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-i586.rpm
- mysql5.6.36官网下载地址:
- https://dev.mysql.com/get/Downloads/MySQL-5.6/mysql-5.6.36-linux-glibc2.5-x86_64.tar.gz
- mysql连接驱动下载地址:
- https://dev.mysql.com/downloads/connector/j/
- 笔者使用VMware进行模拟安装,为了满足基础配置,这里分配3台虚拟机(hadoopb01(4g/4核/40g/192.168.75.136/master),hadoopb02(2g/2核/40g/192.168.75.144/slave),hadoopb03(2g/2核/40g/192.168.75.145/slave)),另外wget下载文件非常缓慢并且不方便反复测试及安装,笔者这里下载好所有资源文件放在本地(.sha1文件结尾的文件统一重命名为.sha)。(如果虚拟机开机后无网络(使用net桥接方式),可与命令行运行dhclient分配ip,若后续断开了可运行dhclient -r后再运行dhclient)
- 本地虚拟机,本地文件
- 在3台服务器共同操作:
- 安装基本工具: yum install vim wget net-tools unzip -y
- 关闭SELINUX:
- 运行 setenforce 0 临时关闭
- 永久关闭SELINUX:
echo “SELINUX=disabled” > /etc/selinux/config
echo “SELINUXTYPE=targeted” >> /etc/selinux/config - 关闭iptables防火墙:
- 运行 systemctl stop firewalld.service 停止防火墙
- 运行 systemctl status firewalld.service 查看防火墙状态
- 运行 systemctl disable firewalld.service 关闭防火墙开机启动
- 更改主机名:
- 运行 hostname hadoopb01 更改主机名,根据不同主机选择不同主机名(hadoopb01/hadoopb02/hadoopb03)
- 运行 echo “HOSTNAME=hadoopb01” > /etc/sysconfig/network ,根据不同主机选择不同主机名(hadoopb01/hadoopb02/hadoopb03)
- 运行 echo “hadoopb01” > /etc/hostname ,根据不同主机选择不同主机名(hadoopb01/hadoopb02/hadoopb03)
- 安装jdk(参考http://www.jianshu.com/p/c5fc4dd47754)
- 密钥生成:
- 运行 ssh-keygen 一路回车回车生成
- ip与hostname绑定:
echo “192.168.75.136 hadoopb01” >> /etc/hosts
echo “192.168.75.144 hadoopb02” >> /etc/hosts
echo “192.168.75.145 hadoopb03” >> /etc/hosts - 运行python –version 检查python版本,如果不是python2.6.6建议改为python2.6.6
- 时间同步:
- 运行 yum install ntp -y 安装ntp服务
- 运行 systemctl enable ntpd 使ntp服务开机启动
- 在从服务器操作(hadoopb02(2g/2核/40g/192.168.75.144/slave),hadoopb03(2g/2核/40g/192.168.75.145/slave):
- 配置时钟同步:
echo “driftfile /var/lib/ntp/ntp.drift” > /etc/ntp.conf
echo “statistics loopstats peerstats clockstats” >> /etc/ntp.conf
echo “filegen loopstats file loopstats type day enable” >> /etc/ntp.conf
echo “filegen peerstats file peerstats type day enable” >> /etc/ntp.conf
echo “filegen clockstats file clockstats type day enable” >> /etc/ntp.conf
echo “server 192.168.75.136 prefer” >> /etc/ntp.conf
echo “fudge 192.168.75.136 stratum 5” >> /etc/ntp.conf
echo “restrict -4 default kod notrap nomodify nopeer noquery” >> /etc/ntp.conf
echo “restrict -6 default kod notrap nomodify nopeer noquery” >> /etc/ntp.conf
echo “restrict 127.0.0.1” >> /etc/ntp.conf
echo “restrict ::1” >> /etc/ntp.conf - 运行 service ntpd restart 重启ntpd并使配置生效
- 运行 ntpdate –u 192.168.75.136 同步时间
- 在主服务器操作(hadoopb01(4g/4核/40g/192.168.75.136/master):
- 运行
ssh-copy-id hadoopb01
ssh-copy-id hadoopb02
ssh-copy-id hadoopb03 完成各服务器免密登录 - 配置时钟同步:
echo “driftfile /var/lib/ntp/ntp.drift” > /etc/ntp.conf
echo “restrict 192.168.75.0 mask 255.255.255.0 nomodify notrap” >> /etc/ntp.conf
echo “server 210.72.145.44” >> /etc/ntp.conf
echo “server 202.112.10.36 perfer” >> /etc/ntp.conf
echo “server 59.124.196.83” >> /etc/ntp.conf
echo “restrict 210.72.145.44 nomodify notrap noquery” >> /etc/ntp.conf
echo “restrict 202.112.10.36 nomodify notrap noquery” >> /etc/ntp.conf
echo “restrict 59.124.196.83 nomodify notrap noquery” >> /etc/ntp.conf
echo “server 127.127.1.0” >> /etc/ntp.conf
echo “fudge 127.127.1.0 stratum 10” >> /etc/ntp.conf - 运行 service ntpd restart 重启ntpd并使配置生效
- 运行 ntpdate –u 202.112.10.36 同步时间
- 使用filezilla将上文所述文件传输至主服务器(放置于root用户~目录)
- 安装配置mysql5.6(可参考http://www.jianshu.com/p/c5fc4dd47754)
- 创建相关用户及数据库:
create database cmf default character set utf8
grant all on cmf.* to ‘cmf’@’localhost’ identified by ‘cmf_password’
grant all on cmf.* to ‘cmf’@’%’ identified by ‘cmf_password’
grant all on cmf.* to ‘cmf’@’ mhosts[$cindex]′identifiedby‘cmfpassword′dropdatabasecmfcreatedatabasecmfcreatedatabasehivedefaultcharactersetutf8grantallonhive.∗to‘hive′@′localhost′identifiedby‘hivepassword′grantallonhive.∗to‘hive′@′grantallonhive.∗to‘hive′@′ m h o s t s [ $ c i n d e x ] ′ i d e n t i f i e d b y ‘ c m f p a s s w o r d ′ d r o p d a t a b a s e c m f c r e a t e d a t a b a s e c m f c r e a t e d a t a b a s e h i v e d e f a u l t c h a r a c t e r s e t u t f 8 g r a n t a l l o n h i v e . ∗ t o ‘ h i v e ′ @ ′ l o c a l h o s t ′ i d e n t i f i e d b y ‘ h i v e p a s s w o r d ′ g r a n t a l l o n h i v e . ∗ t o ‘ h i v e ′ @ ′ g r a n t a l l o n h i v e . ∗ t o ‘ h i v e ′ @ ′ {m_hosts[{c_index}]}’ identified by ‘hive_password’
drop database hive
create database hive
create database hue default character set utf8
grant all on hue.* to ‘hue’@’localhost’ identified by ‘hue_password’
grant all on hue.* to ‘hue’@’%’ identified by ‘hue_password’
grant all on hue.* to ‘hue’@’ {c_index}]}’ identified by ‘hive_password’ drop database hive create database hive create database hue default character set utf8 grant all on hue.* to ‘hue’@’localhost’ identified by ‘hue_password’ grant all on hue.* to ‘hue’@’%’ identified by ‘hue_password’ grant all on hue.* to ‘hue’@’ {m_hosts[{c_index}]}’ identified by ‘hue_password’
drop database hue
create database hue
create database am default character set utf8
grant all on am.* to ‘am’@’localhost’ identified by ‘am_password’
grant all on am.* to ‘am’@’%’ identified by ‘am_password’
grant all on am.* to ‘am’@’ {c_index}]}’ identified by ‘hue_password’ drop database hue create database hue create database am default character set utf8 grant all on am.* to ‘am’@’localhost’ identified by ‘am_password’ grant all on am.* to ‘am’@’%’ identified by ‘am_password’ grant all on am.* to ‘am’@’ {m_hosts[{c_index}]}’ identified by ‘am_password’
drop database am
create database am
create database os default character set utf8
grant all on os.* to ‘os’@’localhost’ identified by ‘os_password’
grant all on os.* to ‘os’@’%’ identified by ‘os_password’
grant all on os.* to ‘os’@’ {c_index}]}’ identified by ‘am_password’ drop database am create database am create database os default character set utf8 grant all on os.* to ‘os’@’localhost’ identified by ‘os_password’ grant all on os.* to ‘os’@’%’ identified by ‘os_password’ grant all on os.* to ‘os’@’ {m_hosts[${c_index}]}’ identified by ‘os_password’
drop database os
create database os
flush privileges - 安装并启动httpd服务:
- 运行 yum install -y httpd 安装httpd服务
- 运行 systemctl enable httpd 设置httpd服务开机启动
- 运行 service httpd start 启动httpd服务
- 创建parcels服务(依次运行如下命令):
cd /var/www/html
mkdir parcels
cd parcels
mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel ./
mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha ./
mv ~/manifest.json ./ - 创建cm5服务(依次运行如下命令):
mkdir /opt/rpminstall
cd /opt/rpminstall
mv ~/cm5.11.1-centos7.tar.gz ./
mv ~/cm5.11.1-centos7.tar.gz.sha ./
tar -zxf cm5.11.1-centos7.tar.gz -C /var/www/html/
cd /var/www/html/
mkdir -p cm5/redhat/7/x86_64/
mv cm cm5/redhat/7/x86_64/
chown apache:apache -R /var/www/html
cd /var/www/html/cm5/redhat/7/x86_64/cm/5/RPMS/x86_64
yum install -y cloudera-manager-daemons-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm
yum install -y cloudera-manager-server-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm
mkdir /usr/share/java
cd /usr/share/java
mv ~/mysql-connector-java-5.1.42.zip ./
unzip mysql-connector-java-5.1.42.zip
cd mysql-connector-java-5.1.42
cp mysql-connector-java-5.1.42-bin.jar ../mysql-connector-java.jar
echo “com.cloudera.cmf.db.type=mysql” > /etc/cloudera-scm-server/db.properties
echo “com.cloudera.cmf.db.host=localhost” >> /etc/cloudera-scm-server/db.properties
echo “com.cloudera.cmf.db.name=cmf” >> /etc/cloudera-scm-server/db.properties
echo “com.cloudera.cmf.db.user=cmf” >> /etc/cloudera-scm-server/db.properties
echo “com.cloudera.cmf.db.password=cmf_password” >> /etc/cloudera-scm-server/db.properties - 运行 service cloudera-scm-server start 启动 cloudera-scm 服务
- 运行 tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log 查看运行日志,待日志停止且显示Started Jetty server.表示服务运行完成。
- 与浏览器输入 http://192.168.75.136:7180开始正式的集群部署
- 默认用户名密码都为admin
- 同意并继续
- 选择适合自己的版本并继续
- 点击继续
- 填入3台服务器的ip并点击搜索
- 确认可行后点击继续
- 与浏览器访问 http://192.168.75.136/parcels/确认parcels服务,若出现如图则表示ok
- 与浏览器访问 http://192.168.75.136/cm5/redhat/7/x86_64/cm/5/ 确认cm5服务,若出现如图则表示ok
- 点击更多选项配置parcel服务
- 填入之前部署的本地parcel服务并删除多余的url点击保存更改
- 点击自定义存储库配置cm5服务
- 配置cm5服务并点击继续
- 直接点击继续
- 直接点击继续
- 选择合适的登陆配置(密码或私钥),点击继续
- 等待安装完成点击继续
- 等待安装完parcel点击继续
- 检测完成后点击完成(这里忽略这两个警告,可自行修复)
- 这里选择所有服务以测试是否可以完成所有服务的安装
- 这里直接点继续,如果需要自定义可自行定义。
- 填写创建好的数据库信息测试链接ok后点击继续
- 这里直接点击继续(用户可自行配置)
- 等都通过后点击继续,如果hive的安装出现问题可参考文件末尾教程再操作。
- 点击完成,完成安装
- 至此完成安装
- 运行
- 文章的最后附上个人的脚本(吐槽下简书的书写为了格式会导致粘贴部分文案行不对号,如果要使用这个脚本需要更改( 主机ip,主机名)):
- 最后为了使脚本能够运行,需要运行 chmod +x filepath 为文件添加权限,运行时执行 bash filepath n 例如(bash cdh.sh 0),运行此脚本可能mysql初始化不完全,如果可以,最好还是弄懂所有的步骤。
====================================================================================================master
#!/bin/bash
c_index=$1
m_names=(“master” “slave01” “slave02”)
m_hosts=(“192.168.1.131” “192.168.1.129” “192.168.1.226”)
function auto_ssh_keygen(){
expect -c “set timeout -1;
spawn ssh-keygen;
expect {
id_rsa {
send \n\r;
exp_continue;
};
Overwrite {
send y\n\r;
exp_continue;
};
passphrase {
send \n\r;
exp_continue;
};
again {
send \n\r;
exp_continue;
};
};
”
}
function auto_ssh_copy(){
expect -c “set timeout -1;
spawn ssh-copy-id $1;
expect {
yes/no {
send yes\n\r;
exp_continue;
};
password {
send 2\n\r;
exp_continue;
};
};
”
}
#初始化必要工具
yum install expect -y
yum install unzip -y
#关闭selinux
setenforce 0
echo “SELINUX=disabled” > /etc/selinux/config
echo “SELINUXTYPE=targeted” >> /etc/selinux/config
#关闭防火墙
systemctl stop firewalld.service
systemctl status firewalld.service
systemctl disable firewalld.service
#定义主机名
hostname 2\n\r; exp_continue; }; }; ” } #初始化必要工具 yum install expect -y yum install unzip -y #关闭selinux setenforce 0 echo “SELINUX=disabled” > /etc/selinux/config echo “SELINUXTYPE=targeted” >> /etc/selinux/config #关闭防火墙 systemctl stop firewalld.service systemctl status firewalld.service systemctl disable firewalld.service #定义主机名 hostname {m_names[{c_index}]}
echo “HOSTNAME= {c_index}]} echo “HOSTNAME= {m_names[{c_index}]}” > /etc/sysconfig/network
#ip与hostname绑定
echo “ {c_index}]}” > /etc/sysconfig/network #ip与hostname绑定 echo “ {m_names[{c_index}]}” > /etc/hostname
t_index=0
for m_name in {c_index}]}” > /etc/hostname t_index=0 for m_name in {m_names[@]};
do
echo “ mhosts[tindex] m h o s t s [ t i n d e x ] m_name” >> /etc/hosts
let t_index++
done
#时间同步
yum install ntp -y
systemctl enable ntpd
if [ {c_index} == 0 ]
then
echo “driftfile /var/lib/ntp/ntp.drift” > /etc/ntp.conf
echo “restrict 192.168.75.0 mask 255.255.255.0 nomodify notrap” >> /etc/ntp.conf
echo “server 210.72.145.44” >> /etc/ntp.conf
echo “server 202.112.10.36 perfer” >> /etc/ntp.conf
echo “server 59.124.196.83” >> /etc/ntp.conf
echo “restrict 210.72.145.44 nomodify notrap noquery” >> /etc/ntp.conf
echo “restrict 202.112.10.36 nomodify notrap noquery” >> /etc/ntp.conf
echo “restrict 59.124.196.83 nomodify notrap noquery” >> /etc/ntp.conf
echo “server 127.127.1.0” >> /etc/ntp.conf
echo “fudge 127.127.1.0 stratum 10” >> /etc/ntp.conf
service ntpd restart
ntpdate –u 202.112.10.36
#免密登陆
for m_name in {c_index} == 0 ] then echo “driftfile /var/lib/ntp/ntp.drift” > /etc/ntp.conf echo “restrict 192.168.75.0 mask 255.255.255.0 nomodify notrap” >> /etc/ntp.conf echo “server 210.72.145.44” >> /etc/ntp.conf echo “server 202.112.10.36 perfer” >> /etc/ntp.conf echo “server 59.124.196.83” >> /etc/ntp.conf echo “restrict 210.72.145.44 nomodify notrap noquery” >> /etc/ntp.conf echo “restrict 202.112.10.36 nomodify notrap noquery” >> /etc/ntp.conf echo “restrict 59.124.196.83 nomodify notrap noquery” >> /etc/ntp.conf echo “server 127.127.1.0” >> /etc/ntp.conf echo “fudge 127.127.1.0 stratum 10” >> /etc/ntp.conf service ntpd restart ntpdate –u 202.112.10.36 #免密登陆 for m_name in {m_names[@]};
do
auto_ssh_copy mnamerootdoneelseecho“driftfile/var/lib/ntp/ntp.drift”>/etc/ntp.confecho“statisticsloopstatspeerstatsclockstats”>>/etc/ntp.confecho“filegenloopstatsfileloopstatstypedayenable”>>/etc/ntp.confecho“filegenpeerstatsfilepeerstatstypedayenable”>>/etc/ntp.confecho“filegenclockstatsfileclockstatstypedayenable”>>/etc/ntp.confecho“server m n a m e r o o t d o n e e l s e e c h o “ d r i f t f i l e / v a r / l i b / n t p / n t p . d r i f t ” > / e t c / n t p . c o n f e c h o “ s t a t i s t i c s l o o p s t a t s p e e r s t a t s c l o c k s t a t s ” >> / e t c / n t p . c o n f e c h o “ f i l e g e n l o o p s t a t s f i l e l o o p s t a t s t y p e d a y e n a b l e ” >> / e t c / n t p . c o n f e c h o “ f i l e g e n p e e r s t a t s f i l e p e e r s t a t s t y p e d a y e n a b l e ” >> / e t c / n t p . c o n f e c h o “ f i l e g e n c l o c k s t a t s f i l e c l o c k s t a t s t y p e d a y e n a b l e ” >> / e t c / n t p . c o n f e c h o “ s e r v e r {m_hosts[0]} prefer” >> /etc/ntp.conf
echo “fudge mhosts[0]stratum5”>>/etc/ntp.confecho“restrict−4defaultkodnotrapnomodifynopeernoquery”>>/etc/ntp.confecho“restrict−6defaultkodnotrapnomodifynopeernoquery”>>/etc/ntp.confecho“restrict127.0.0.1”>>/etc/ntp.confecho“restrict::1”>>/etc/ntp.confservicentpdrestartntpdate–u m h o s t s [ 0 ] s t r a t u m 5 ” >> / e t c / n t p . c o n f e c h o “ r e s t r i c t − 4 d e f a u l t k o d n o t r a p n o m o d i f y n o p e e r n o q u e r y ” >> / e t c / n t p . c o n f e c h o “ r e s t r i c t − 6 d e f a u l t k o d n o t r a p n o m o d i f y n o p e e r n o q u e r y ” >> / e t c / n t p . c o n f e c h o “ r e s t r i c t 127.0.0.1 ” >> / e t c / n t p . c o n f e c h o “ r e s t r i c t :: 1 ” >> / e t c / n t p . c o n f s e r v i c e n t p d r e s t a r t n t p d a t e – u {m_hosts[0]}
fi
#生成私钥
rm -rf /root/.ssh/id_rsa
rm -rf /root/.ssh/id_rsa.pub
auto_ssh_keygen
#安装jdk
rpm -Uvh jdk-8u121-linux-x64.rpm
echo “JAVA_HOME=/usr/java/jdk1.8.0_121” > /etc/profile.d/java.sh
echo “PATH=$PATH:$JAVA_HOME/bin” >> /etc/profile.d/java.sh
echo “CLASSPATH=.:$JAVA_HOME/lib/jt.jar:$JAVA_HOME/lib/tools.jar” >> /etc/profile.d/java.sh
echo “export JAVA_HOME PATH CLASSPATH” >> /etc/profile.d/java.sh
source /etc/profile
if [ {c_index} == 0 ]
then
#安装并初始化配置mysql
tar -zxvf mysql-5.6.36-linux-glibc2.5-x86_64.tar.gz
mv mysql-5.6.36-linux-glibc2.5-x86_64 /usr/local/mysql
touch /etc/my.cnf
echo “[mysql]” > /etc/my.cnf
echo “default-character-set=utf8” >> /etc/my.cnf
echo “socket=/var/lib/mysql/mysql.sock” >> /etc/my.cnf
echo “[mysqld]” >> /etc/my.cnf
echo “skip-name-resolve” >> /etc/my.cnf
echo “port=3306” >> /etc/my.cnf
echo “socket=/var/lib/mysql/mysql.sock” >> /etc/my.cnf
echo “basedir=/usr/local/mysql” >> /etc/my.cnf
echo “datadir=/usr/local/mysql/data” >> /etc/my.cnf
echo “max_connections=200” >> /etc/my.cnf
echo “character-set-server=utf8” >> /etc/my.cnf
echo “default-storage-engine=INNODB” >> /etc/my.cnf
echo “lower_case_table_names=1” >> /etc/my.cnf
echo “max_allowed_packet=16M” >> /etc/my.cnf
mkdir -p /var/lib/mysql/mysql
groupadd mysql
useradd -g mysql mysql
chown -R mysql:mysql /usr/local/mysql
chown -R mysql:mysql /var/lib/mysql
yum install -y perl-Module-Install.noarch
/usr/local/mysql/scripts/mysql_install_db –user=mysql –basedir=/usr/local/mysql
chown -R mysql:mysql /usr/local/mysql
chown 777 /etc/my.cnf
cp /usr/local/mysql/support-files/mysql.server /etc/rc.d/init.d/mysqld
chmod +x /etc/rc.d/init.d/mysqld
chkconfig –add mysqld
echo “export PATH=\$PATH:/usr/local/mysql/bin” > /etc/profile.d/mysql.sh
source /etc/profile
service mysqld restart
/usr/local/mysql/bin/mysql -e “delete from mysql.user where user = ””
/usr/local/mysql/bin/mysql -e “update mysql.user set password = (select password(‘root’)) where user = ‘root’”
/usr/local/mysql/bin/mysql -e “create database cmf default character set utf8”
/usr/local/mysql/bin/mysql -e “grant all on cmf.* to ‘cmf’@’localhost’ identified by ‘cmf_password’”
/usr/local/mysql/bin/mysql -e “grant all on cmf.* to ‘cmf’@’%’ identified by ‘cmf_password’”
/usr/local/mysql/bin/mysql -e “grant all on cmf.* to ‘cmf’@’ {c_index} == 0 ] then #安装并初始化配置mysql tar -zxvf mysql-5.6.36-linux-glibc2.5-x86_64.tar.gz mv mysql-5.6.36-linux-glibc2.5-x86_64 /usr/local/mysql touch /etc/my.cnf echo “[mysql]” > /etc/my.cnf echo “default-character-set=utf8” >> /etc/my.cnf echo “socket=/var/lib/mysql/mysql.sock” >> /etc/my.cnf echo “[mysqld]” >> /etc/my.cnf echo “skip-name-resolve” >> /etc/my.cnf echo “port=3306” >> /etc/my.cnf echo “socket=/var/lib/mysql/mysql.sock” >> /etc/my.cnf echo “basedir=/usr/local/mysql” >> /etc/my.cnf echo “datadir=/usr/local/mysql/data” >> /etc/my.cnf echo “max_connections=200” >> /etc/my.cnf echo “character-set-server=utf8” >> /etc/my.cnf echo “default-storage-engine=INNODB” >> /etc/my.cnf echo “lower_case_table_names=1” >> /etc/my.cnf echo “max_allowed_packet=16M” >> /etc/my.cnf mkdir -p /var/lib/mysql/mysql groupadd mysql useradd -g mysql mysql chown -R mysql:mysql /usr/local/mysql chown -R mysql:mysql /var/lib/mysql yum install -y perl-Module-Install.noarch /usr/local/mysql/scripts/mysql_install_db –user=mysql –basedir=/usr/local/mysql chown -R mysql:mysql /usr/local/mysql chown 777 /etc/my.cnf cp /usr/local/mysql/support-files/mysql.server /etc/rc.d/init.d/mysqld chmod +x /etc/rc.d/init.d/mysqld chkconfig –add mysqld echo “export PATH=\$PATH:/usr/local/mysql/bin” > /etc/profile.d/mysql.sh source /etc/profile service mysqld restart /usr/local/mysql/bin/mysql -e “delete from mysql.user where user = ”” /usr/local/mysql/bin/mysql -e “update mysql.user set password = (select password(‘root’)) where user = ‘root’” /usr/local/mysql/bin/mysql -e “create database cmf default character set utf8” /usr/local/mysql/bin/mysql -e “grant all on cmf.* to ‘cmf’@’localhost’ identified by ‘cmf_password’” /usr/local/mysql/bin/mysql -e “grant all on cmf.* to ‘cmf’@’%’ identified by ‘cmf_password’” /usr/local/mysql/bin/mysql -e “grant all on cmf.* to ‘cmf’@’ {m_hosts[{c_index}]}’ identified by ‘cmf_password’”
/usr/local/mysql/bin/mysql -e “drop database cmf”
/usr/local/mysql/bin/mysql -e “create database cmf”
/usr/local/mysql/bin/mysql -e “create database hive default character set utf8”
/usr/local/mysql/bin/mysql -e “grant all on hive.* to ‘hive’@’localhost’ identified by ‘hive_password’”
/usr/local/mysql/bin/mysql -e “grant all on hive.* to ‘hive’@’%’ identified by ‘hive_password’”
/usr/local/mysql/bin/mysql -e “grant all on hive.* to ‘hive’@’ {c_index}]}’ identified by ‘cmf_password’” /usr/local/mysql/bin/mysql -e “drop database cmf” /usr/local/mysql/bin/mysql -e “create database cmf” /usr/local/mysql/bin/mysql -e “create database hive default character set utf8” /usr/local/mysql/bin/mysql -e “grant all on hive.* to ‘hive’@’localhost’ identified by ‘hive_password’” /usr/local/mysql/bin/mysql -e “grant all on hive.* to ‘hive’@’%’ identified by ‘hive_password’” /usr/local/mysql/bin/mysql -e “grant all on hive.* to ‘hive’@’ {m_hosts[{c_index}]}’ identified by ‘hive_password’”
/usr/local/mysql/bin/mysql -e “drop database hive”
/usr/local/mysql/bin/mysql -e “create database hive”
/usr/local/mysql/bin/mysql -e “create database hue default character set utf8”
/usr/local/mysql/bin/mysql -e “grant all on hue.* to ‘hue’@’localhost’ identified by ‘hue_password’”
/usr/local/mysql/bin/mysql -e “grant all on hue.* to ‘hue’@’%’ identified by ‘hue_password’”
/usr/local/mysql/bin/mysql -e “grant all on hue.* to ‘hue’@’ {c_index}]}’ identified by ‘hive_password’” /usr/local/mysql/bin/mysql -e “drop database hive” /usr/local/mysql/bin/mysql -e “create database hive” /usr/local/mysql/bin/mysql -e “create database hue default character set utf8” /usr/local/mysql/bin/mysql -e “grant all on hue.* to ‘hue’@’localhost’ identified by ‘hue_password’” /usr/local/mysql/bin/mysql -e “grant all on hue.* to ‘hue’@’%’ identified by ‘hue_password’” /usr/local/mysql/bin/mysql -e “grant all on hue.* to ‘hue’@’ {m_hosts[{c_index}]}’ identified by ‘hue_password’”
/usr/local/mysql/bin/mysql -e “drop database hue”
/usr/local/mysql/bin/mysql -e “create database hue”
/usr/local/mysql/bin/mysql -e “create database am default character set utf8”
/usr/local/mysql/bin/mysql -e “grant all on am.* to ‘am’@’localhost’ identified by ‘am_password’”
/usr/local/mysql/bin/mysql -e “grant all on am.* to ‘am’@’%’ identified by ‘am_password’”
/usr/local/mysql/bin/mysql -e “grant all on am.* to ‘am’@’ {c_index}]}’ identified by ‘hue_password’” /usr/local/mysql/bin/mysql -e “drop database hue” /usr/local/mysql/bin/mysql -e “create database hue” /usr/local/mysql/bin/mysql -e “create database am default character set utf8” /usr/local/mysql/bin/mysql -e “grant all on am.* to ‘am’@’localhost’ identified by ‘am_password’” /usr/local/mysql/bin/mysql -e “grant all on am.* to ‘am’@’%’ identified by ‘am_password’” /usr/local/mysql/bin/mysql -e “grant all on am.* to ‘am’@’ {m_hosts[{c_index}]}’ identified by ‘am_password’”
/usr/local/mysql/bin/mysql -e “drop database am”
/usr/local/mysql/bin/mysql -e “create database am”
/usr/local/mysql/bin/mysql -e “create database os default character set utf8”
/usr/local/mysql/bin/mysql -e “grant all on os.* to ‘os’@’localhost’ identified by ‘os_password’”
/usr/local/mysql/bin/mysql -e “grant all on os.* to ‘os’@’%’ identified by ‘os_password’”
/usr/local/mysql/bin/mysql -e “grant all on os.* to ‘os’@’ {c_index}]}’ identified by ‘am_password’” /usr/local/mysql/bin/mysql -e “drop database am” /usr/local/mysql/bin/mysql -e “create database am” /usr/local/mysql/bin/mysql -e “create database os default character set utf8” /usr/local/mysql/bin/mysql -e “grant all on os.* to ‘os’@’localhost’ identified by ‘os_password’” /usr/local/mysql/bin/mysql -e “grant all on os.* to ‘os’@’%’ identified by ‘os_password’” /usr/local/mysql/bin/mysql -e “grant all on os.* to ‘os’@’ {m_hosts[${c_index}]}’ identified by ‘os_password’”
/usr/local/mysql/bin/mysql -e “drop database os”
/usr/local/mysql/bin/mysql -e “create database os”
/usr/local/mysql/bin/mysql -e “flush privileges”
#安装httpd服务
yum install -y httpd
systemctl enable httpd
service httpd restart
cd /var/www/html
mkdir parcels
cd parcels
mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel ./
mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha ./
mv ~/manifest.json ./
mkdir /opt/rpminstall
cd /opt/rpminstall
mv ~/cm5.11.1-centos7.tar.gz ./
mv ~/cm5.11.1-centos7.tar.gz.sha ./
tar -zxf cm5.11.1-centos7.tar.gz -C /var/www/html/
cd /var/www/html/
mkdir -p cm5/redhat/7/x86_64/
mv cm cm5/redhat/7/x86_64/
chown apache:apache -R /var/www/html
cd /var/www/html/cm5/redhat/7/x86_64/cm/5/RPMS/x86_64
yum install -y cloudera-manager-daemons-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm
yum install -y cloudera-manager-server-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm
mkdir /usr/share/java
cd /usr/share/java
mv ~/mysql-connector-java-5.1.42.zip ./
unzip mysql-connector-java-5.1.42.zip
cd mysql-connector-java-5.1.42
cp mysql-connector-java-5.1.42-bin.jar ../mysql-connector-java.jar
echo “com.cloudera.cmf.db.type=mysql” > /etc/cloudera-scm-server/db.properties
echo “com.cloudera.cmf.db.host=localhost” >> /etc/cloudera-scm-server/db.properties
echo “com.cloudera.cmf.db.name=cmf” >> /etc/cloudera-scm-server/db.properties
echo “com.cloudera.cmf.db.user=cmf” >> /etc/cloudera-scm-server/db.properties
echo “com.cloudera.cmf.db.password=cmf_password” >> /etc/cloudera-scm-server/db.properties
service cloudera-scm-server restart
source /etc/profile
fi
======================================================================================================end 坑锦集:
- 脚本运行完成后可能需要再次于master运行source /etc/profile 使环境变量生效。
- 如果遇到hive安装失败的问题可能需要再开个浏览器窗口访问cdh搭建主网站更改hive配置,且需要使用独立的mysql数据库及用户。
更改hive配置
ntp时间同步服务如果报警,可于crontab -e添加定时同步时间。
- 当3台机器服务时,可将master也添加namenode节点。