CDH 6.3.2 集群搭建

1 资源准备

CDH 6.3.2 版本介绍
官方部署文档

1.1 rpm包

提前下载好CDH6.3.2相关rpm包

1.2 基础环境

关闭防火墙、修改hosts、SSH免密、时钟对齐、mysql部署均需提前弄好

1.3 安装jdk

rpm -qa | grep java
rpm -e --nodeps XXX
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

vim /etc/profile
source /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
export PATH=$PATH:${JAVA_HOME}/bin

1.4 mysql建库信息

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
GRANT ALL ON amon.* TO 'amon'@'qt-mr-header1' IDENTIFIED BY 'amon';
FLUSH PRIVILEGES;
show grants for 'amon'@'%';  
show grants for 'rman'@'%';  
show grants for 'hive'@'%';
show grants for 'hue'@'%';
show grants for 'root'@'%';

2 CDH搭建

2.1 主节点rpm安装

rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm 

2.2 从节点rpm安装

rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

2.3 所有节点驱动更新

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar -zxvf mysql-connector-java-5.1.46.tar.gz
cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

2.4 配置本地yum云

注意此处只需要在主节点机器上执行

#更新yum源httpd
yum -y install httpd
#查看httpd状态
systemctl status httpd.service
#启动httpd
service httpd start
#配置httpd永久生效（重启生效）
chkconfig httpd on

#更新yum源yum-utils createrepo
yum -y install yum-utils createrepo
#进入yum源路径
cd /var/www/html/
#创建cm文件夹
mkdir cm

将下载好的资源添加进本地yum云

# parcel包放到 /opt/cloudera/parcel-repo/  并且需要做校验
sha1sum CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel | awk '{ print $1 }' > CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

cd /var/www/html/cm
cp /u-data/clj/cdh6.3.2/cm6.3.1/* .
cp /u-data/clj/cdh6.3.2/cdh6.3.2/* .
createrepo /var/www/html/cm/

打开：http://172.17.2.143/cm/

创建本地repo文件（每个节点都要操作）

vim /etc/yum.repos.d/cloudera-manager.repo

---->添加如下内容(注意IP改成本地yum主机IP，只能是ip)
[cloudera-manager]
name=Cloudera Manager, Version yum
baseurl=http://172.17.2.143/cm
gpgcheck=0
enabled=1
----<

更新yum源
yum clean all
yum makecache

验证
yum list | grep cloudera-manager

2.5 安装

主节点

./cloudera-manager-installer.bin 

其他设置

1） 执行脚本scm_prepare_database.sh
#设置Cloudera Manager 数据库

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -uroot -pK0VFTN1EEEtx -P13306 scm scm scm

#进入mysql(这里是上面改的密码root)

mysql -uroot -pK0VFTN1EEEtx -P13306
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
FLUSH PRIVILEGES;
show grants for 'scm'@'%';
quit

2） 停止ClouderaManager服务

service cloudera-scm-agent stop
su mysql service mysql stop
service cloudera-scm-server-db stop
service cloudera-scm-server stop

3） 删除内嵌的默认数据库PostgreSQL的配置

rm -f /etc/cloudera-scm-server/db.mgmt.properties

4） 启动ClouderaManager服务

service cloudera-scm-server start

页面操作

7180页面开始部署，默认用户密码：admin/admin

选择自定义存储，更多选项里面新增刚刚创建的本地yum云

等待部署完成

配置所需数据库

蛮长的等待后----搭建完成（有一些页面报警需要自己处理）

ps.服务节点部署均衡策略 略。