centos7下搭建CM5.14和CDH5.14集群

纯手打文字，图片太多，不懂的地方可自行百度

下载安装包（要求版本对应）:
JDK:
    jdk1.8.0_181
CM:
    cloudera-manager-centos7-cm5.14.1_x86_64.tar.gz
CDH:
    CDH-5.14.4-1.cdh5.14.4.p0.3-el7.parcel.sha1
    CDH-5.14.4-1.cdh5.14.4.p0.3-el7.parcel
    manifest.json
MYSQL：
    mysql-5.7.24-1.el7.x86_64.rpm-bundle.tar
    mysql-connector-java-5.1.32.jar
    ojdbc6.jar

一、修改主机名：
所有主机登录密码：
192.168.9.1xx   用户名：root  密码：xxxxxxxxx
192.168.9.1xx   用户名：root  密码：xxxxxxxxx
192.168.9.1xx   用户名：root  密码：xxxxxxxxx
192.168.9.1xx   用户名：root  密码：xxxxxxxxx
192.168.9.1xx   用户名：root  密码：xxxxxxxxx
192.168.9.1xx   用户名：root  密码：xxxxxxxxx
192.168.9.1xx   用户名：root  密码：xxxxxxxxx
192.168.9.1xx   用户名：root  密码：xxxxxxxxx

在主机上
[root@ztmy9_1xx ~]# vim /etc/hosts
192.168.9.1xx cdh0
192.168.9.1xx cdh1
192.168.9.1xx cdh2
192.168.9.1xx cdh3
192.168.9.1xx cdh4
192.168.9.1xx cdh5
192.168.9.1xx cdh6
192.168.9.1xx cdh7
保存:wq
[root@cdh0 ~]# cat /etc/hosts
分配给其他机器
[root@cdh0 ~]# scp /etc/hosts 192.168.9.1xx:/etc/hosts
[root@cdh0 ~]# scp /etc/hosts 192.168.9.1xx:/etc/hosts
......
二、关闭selinux（每一台）
[root@cdh0 ~]# vim /etc/sysconfig/selinux
SELINUX=disabled
保存:wq
[root@cdh0 ~]# reboot
[root@cdh0 ~]# sestatus -v
三、关闭防火墙（每一台）
[root@cdh0 ~]# systemctl stop firewalld
[root@cdh0 ~]# systemctl disable firewalld
[root@cdh0 ~]# systemctl status firewalld
四、禁止交换（每一台）
[root@cdh0 ~]# vim /etc/sysctl.conf
vm.swappiness=0
[root@cdh0 ~]# sysctl vm.swappiness=0
五、禁用大页面
[root@cdh0 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@cdh0 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@cdh0 ~]# vim /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@cdh0 ~]# chmod +x /etc/rc.d/rc.local
配置ntp服务（每一台）
yum源下载
[root@cdh1 ~]# yum -y install ntp
开启
[root@cdh1 ~]# service ntpd start
查看
[root@cdh1 ~]# systemctl status ntpd.service
配置ntp服务（server主机）
[root@cdh0 cloudera-scm-agent]# vim /etc/ntp.conf
添加一句（自己网络的网关）
restrict 192.168.9.0 mask 255.255.255.0 nomodify notrap
配置ntp服务（agent主机）
[root@cdh1 ~]# vim /etc/ntp.conf
添加一句（server服务主机ip）
server 192.168.9.1xx
并注释掉
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
然后保存
重启NTP服务（每一台）
[root@cdh1 ~]# systemctl restart ntpd
等几分钟查看同步效果
[root@cdh1 ~]# ntpstat

六、安装mysql
查看自带的mysql依赖
[root@ztmy9_1xx ~]# rpm -qa | grep -i  mariadb
[root@ztmy9_1xx ~]# rpm -qa | grep  mysql
删除依赖
[root@ztmy9_1xx ~]# rpm -e mariadb-libs-5.5.65-1.el7.x86_64 --nodeps
两种方式安装：
1、离线rpm包安装：
把我们之前下载好的mysql安装文件MySQL-5.6.34-1.linux_glibc2.5.x86_64.rpm-bundle.tar用工具上传到主机/opt目录下。
新建一个目录/opt/mysql，把mysql安装文件解压到该文件夹下。
解压mysql文件的目录里发现了除了
执行解压：tar -xvf MySQL-5.6.34-1.linux_glibc2.5.x86_64.rpm-bundle.tar
依次安装：
[root@ztmy9_1xx mysql]#  rpm -ivh mysql-community-common-5.7.24-1.el7.x86_64.rpm
[root@ztmy9_1xx mysql]#  rpm -ivh mysql-community-libs-5.7.24-1.el7.x86_64.rpm
[root@ztmy9_1xx mysql]#  rpm -ivh mysql-community-devel-5.7.24-1.el7.x86_64.rpm
[root@ztmy9_1xx mysql]#  rpm -ivh mysql-community-libs-compat-5.7.24-1.el7.x86_64.rpm
[root@ztmy9_1xx mysql]#  rpm -ivh  mysql-community-client-5.7.24-1.el7.x86_64.rpm
报错
[root@ztmy9_1xx mysql]#  rpm -ivh  mysql-community-server-5.7.24-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.24-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
error: Failed dependencies:
        net-tools is needed by mysql-community-server-5.7.24-1.el7.x86_64
        
缺少net-tools依赖
安装
[root@cdh0 mysql]# yum install net-tools
启动mysql报错
Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
/etc/my.cnf的validate_password=off注释掉
启动mysql
[root@cdh0 mysql]# service mysqld start
2、在线yum源安装：
依次执行命令，根据提示信息操作
[root@cdh0 mysql]# wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm
[root@cdh0 mysql]# rpm -ivh mysql57-community-release-el7-8.noarch.rpm
[root@cdh0 mysql]# yum -y install mysql-server
[root@cdh0 mysql]# service mysqld start
修改mysql配置：
[root@cdh0 mysql]# vim /etc/my.cnf
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html

[client]
default-character-set=utf8

[mysqld]
character-set-server=utf8
collation-server=utf8_general_ci

#允许使用简单密码

validate_password=off

#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
重启
[root@cdh0 mysql]# systemctl restart mysqld.service
登录 
查看默认密码：grep "password" /var/log/mysqld.log  或者 cat /var/log/mysqld.log |grep "passw"
[root@cdh0 mysql]# mysql -u root -p tdhKwdfpa4)F
修改初始密码：
[root@cdh0 mysql]# mysql -u root -p
mysql> alter user 'root'@'localhost' identified by 'xxxxx';
创建用户以及授权
mysql> grant all on *.* to 'root'@'%' identified by 'xxxxx'  with grant option;
mysql> CREATE USER 'scm'@'%' IDENTIFIED BY 'scm';（cm的元数据库时使用）
mysql> grant all on *.* to 'scm'@'%' identified by 'scm'  with grant option;
mysql> CREATE USER 'sen'@'%' IDENTIFIED BY 'sen';（其他的元数据库时使用）
mysql> grant all on *.* to 'sen'@'%' identified by 'sen'  with grant option;
刷新：
mysql> flush privileges; 
使用sen登录mysql
创建数据库
mysql> create database hive;
mysql> create database hue;
mysql> create database azkaban;
mysql> create database oozie;
使用scm登录mysql
创建数据库
create database scm;
退出quit
七、安装CM和CDH
创建目录（每一台）
[root@cdh0 ~]# mkdir /opt/cloudera-manager
分发
[root@cdh0 cloudera-manager]# scp /opt/cloudera-manager/cloudera-manager-centos7-cm5.14.1_x86_64.tar.gz cdh1:/opt/cloudera-manager/
[root@cdh0 cloudera-manager]# scp /opt/cloudera-manager/cloudera-manager-centos7-cm5.14.1_x86_64.tar.gz cdh2:/opt/cloudera-manager/
[root@cdh0 cloudera-manager]# scp /opt/cloudera-manager/cloudera-manager-centos7-cm5.14.1_x86_64.tar.gz cdh3:/opt/cloudera-manager/
....
解压（每一台）：
[root@cdh0 ~]# tar -zxvf /opt/cloudera-manager-centos7-cm5.14.1_x86_64.tar.gz -C /opt/cloudera-manager
在所有节点，创建用户，这是CM需要使用的用户（cloudera-scm）
[root@cdh0 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.14.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
在server节点，创建CM服务本地数据存放目录，并赋予权限
[root@cdh0 ~]# mkdir /var/lib/cloudera-scm-server
[root@cdh0 ~]# chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server
在所有的agent节点，执行命令
[root@cdh1 cloudera-manager]# vim /opt/cloudera-manager/cm-5.14.1/etc/cloudera-scm-agent/config.ini
# 将sever主机名写上 
server_host=cdh0
将mysql驱动包传给每一台主机
[root@cdh0 java]# scp ./mysql-connector-java.jar master_cdh1:/usr/share/java/
......
初始化数据库scm
在CM的server主机和mysql主机在一台主机上
[root@cdh0 ~]#  /opt/cloudera-manager/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh mysql scm scm scm
在CM的server主机和mysql主机不在一台主机上
[root@cdh0 ~]#  /opt/cloudera-manager/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh -h 192.168.9.181 mysql scm scm scm
创建CDH目录
[root@cdh0 ~]#  mkdir -P /opt/cloudera/parcel-repo
将3个CDH文件上传至上传至/opt/cloudera/parcel-repo路径下（CDH-5.14.4-1.cdh5.14.4.p0.3-el7.parcel.sha1 去掉后面的1）
CDH-5.14.4-1.cdh5.14.4.p0.3-el7.parcel
CDH-5.14.4-1.cdh5.14.4.p0.3-el7.parcel.sha
manifest.json
在server节点，启动CM server
[root@cdh0 ~]# /opt/cloudera-manager/cm-5.14.1/etc/init.d/cloudera-scm-server start
在agent节点，启动CM server
[root@cdh1 ~]# /opt/cloudera-manager/cm-5.14.1/etc/init.d/cloudera-scm-agent start
访问网页：http://192.168.9.1xx:7180/cmf/login:7180  出现界面如下，用户名密码都是admin
填写我们集群定义的ip或者服务器名称（包括安装CM的主机本身）
用同一个密码：xxxxxxxxx

报错目录：
ps -ef | grep supervisord
watch netstat -lntp

agent安装报错：
tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.log' for reading: No such file or directory 等等 
修改主机名  不要带特殊符号 - * 空格等等 
主机检查报错
cdh0: Inspector did not run successfully.
0主机和1-7主机jdk环境变量配置不同
export JAVA_HOME=/opt/jdk1.8.0_181
export JAVA_HOME=/opt/jdk1.8.0_51

卸载
用户root
/opt/cloudera-manager/cm-5.14.1/etc/init.d/cloudera-scm-server stop
/opt/cloudera-manager/cm-5.14.1/etc/init.d/cloudera-scm-agent stop
umount cm_processes
rm -rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*
rm -rf /tmp/.scmpreparenode.lock
rm -rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper
rm -rf datadrivepath/dfs datadrivepath/mapred datadrivepath/yarn
rm -rf /var/lib/hadoop-* /var/lib/impala /var/lib/solr /var/lib/zookeeper /var/lib/hue /var/lib/oozie  /var/lib/pgsql  /var/lib/sqoop2  /data/dfs/  /data/impala/ /data/yarn/  /dfs/ /impala/ /yarn/  /var/run/hadoop-*/ /var/run/hdfs-*/ /usr/bin/hadoop* /usr/bin/zookeeper* /usr/bin/hbase* /usr/bin/hive* /usr/bin/hdfs /usr/bin/mapred /usr/bin/yarn /usr/bin/sqoop* /usr/bin/oozie /etc/hadoop* /etc/zookeeper* /etc/hive* /etc/hue /etc/impala /etc/sqoop* /etc/oozie /etc/hbase* /etc/hcatalog
systemctl stop mysqld.service
rm -rf /var/lib/mysql
rm -rf /var/log/mysqld.log
rm -rf /var/lib/mysql/mysql
rm -rf /usr/lib64/mysql
rm -rf /usr/share/mysql
rm -rf /opt/cloudera

[root@master_cdh0 opt]# rpm -qa | grep -i mysql
mysql-community-libs-5.7.24-1.el7.x86_64
mysql-community-common-5.7.24-1.el7.x86_64
mysql-community-devel-5.7.24-1.el7.x86_64
mysql-community-libs-compat-5.7.24-1.el7.x86_64
MySQL-python-1.2.5-1.el7.x86_64

hive的外键操作
删除外键
alter table CLIENT_CONFIGS_TO_HOSTS drop foreign key FK_CLIENT_CFG2HOST_HOST;
alter table COMMANDS drop foreign key FK_COMMAND_HOST;
增加外键
alter table CLIENT_CONFIGS_TO_HOSTS add foreign key FK_CLIENT_CFG2HOST_HOST(HOST_ID) references HOSTS(HOST_ID);
alter table COMMANDS add foreign key FK_COMMAND_HOST(HOST_ID) references HOSTS(HOST_ID);

重装准备：
1.停止所有cloudera-scm-agent服务 和cloudera-scm-server服务
/opt/cloudera-manager/cm-5.14.1/etc/init.d/cloudera-scm-server stop
/opt/cloudera-manager/cm-5.14.1/etc/init.d/cloudera-scm-agent stop
2. 删除scm数据库，然后重新建立scm数据库
3. 每台服务器上删除数据：
rm -rf /opt/cloudera-manager/cm-5.14.1/lib/cloudera-scm-agent/*
（每台服务器上都执行）
如果删除不了执行以下命令再删除：
sudo umount /opt/cloudera-manager/cm-5.14.1/run/cloudera-scm-agent/process
4. 执行数据库初始化
/opt/cloudera-manager/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh mysql scm scm scm
（cdh01机器上执行）
5. 重启服务
/opt/cloudera-manager/cm-5.14.1/etc/init.d/cloudera-scm-server start
/opt/cloudera-manager/cm-5.14.1/etc/init.d/cloudera-scm-agent start

其他可能用到的命令
删除id
rm -f /opt/cloudera-manager/cm-5.14.1/lib/cloudera-scm-agent/cm_guid
rm -f /opt/cloudera-manager/cm-5.14.1/lib/cloudera-scm-agent/uuid
查看日志
tail -f /opt/cloudera-manager/cm-5.14.1/log/cloudera-scm-server/cloudera-scm-server.log
tail -f /opt/cloudera-manager/cm-5.14.1/log/cloudera-scm-agent/cloudera-scm-agent.log
删除agent所有
rm -rf /opt/cloudera-manager/cm-5.13.0/lib/cloudera-scm-agent/*

文件下载官网：

http://archive.cloudera.com/cm5/

参考文章：

http://blog.csdn.net/wulantian/article/details/42706777

https://yuhui.blog.csdn.net/article/details/54893826?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.channel_param&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.channel_param

https://blog.csdn.net/qq_37503890/article/details/103238799

https://blog.csdn.net/weixin_36439837/article/details/88758267

2021-08-18：更新百度云盘CDH软件包及主要文件包：

地址：https://pan.baidu.com/s/1WhptGU68m6t5DHurCA8Amg 

提取码: j2cv