基于Centos7.6安裝CDH6集群

前提：所有cdh相关安装文件均放在master节点（cdh节点）的/root/cdh目录下

1. 配置环境（所有节点）
   1.1配置主机名
   hostnamectl set-hostname cdh
   hostnamectl set-hostname cdh01
   hostnamectl set-hostname cdh02
   hostnamectl set-hostname cdh03
   注意：这四条命令各自在各自机器上执行，除了这四条命令，下面命令需在所有机器上执行。

1.2配置hosts文件
echo -e ‘10.0.24.6 cdh\n10.0.20.2 cdh01\n10.0.20.7 cdh02\n10.0.20.10 cdh03’ >> /etc/hosts

cat /etc/hosts

1.3关闭selinux
setenforce 0

1.4关闭防火墙
关闭防火墙
systemctl stop firewalld.service
关闭开机自启动
systemctl disable firewalld.service
查看防火墙状态
systemctl status firewalld.service

1.5配置免密登陆
ssh-keygen -t rsa
然后一路回车

ssh-copy-id -p6323 cdh
ssh-copy-id -p6323 cdh01
ssh-copy-id -p6323 cdh02
ssh-copy-id -p6323 cdh03
注意：这四条命令要在所有主机上执行，担心有的人不理解，特此提醒。

选择yes，然后会让你输入主机ssh连接的密码

注意：因为我用的是云服务器，之前服务器被黑，所以将ssh端口改为了6323。如果你未改过ssh端口，命令直接打ssh-copy-id cdh即可，若是换为了其它端口，将6323换为其它端口号即可。下面所有涉及-p6323指定端口号的操作均一样，就不一一介绍了。

设置authorized_keys权限（只是为了安全，可不设置）
chmod 600 /root/.ssh/authorized_keys

1.6配置时间同步
查看是否安装了ntp服务
rpm -qa | grep ntp

图中说明已经安装了ntp服务，若是没安装，则不会有任何返回

若是没有ntp服务，执行以下命令安装
yum install -y ntp

编辑ntp.conf
vim /etc/ntp.conf
加入以下内容（未测试，因使用腾讯云服务器，该部分被注释，添加的为云服务时间）
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

上面内容不行，可尝试下边内容
disable monitor
server time1.tencentyun.com iburst
server time2.tencentyun.com iburst
server time3.tencentyun.com iburst
server time4.tencentyun.com iburst
server time5.tencentyun.com iburst
interface ignore wildcard
interface listen eth0

启动ntp服务
systemctl start ntpd
设置开机自启动
systemctl enable ntpd
同步服务器时间
ntpdate -u 0.centos.pool.ntp.org
将硬件时钟调整为与目前的系统时钟一致
hwclock --systohc

1.7安装JDK
jdk需要安装在/usr/java目录下
mkdir /usr/java
cd /usr/java

拷贝安装包到当前目录（注意最后那个点是命令的一部分）
scp -P6323 cdh:/root/cdh/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm .

安装
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

安装完之后会出现jdk1.8.0_181-cloudera

配置环境变量
echo ‘export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera’ >> /etc/profile
echo ‘export JRE_HOME=$JAV{A}_{H}OME/jr{e}^{\prime }>>/etc/profileech{o}^{\prime }exportCLASSPATH=.:${JAVA_HOME}/lib:$JR{E}_{H}OME/lib:$CLASSPATH’ >> /etc/profile
echo ‘export JAVA_PATH=$JAV{A}_{H}OME/bin:${JRE_HOME}/bin’ >> /etc/profile
echo ‘export PATH=$PATH:${JAVA_PATH}’ >> /etc/profile
查看环境变量
cat /etc/profile

使环境变量生效
source /etc/profile

验证
java -version

1.8禁用透明大页
查看
cat /sys/kernel/mm/transparent_hugepage/defrag

cat /sys/kernel/mm/transparent_hugepage/enabled

echo never > /sys/kernel/mm/transparent_hugepage/defrag
执行完命令后，再查看

echo never > /sys/kernel/mm/transparent_hugepage/enabled
执行完命令后再查看

永久关闭（将命令加入开机启动）
echo ‘echo never > /sys/kernel/mm/transparent_hugepage/defrag’ >> /etc/rc.d/rc.local
echo ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled’ >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local

1.9修改linux swappiness参数
cat /proc/sys/vm/swappiness

sysctl vm.swappiness=10

修改生效
sysctl -p

永久修改（加入开机启动）
echo ‘vm.swappiness=10’>> /etc/sysctl.conf
cat /etc/sysctl.conf

2.安装mysql（master节点，安装目录/usr/local）
2.1下载mysql安装源
cd /usr/local
wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm

2.2安装mysql安装源
yum -y localinstall mysql57-community-release-el7-11.noarch.rpm

2.3在线安装mysql（这里取消了gpgkey验证）
yum -y install mysql-community-server --nogpgcheck

2.4启动mysql
systemctl start mysqld

2.5设置开机启动
systemctl enable mysqld
systemctl daemon-reload

2.6登陆mysql
寻找mysql密码
cat /var/log/mysqld.log

登陆
mysql -uroot -p’nl,pAIO4<5uX’

修改密码
ALTER USER ‘root’@‘localhost’ IDENTIFIED BY ‘P@ss123123’;

设置远程登陆
GRANT ALL PRIVILEGES ON . TO ‘root’@‘%’ IDENTIFIED BY ‘P@ss123123’ WITH GRANT OPTION;

2.7修改mysql配置文件/etc/my.cnf
源文件备份
mv /etc/my.cnf /etc/my.cnf.bk

创建新文件
vim /etc/my.cnf

添加以下内容
[client]
port=3306
socket=/var/lib/mysql/mysql.sock
[mysqld]
port=3306
user=mysql
socket=/var/lib/mysql/mysql.sock
basedir=/var/lib/mysql
datadir=/var/lib/mysql
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
transaction-isolation = READ-COMMITTED
character-set-server=utf8
collation-server=utf8_general_ci
lower_case_table_names=1
sql_mode=“STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO”

Disabling symbolic-links is recommended to prevent assorted security risks;

to do so, uncomment this line:

symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 16M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace ‘/var/lib/mysql/mysql_binary_log’ with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

InnoDB settings

innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

重启mysql
systemctl restart mysqld

2.8创建元数据库
重新登陆mysql
mysql -uroot -pP@ss123123

###降低密码安全策略验证等级强度
###set global validate_password_policy=LOW;

###修改密码最短位数
###set global validate_password_length=6;

创建元数据库
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO ‘scm’@‘%’ IDENTIFIED BY ‘scm@123’;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO ‘amon’@‘%’ IDENTIFIED BY ‘amon@123’;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO ‘rman’@‘%’ IDENTIFIED BY ‘rman@123’;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO ‘hue’@‘%’ IDENTIFIED BY ‘hue@123’;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO ‘metastore’@‘%’ IDENTIFIED BY ‘metastore@123’;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO ‘sentry’@‘%’ IDENTIFIED BY ‘sentry@123’;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO ‘nav’@‘%’ IDENTIFIED BY ‘nav@123’;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO ‘navms’@‘%’ IDENTIFIED BY ‘navms@123’;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO ‘oozie’@‘%’ IDENTIFIED BY ‘oozie@123’;
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hive.* TO ‘hive’@‘%’ IDENTIFIED BY ‘hive@123’;
CREATE DATABASE cm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON cm.* TO ‘cm’@‘%’ IDENTIFIED BY ‘cm@123’;

2.9安装mysql-connector（所有节点）
mkdir /root/cdh1
cd /root/cdh1

分发安装文件到所有节点
scp -P6323 cdh:/root/cdh/mysql-connector-java-8.0.22-1.el7.noarch.rpm .

安装
yum install -y /root/cdh1/mysql-connector-java-8.0.22-1.el7.noarch.rpm
安装完成后，在/usr/share/java下有如下文件

3.安装cloudera-manager
3.1制作yum源（master节点）
安装httpd
yum install -y httpd

启用httpd
systemctl start httpd
systemctl enable httpd
启用成功后，浏览器输入ip可以看到（cdh为我在本地hosts配置的映射）

mkdir -p /var/www/html/cloudera-repos/cm6
创建完此文件夹，可以看到如图

cp /root/cdh/*.rpm /var/www/html/cloudera-repos/cm6/
cp /root/cdh/allkeys.asc /var/www/html/cloudera-repos/cm6/

cd /var/www/html/cloudera-repos/cm6

安装createrepo（制作yum源的包）
yum install -y createrepo

制作yum源
createrepo .

同时该目录下会多一个文件夹

创建文件cloudera-manager.repo（需要所有节点都有这个文件）
vim /etc/yum.repos.d/cloudera-manager.repo
加入以下内容
[cloudera-manager]
name=Cloudera Manager 6.3.1
baseurl=http://cdh/cloudera-repos/cm6
gpgkey=https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-cloudera
gpgcheck=0
enabled=1
autorefresh=0
type=rpm-md

分发到其它节点
scp -P6323 /etc/yum.repos.d/cloudera-manager.repo cdh01:/etc/yum.repos.d/
scp -P6323 /etc/yum.repos.d/cloudera-manager.repo cdh02:/etc/yum.repos.d/
scp -P6323 /etc/yum.repos.d/cloudera-manager.repo cdh03:/etc/yum.repos.d/

清楚yum缓存
yum clean all

将安装包进行缓存
yum makecache

这三个出现说明成功

3.2安装cloudera-manager
在master节点执行
yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

在其它子节点执行（子节点只安装客户端就可以）
yum install -y cloudera-manager-agent

3.3parcel文件设置（master节点）
移动parcel文件
cp /root/cdh/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /opt/cloudera/parcel-repo/
cp /root/cdh/manifest.json /opt/cloudera/parcel-repo/

mkdir -p /var/www/html/cdh6
cp /root/cdh/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /var/www/html/cdh6/
cp /root/cdh/manifest.json /var/www/html/cdh6/

执行校验
cd /opt/cloudera/parcel-repo
sha1sum CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel | awk ‘{ print $1 }’ > CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

校验完成后会出现图中的文件

4.启动cloudera-manager
4.1初始化元数据库
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h cdh scm root P@ss123123

如图，代表成功
4.2启动cloudera-scm-server服务
启动命令
systemctl start cloudera-scm-server

查看日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

如图，表示启动成功。
5. cloudera-manager界面配置
登陆http://cdh:7180
默认密码：admin/admin

点击右下角继续

勾选，点击继续

选择免费版，点击继续

集群命名后，点击继续

添加主机（注意端口号选择，我的为6323，默认为22），点击继续

agent端选择自定义存储库，配置如上图（配置的yum源地址）

parcel点击更多选项（如上图），将远程parcel存储库url多余url删除，配置下图圈红url（自己利用httpd创造的地址，请查看3.3节。这一步其实就是为了以防万一，它最终读的parcel是本地路径），点击保存更改，然后点击继续

JDK我们已经安装过，这一步直接点击继续，不需要勾选

提供ssh登录凭据如下图，注意端口号

Agent安装成功后，点击继续

Parcel安装成功后（没截到图），如果继续为灰色，勾选最后一个，点击继续

6.安装CDH组件
选自定义服务

选择HDFS，HIVE，YARN，ZOOKEEPER（HDFS截图截不到），点击继续（组件根据自己需要选择，我因为配置低，选太多内存撑不住）

角色分配可以根据自己需要选择

配置hive元数据库，测试连接

审核更改直接点继续，之后进行首次运行命令

命令执行完成，点击继续

最后，完成

点击完成后，出现下图，就可以使用了