Centos7.8+CDH6.2.1从零开始安装

前言，为什么我要开始重新装一遍呢，因为，我攒了个台式机，还没有装CDH，学习需要用，顺道记录一下安装记录，当做以后的笔记，以后查漏补缺。

一、CDH是什么？

CDH是Cloudera的100%开源平台发行版，包括Apache Hadoop，专门为满足企业需求而构建。CDH提供了企业使用所需的一切。通过将Hadoop与十几个其他关键的开源项目集成，Cloudera创建了一个功能先进的系统，帮助您执行端到端的大数据工作流。

1.1 CDH的重点项目有哪些？

相信大家对这些大数据组件应该都是很熟悉了，那么我们接下来步入正题，进行安装我们的CDH。

二、提前准备

2.1 安装前需要准备环境以及文件

2.1.1 Linux镜像

这里我使用的linux镜像是CentOS-7-x86_64-DVD-2003.iso，如果大家没有，可以选择以下方式进行下载，或者选用自己的Linux镜像。

1. 进入阿里云站点：Linux镜像下载
   

2.1.2 JDK1.8+ Linux版本tar包

百度云连接：https://pan.baidu.com/s/1Re07mLl0t3X17n74GVyUSQ
提取码：g7ve

2.1.3 CDH安装包

链接：https://pan.baidu.com/s/1XVBJpObR4O3b0FVScyaOiA
提取码：or98

2.1.4 一台能够上网的计算机

这里多一句嘴，我以前用的是win7，我感觉win7确实是很好用，但是后来因为一些原因，改换win10，我尝试过很多的win10版本，发现最好的是Windows 10 Enterprise LTSC 2019 (x64) - DVD (Chinese-Simplified)
第一他会更新，但是并不会像别的win10那样进行强制更新，这个更新只会在一个月一次，或者不更新，而且是特别纯净，无任何自带软件。

1. 进入我告诉你 Windows 10 Enterprise LTSC 2019

2.1.5 VMware

https://blog.csdn.net/weixin_43704599/article/details/106441149

三、环境准备

3.1 Linux机器配置

3.1.1 机器规划

hostname	IP	CPU	内存	硬盘
cdh01	192.168.1.101	2c2核	16G	100G
cdh02	192.168.1.102	1c2核	8G	100G
cdh03	192.168.1.103	1c2核	8G	100G
cdh04	192.168.1.104	1c1核	4G	100G

因为我的台式机配置比较高，可根据自己的实际情况自定义搭配

3.1.2 Hosts配置

[root@cdh01 ~]# vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.101 cdh01
192.168.1.102 cdh02
192.168.1.103 cdh03
192.168.1.104 cdh04

3.1.3 同步脚本

所有节点安装rsync

[root@cdh01 .ssh]# cd /root/
[root@cdh01 ~]# mkdir bin
[root@cdh01 ~]# cd bin/
[root@cdh01 bin]# touch xsync
[root@cdh01 bin]# vi xsync 
#!/bin/bash
#1 获取输入参数个数，如果没有参数，直接退出
pcount=$#
if((pcount==0)); then
echo no args;
exit;
fi

#2 获取文件名称
p1=$1
fname=`basename $p1`
echo fname=$fname

#3 获取上级目录到绝对路径
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$pdir

#4 获取当前用户名称
user=`whoami`

#5 循环
for host in 01 02 03 04
do
        echo ------------------- cdh$host --------------
        rsync -av $pdir/$fname $user@cdh$host:$pdir
done
[root@cdh01 bin]# chmod u+x xsync 

3.1.4 配置免密

[root@cdh01 ~]#    ssh-keygen -t rsa  
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:hWgtOj7f96xCfIvZtFvGX6mvga455kacgD2ePajP3qQ root@cdh01
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|       o .       |
|      * o .      |
|     + = .       |
|    o ..S .      |
|   . . +o*o..   .|
|    o ...B.++. ..|
|     +..B+B+ .o. |
|      +E=B==oo+. |
+----[SHA256]-----+
[root@cdh01 .ssh]# ssh-copy-id cdh01
[root@cdh01 ~]# xsync .ssh/

3.1.5 修改yum源

cd /etc/yum.repos.d/
yum install -y wget
mv CentOS-Base.repo CentOS-Base.repo.bak
wget http://mirrors.aliyun.com/repo/Centos-7.repo
mv Centos-7.repo CentOS-Base.repo
yum clean all
yum makecache

3.1.6 关闭SELINUX（所有节点）

[root@cdh01 ~]# vi /etc/selinux/config
将这一行修改为 SELINUX=disabled
[root@cdh01 ~]# xsync /etc/selinux/config

3.1.7 配置集群时间同步

1. 安装ntp（所有节点）

[root@cdh01 ~]# yum install ntp -y

2. 在主节点配置时间同步

[root@cdh01 ~]# vi /etc/ntp.conf

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server 127.127.1.0
fudge 127.127.1.0 stratum 10

2. 修改系统时间与硬件时间同步

vi /etc/sysconfig/ntpd
# Command line options for ntpd
OPTIONS="-g"
SYNC_HWCLOCK=yes

3. 配置其他节点时间同步

crontab -e
*/1 * * * * /usr/sbin/ntpdate cdh01

4. 开启自启（所有节点）

systemctl start  ntpd
systemctl enable  ntpd

3.1.8 优化虚拟内存（所有节点）

[root@cdh01 ~]# echo 'vm.swappiness = 0' > /etc/sysctl.d/swappiness.conf
[root@cdh01 ~]# sysctl -p

3.1.9 禁用大叶内存（所有节点）

[root@cdh01 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@cdh01 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@cdh01 ~]# echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >>  /etc/rc.local
[root@cdh01 ~]# echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>  /etc/rc.local
[root@cdh01 ~]# chmod u+x  /etc/rc.local

3.1.11 重启所有节点(所有节点关闭防火墙)

[root@cdh01 ~]# systemctl stop firewalld.service
[root@cdh01 ~]# systemctl disable firewalld.service

3.1.11 重启所有节点

reboot

3.2 MySQL部署

3.2.1 MySQL安装

1. 下载MySQL仓库和安装MySQL

wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
yum -y install mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql-community-server

2. 启动MySQL

systemctl start  mysqld.service
systemctl status mysqld.service

3. 修改密码

grep "password" /var/log/mysqld.log

mysql -uroot -p 
然后输入我们之前获取到的密码g&0WNnwv8V:p
ALTER USER 'root'@'localhost' IDENTIFIED BY 'Admin@123';

4. 修改MySQL默认编码

vi /etc/my.cnf
character_set_server=utf8
init_connect='SET NAMES utf8'

3.2.2 配置root用户远程访问

mysql -uroot -pAdmin@123
grant all privileges on *.* to 'root' @'%' identified by 'Admin@123';
flush privileges;

3.2.3 创建CDH需要用到的库

mysql> CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

mysql> CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

mysql> CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

mysql> CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

mysql> CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

mysql> CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

mysql> quit

Mysql驱动

[root@cdh01 data]# mkdir -p /usr/share/java
[root@cdh01 data]# mv mysql-connector-java-5.1.27-bin.jar /usr/share/java/mysql-connector-java.jar
[root@cdh01 data]# xsync /usr/share/java/

此时建议大家创建快照哦

3.3 Java安装

[root@cdh01 data]# mkdir -p /usr/java/
[root@cdh01 data]# tar -zxvf jdk-8u144-linux-x64.tar.gz -C /usr/java/
[root@cdh01 data]# vi /etc/profile
#JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_144
export PATH=$PATH:$JAVA_HOME/bin
[root@cdh01 data]# source /etc/profile
[root@cdh01 data]# xsync /usr/share/java/ 
[root@cdh02 data]# source /etc/profile
[root@cdh03 data]# source /etc/profile
[root@cdh04 data]# source /etc/profile

四、CDH安装

4.1 jar包上传

1. unzip 安装

yum install unzip -y

2. cm准备

[root@cdh01 data]# unzip cm6.2.1.zip
[root@cdh01 data]# xsync cm6.2.1/

4. parcels准备

[root@cdh01 data]# unzip parcels.zip
[root@cdh01 data]# cd parcels
[root@cdh01 parcels]# mkdir -p /opt/cloudera/parcel-repo/
[root@cdh01 parcels]# cp ./* /opt/cloudera/parcel-repo/
[root@cdh01 parcels]# cd /opt/cloudera/parcel-repo/
[root@cdh01 parcel-repo]# mv CDH-6.2.1-1.cdh6.2.1.p0.1580995-el7.parcel.sha1 CDH-6.2.1-1.cdh6.2.1.p0.1580995-el7.parcel.sha

4.2 CM安装

主节点

[root@cdh01 cloudera]# cd /opt/data/cm6.2.1/RPMS/x86_64
[root@cdh01 x86_64]# yum install cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm -y
[root@cdh01 x86_64]# yum install cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm -y
[root@cdh01 x86_64]# yum install cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm -y

其他节点

[root@cdh02 cloudera]# cd /opt/data/cm6.2.1/RPMS/x86_64
[root@cdh02 x86_64]# yum install cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm -y
[root@cdh02 x86_64]# yum install cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm -y

4.3 配置Server节点（所有节点）

[root@cdh01 x86_64]vim /etc/cloudera-scm-agent/config.ini
# Hostname of the CM server.
server_host=cdh01

4.3 配置数据库

[root@cdh01 x86_64]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root Admin@123

4.3 启动CM

[root@cdh01 java]# systemctl start cloudera-scm-agent
[root@cdh02 java]# systemctl start cloudera-scm-agent
[root@cdh03 java]# systemctl start cloudera-scm-agent
[root@cdh04 java]# systemctl start cloudera-scm-agent
[root@cdh01 java]# systemctl start cloudera-scm-server

此时观察我们的日志tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log，看到Started Jetty server.就好了

五、组件安装

浏览器输入http://cdh01:7180/cmf/login
账号：admin
密码：admin

5.1集群配置