CDH6.3.2及hadoop生态圈工具安装指南

一、前期准备

1.1 软硬件要求

服务器3台，系统要求Centos7

1.2 服务器部署规划

内存

磁盘

CPU

cm

cdh

Mysql

Hive

Impala

Kudu

Flink

Spark

Kafka

node-01

server

namenode

√

√

√

√

√

√

√

node-02

agent

datanode

√

√

√

√

√

√

√

node-03

agent

datanode

√

√

√

√

√

√

√

1.3 端口规划

服务名称	端口号
cloudera manager	7180
resourcemanager	8088
hue	8888
namennode	50070
spark	19888
kafka	9092
mysql	3306
hive	10000
hbase	60030
zookeeper	2181

4.数据存储格式及压缩规划

orc列式存储+snappy压缩

5.关闭tuned

systemctl stop tuned
systemctl disable tuned
systemctl status tuned

二、集群环境配置

1.修改主机名

#在不同主机执行不同命令
hostnamectl set-hostname node-01
hostnamectl set-hostname node-02
hostnamectl set-hostname node-03

2.关闭防火墙

方法一：
service iptables stop   #关闭防火墙 
chkconfig iptables off  #禁止开机启动
方法二：
systemctl disable firewalld
systemctl status firewalld

3.关闭selinux

vim /etc/selinux/config
#找到这个改为关闭
selinux=disabled

4.更改文件句柄

vim /etc/security/limits.conf
添加
*		soft		nofile		1769445
*		hard		nofile		1029345
*		soft		nproc		unlimited

5. 关闭透明大页面（所有节点）

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

vim /etc/rc.d/rc.local
添加如下内容
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
	echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
	echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

 6.设置swappiness（所有节点）

sysctl vm.swappiness=1
echo 1 > /proc/sys/vm/swappiness

vim /etc/sysctl.conf
vm.swappiness=1

7.配置host映射（对于本机也不要写127.0.0.1）

vim /etc/hosts
10.0.8.9    node-01
10.0.8.14   node-02
10.0.8.12   node-03

8.配置免密登录

在每台服务器下执行以下操作：

ssh-keygen -t rsa

一路回车即可。可以发现 ssh目录下的两枚钥匙，分别是id_rsa、id_rsa.pub，其中第一个为密钥，第二个为公钥。

拷贝node-01的公钥到node-02、node-03:

ssh-copy-id -i .ssh/id_rsa.pub root@fluxes-node-02
ssh-copy-id -i .ssh/id_rsa.pub root@fluxes-node-03

拷贝node-02的公钥到node-01、node-03:

ssh-copy-id -i .ssh/id_rsa.pub root@fluxes-node-01
ssh-copy-id -i .ssh/id_rsa.pub root@fluxes-node-03

 拷贝node-03的公钥到node-01、node-02:

ssh-copy-id -i .ssh/id_rsa.pub root@fluxes-node-01
ssh-copy-id -i .ssh/id_rsa.pub root@fluxes-node-02

9.安装JDK

略

10.安装MySQL

参考：CentOS 7.6下安装MySQL 5.7_JAVA飘香-CSDN博客

11.初始化MySQL权限

mysql -uroot -p

# 新建scm用户
CREATE USER 'cdh'@'%' IDENTIFIED BY '123456'; 
grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;
grant all privileges on *.* to 'cdh'@'%' identified by '123456' with grant option;
flush privileges;

10.创建CM需要的库

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

11.安装MySQL JDBC驱动（yum安装的不可用）

# 下载
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.49.tar.gz
# 解压
tar -zxf mysql-connector-java-5.1.49.tar.gz
# 创建目录
mkdir /usr/share/java/
# 放到指定目录
（一定要改名为mysql-connector-java.jar）
sudo cp mysql-connector-java-5.1.49/mysql-connector-java-5.1.49-bin.jar /usr/share/java/mysql-connector-java.jar

三、Cloudera Manager 安装部署

1.下载CM的压缩包并上传到集群中的某台主机(hadoop102.example.com)，并解压

tar -zxvf cm6.3.1-redhat7.tar.gz

2.进入解压之后的路径，执行以下命令，以发布该yum仓库

cd cm6.3.1
nohup python -m SimpleHTTPServer 8900 &

可用浏览器访问
http://node-01:8900 地址，如响应如下页面，则表示yum仓库发布成功

3.在所有主机上创建yum仓库repo文件

vim /etc/yum.repos.d/cloudera-manager.repo

[cloudera-manager]
name=Cloudera Manager 6.3.1
baseurl=http://node-01:8900/
gpgkey=http://node-01:8900/RPM-GPG-KEY-cloudera
gpgcheck=1
enabled=1
autorefresh=0
type=rpm-md

4.在主节点执行：

yum -y install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

5.在从节点执行：

sudo yum -y install cloudera-manager-daemons cloudera-manager-agent

6.在所有节点修改配置文件/etc/cloudera-scm-agent/config.ini ,修改其server_host参数

[General]
# Hostname of the CM server.
server_host=node-01

四、初始化数据库

4.1 建库

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

4.2 初始化

/opt/cloudera/cm/schema/scm_prepare_database.sh --host node-01 --scm-host node-01  mysql scm root 123456

 五 启动CM

1.在主节点（node-01）执行以下命令，启动Server 和 Agent

systemctl start cloudera-scm-server cloudera-scm-agent

2.在其余节点执行以下命令，启动Agent

systemctl start cloudera-scm-agent

 3.本地访问：

curl localhost:7180
<head><meta http-equiv="refresh" content="0;url=/cmf/"></head>

4.浏览器访问：

 

六 部署CDH

Cloudera Manager d节点的/opt/cloudera/p

1.上传CDH parcel到Cloudera Manager Server

节点的/opt/cloudera/parcel-repo路径下，上传CDH所需parcel

2.为parcel文件生成SHA1校验文件

sha1sum CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel | awk '{ print $1 }' > CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

3.更改parcel文件所有者

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*

4.重启Cloudera Manager，令其识别到本地库

sudo systemctl restart cloudera-scm-server

5.登录Cloudera Manager，初始用户名和密码均为admin 

按照导航，选择所有主机，选择安装方式：

成功安装：

 

6.检查网络和主机 

 确保检查结果完全正确，如有问题点击检查详情，按照引导进行修复，修复完毕后，方可点击“继续”：