CDH 5.16.2 部署

Cloudera 官网

• 本次我们部署环境为 CDH 5.16.2
• 本次部署使用的是三台 2 核 16 G 机器

1. 部署的方式

1.1 bin在线部署

• 优点：通过 shell 脚本一键部署，比较方便。
• 缺点：需要互联网在线下载安装包，如果网络抖动则不会成功。

1.2 rpm 部署

• 优点：官网推荐的，部署还算比较简单。
• 缺点：部署 Server、Agent rpm 时候需要网络下载依赖包。

1.3 tar 部署

• 真正的离线部署，本次部署就是采用 tar 部署。

2. 部署前置条件

CDH 5X 部署的硬件条件

• 本次部署采用的是阿里云 Centos 7.2 的操作系统

3. 部署初始化集群

• 所有的机器都需要执行，把内网 IP 加入到 hosts 文件里面

[root@bigdata001 ~]# echo "172.22.95.62 bigdata001">> /etc/hosts
[root@bigdata001 ~]# echo "172.22.95.63 bigdata002">> /etc/hosts
[root@bigdata001 ~]# echo "172.22.95.64 bigdata003">> /etc/hosts

• 关闭防火墙

[root@bigdata001 ~]# systemctl stop firewalld
[root@bigdata001 ~]# systemctl disable firewalld

• 关闭 selinux

[root@bigdata001 ~]# vim /etc/selinux/config
SELINUX=disabled

• 时区同步（如果使用阿里云机器，可以不用做）

[root@bigdata001 ~]# timedatectl set-timezone Asia/Shanghai

• 时钟同步（如果使用阿里云机器，可以不用做）
• 所有机器都需要安装

[root@bigdata001 ~]# yum install -y ntp

• 选取第一台机器作为主机器
• 下面只有第一台机器需要执行

[root@bigdata001 ~]# vim /etc/ntp.conf
#time
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#当外部时间不可⽤时，可使⽤本地硬件时间
server 127.127.1.0 iburst local clock
#允许哪些⽹段的机器来同步时间
restrict 172.22.95.0 mask 255.255.255.0 nomodify notrap

• 第一台机器启动 ntp 服务

[root@bigdata001 ~]# systemctl start ntpd
[root@bigdata001 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-02-10 12:15:22 CST; 6min ago
 Main PID: 9479 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─9479 /usr/sbin/ntpd -u ntp:ntp -g

Feb 10 12:15:22 bigdata001 systemd[1]: Starting Network Time Service...
Feb 10 12:15:22 bigdata001 ntpd[9479]: proto: precision = 0.080 usec
Feb 10 12:15:22 bigdata001 ntpd[9479]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Feb 10 12:15:22 bigdata001 systemd[1]: Started Network Time Service.

[root@bigdata001 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l  411   64  100    0.000    0.000   0.000
+120.25.115.20   10.137.53.7      2 u   12   64  177   24.430  1081.30   8.310
 10.143.33.49    .INIT.          16 u    - 1024    0    0.000    0.000   0.000
+100.100.3.1     10.137.55.181    2 u   43   64  377    1.482  1081.81   6.647
+100.100.3.2     10.137.55.181    2 u   43   64  377    1.332  1081.10   7.352
+100.100.3.3     10.137.55.181    2 u   38   64  377    1.470  1080.94   7.428
+203.107.6.88    10.137.38.86     2 u   38   64  377   16.084  1081.27   6.672
 10.143.33.50    .INIT.          16 u    - 1024    0    0.000    0.000   0.000
 10.143.33.51    .INIT.          16 u    - 1024    0    0.000    0.000   0.000
 10.143.0.44     .INIT.          16 u    - 1024    0    0.000    0.000   0.000
 10.143.0.45     .INIT.          16 u    - 1024    0    0.000    0.000   0.000
 10.143.0.46     .INIT.          16 u    - 1024    0    0.000    0.000   0.000
+100.100.5.1     10.137.55.181    2 u   29   64  377    1.545  1081.31   7.467
+100.100.5.2     10.137.55.181    2 u   35   64  377    1.819  1082.29   6.611
*100.100.5.3     10.137.55.181    2 u   30   64  377    1.538  1082.21   6.687

• 其他节点禁用服务

[root@bigdata002 ~]# systemctl stop ntpd
[root@bigdata002 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@bigdata002 ~]# ntpdate bigdata001
10 Feb 12:24:33 ntpdate[9517]: step time server 172.22.95.62 offset -0.740151 sec

• 同步服务增加调度

[root@bigdata002 ~]# crontab -e
00 00 * * * ntpdate bigdata

4. 部署 JDK

• 上传 JDK 的 tar 包

[root@bigdata001 ~]# mkdir software
[root@bigdata001 ~]# cd software/
[root@bigdata001 software]# ll
total 185292
-rw-r--r-- 1 root root 189736377 Oct 20 21:13 jdk-8u151-linux-x64.tar.gz

[root@bigdata001 software]# cp jdk-8u151-linux-x64.tar.gz /tmp/
[root@bigdata001 software]# cd /tmp/
[root@bigdata001 tmp]# scp jdk-8u151-linux-x64.tar.gz bigdata002:/tmp
[root@bigdata001 tmp]# scp jdk-8u151-linux-x64.tar.gz bigdata003:/tmp

• 三台都执行

[root@bigdata001 ~]# cd /tmp/
[root@bigdata001 tmp]# mkdir /usr/java
[root@bigdata001 tmp]# tar -xzvf jdk-8u151-linux-x64.tar.gz -C /usr/java
[root@bigdata001 tmp]# cd /usr/java/
[root@bigdata001 java]# ll
total 4
drwxr-xr-x 8 10 143 4096 Sep  6  2017 jdk1.8.0_151
[root@bigdata001 java]# chown root:root jdk1.8.0_151/
[root@bigdata001 java]# chown -R root:root jdk1.8.0_151/*
[root@bigdata001 java]# chmod 777 jdk1.8.0_151/
[root@bigdata001 java]# chmod -R 777 jdk1.8.0_151/*
[root@bigdata001 java]# ll
total 4
drwxrwxrwx 8 root root 4096 Sep  6  2017 jdk1.8.0_151

[root@bigdata001 java]# cd ~
[root@bigdata001 ~]# vim .bashrc 

#### add env
export JAVA_HOME=/usr/java/jdk1.8.0_151
export PATH=${JAVA_HOME}/bin:${PATH}

[root@bigdata001 ~]# source .bashrc 
[root@bigdata001 ~]# which java
/usr/java/jdk1.8.0_151/bin/java

5. 部署 Mysql

Mysql 下载路径

[root@bigdata001 ~]# cd software/
[root@bigdata001 software]# cp mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz /usr/local
[root@bigdata001 software]# cd /usr/local/
[root@bigdata001 local]# tar -xzvf mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz

• 创建mysql用户，并修改Mysql文件权限

[root@bigdata001 local]# useradd mysql
[root@bigdata001 local]#  id mysql
uid=1000(mysql) gid=1000(mysql) groups=1000(mysql)
[root@bigdata001 local]# chown mysql:mysql mysql-5.7.31-linux-glibc2.12-x86_64
[root@bigdata001 local]# chown -R mysql:mysql mysql-5.7.31-linux-glibc2.12-x86_64/*
[root@bigdata001 local]# ln -s /usr/local/mysql-5.7.28-linux-glibc2.12-x86_64 mysql

• 修改 Mysql 配置文件

[root@bigdata001 local]# cat /dev/null >/etc/my.cnf
[root@bigdata001 local]# vim /etc/my.cnf

[client]
port=3306
socket=/usr/local/mysql/data/mysql.sock
default-character-set=utf8
[mysqld]
port=3306
socket=/usr/local/mysql/data/mysql.sock
character-set-server=utf8 

skip-external-locking
key_buffer_size=256M
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size=4M
query_cache_size=32M
max_allowed_packet=16M
myisam_sort_buffer_size=128M
tmp_table_size=32M

table_open_cache=512
thread_cache_size=8
wait_timeout=86400
interactive_timeout=86400
max_connections=600

# Try number of CPU's*2 for thread_concurrency
# thread_concurrency=32

#isolation level and default engine 
default-storage-engine=INNODB
transaction-isolation=READ-COMMITTED

server-id=1
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data
pid-file=/usr/local/mysql/data/hostname.pid

#open performance schema
log-warnings
sysdate-is-now

binlog_format=MIXED
log_bin_trust_function_creators=1
log-error=/usr/local/mysql/data/hostname.err
log-bin=/usr/local/mysql/arch/mysql-bin
#other logs
#general_log =1
#general_log_file=/usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err

#for replication slave
#log-slave-updates 
#sync_binlog = 1

#for innodb options 
innodb_data_home_dir=/usr/local/mysql/data/
innodb_data_file_path=ibdata1:500M:autoextend
innodb_log_group_home_dir=/usr/local/mysql/arch
innodb_log_files_in_group=2
innodb_log_file_size=200M

innodb_buffer_pool_size=1024M
# innodb_additional_mem_pool_size=50M
innodb_log_buffer_size=16M

innodb_lock_wait_timeout = 100
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 1
innodb_locks_unsafe_for_binlog=1

#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on

#case-sensitive file names and separate tablespace
innodb_file_per_table=1
lower_case_table_names=1

[mysqldump]
quick
max_allowed_packet=16M

[mysql]
no-auto-rehash
default-character-set=utf8

[mysqlhotcopy]
interactive-timeout

[myisamchk]
key_buffer_size=256M
sort_buffer_size=256M
read_buffer=2M
write_buffer=2M

[root@bigdata001 local]# chown mysql:mysql /etc/my.cnf
[root@bigdata001 local]# chmod 640 /etc/my.cnf
[root@bigdata001 local]#  ll /etc/my.cnf
-rw-r----- 1 mysql mysql 2169 Oct 23 14:33 /etc/my.cnf

• 初始化 Mysql

[root@bigdata001 local]# yum -y install libaio
[root@bigdata001 local]# cd mysql
[root@bigdata001 mysql]# ll
total 304
drwxr-xr-x  2 mysql mysql   4096 Jun  2  2020 bin
drwxr-xr-x  2 mysql mysql   4096 Jun  2  2020 docs
drwxr-xr-x  3 mysql mysql   4096 Jun  2  2020 include
drwxr-xr-x  5 mysql mysql   4096 Jun  2  2020 lib
-rw-r--r--  1 mysql mysql 275393 Jun  2  2020 LICENSE
drwxr-xr-x  4 mysql mysql   4096 Jun  2  2020 man
-rw-r--r--  1 mysql mysql    587 Jun  2  2020 README
drwxr-xr-x 28 mysql mysql   4096 Jun  2  2020 share
drwxr-xr-x  2 mysql mysql   4096 Jun  2  2020 support-files
[root@bigdata001 mysql]# su mysql
[mysql@bigdata001 mysql]$ mkdir data tmp arch

[mysql@bigdata001 mysql]$ exit
exit
[root@bigdata001 mysql]# /usr/local/mysql/bin/mysqld --user=mysql --basedir=/usr/local/mysql/ --datadir=/usr/local/mysql/data --initialize

[root@bigdata001 mysql]# cd data/
[root@bigdata001 data]# ll
total 512068
-rw-r----- 1 mysql mysql        56 Feb 10 13:26 auto.cnf
-rw------- 1 mysql mysql      1680 Feb 10 13:26 ca-key.pem
-rw-r--r-- 1 mysql mysql      1112 Feb 10 13:26 ca.pem
-rw-r--r-- 1 mysql mysql      1112 Feb 10 13:26 client-cert.pem
-rw------- 1 mysql mysql      1680 Feb 10 13:26 client-key.pem
-rw-r----- 1 mysql mysql      7066 Feb 10 13:26 hostname.err
-rw-r----- 1 mysql mysql       428 Feb 10 13:26 ib_buffer_pool
-rw-r----- 1 mysql mysql 524288000 Feb 10 13:26 ibdata1
drwxr-x--- 2 mysql mysql      4096 Feb 10 13:26 mysql
drwxr-x--- 2 mysql mysql      4096 Feb 10 13:26 performance_schema
-rw------- 1 mysql mysql      1676 Feb 10 13:26 private_key.pem
-rw-r--r-- 1 mysql mysql       452 Feb 10 13:26 public_key.pem
-rw-r--r-- 1 mysql mysql      1112 Feb 10 13:26 server-cert.pem
-rw------- 1 mysql mysql      1676 Feb 10 13:26 server-key.pem
drwxr-x--- 2 mysql mysql     12288 Feb 10 13:26 sys
[root@bigdata001 data]# cat hostname.err |grep password 
2021-02-10T05:26:10.084026Z 1 [Note] A temporary password is generated for root@localhost: 46uadmwCoA<u

• 部署开机自启动

[root@bigdata001 data]#  cd /usr/local/mysql
[root@bigdata001 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql 
[root@bigdata001 mysql]# chmod +x /etc/rc.d/init.d/mysql
[root@bigdata001 mysql]# chkconfig --del mysql
[root@bigdata001 mysql]# chkconfig --add mysql
[root@bigdata001 mysql]# chkconfig --level 345 mysql on

• 登录 Mysql 并修改密码

[root@bigdata001 mysql]#  su - mysql
Last login: Wed Feb 10 13:28:49 CST 2021 on pts/0
[mysql@bigdata001 ~]$ cd /usr/local/mysql
[mysql@bigdata001 mysql]$ bin/mysqld_safe &
[mysql@bigdata001 mysql]$ bin/mysql -uroot -p'46uadmwCoA<u'

mysql> alter user root@localhost identified by '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'ruozedata' ;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye

• Mysql 创建 CDH 部署需要的库以及用户

mysql> create database cmf default character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges  on cmf.* to cmf@'%' identified by '123456';
ult character set utf8;
grant all privileges  on hive.* to hive@'%' identified by '123456';
flush privileges;Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> create database amon default character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges  on amon.* to amon@'%' identified by '123456';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> create database hive default character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges  on hive.* to hive@'%' identified by '123456';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

• 把 Mysql 依赖的 JAR 放到固定的路径下
• 第一个主机先上传 Mysql jar 并执行

[root@bigdata001 ~]# cd software/
[root@bigdata001 software]# cp mysql-connector-java-5.1.47.jar /tmp/
[root@bigdata001 software]# cd /tmp/
[root@bigdata001 tmp]# scp mysql-connector-java-5.1.47.jar bigdata002:/tmp
[root@bigdata001 tmp]# scp mysql-connector-java-5.1.47.jar bigdata003:/tmp

• 所有的机器都需要执行

[root@bigdata001 ~]# cd /tmp/
[root@bigdata001 tmp]# mkdir -p /usr/share/java/
[root@bigdata001 tmp]# mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar
[root@bigdata001 tmp]# mv mysql-connector-java.jar /usr/share/java/

6. 部署 Cloudera Manager

• 需要下载的文件

Cloudera Manager tar包

Parcel 文件

Parcel sha1

manifest

6.1 部署 Cloudera Manager 前置条件

• 先把文件上传到第一台机器上面，并通过第一台机器把文件传输到几台机器上面。

[root@bigdata001 ~]# cd software/
[root@bigdata001 software]# cp cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz /tmp
[root@bigdata001 software]# cd /tmp/
[root@bigdata001 tmp]# scp cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz bigdata002:/tmp
[root@bigdata001 tmp]# scp cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz bigdata003:/tmp

• 后面操作所有的机器，解压 Cloudera Manager

[root@bigdata001 tmp]# mkdir /opt/cloudera-manager
[root@bigdata001 tmp]# tar -xzvf cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz -C /opt/cloudera-manager/
[root@bigdata001 tmp]# cd /opt/cloudera-manager/
[root@bigdata001 cloudera-manager]# ll
total 8
drwxr-xr-x 4 1106 4001 4096 May 18  2019 cloudera
drwxr-xr-x 9 1106 4001 4096 May 18  2019 cm-5.16.2

[root@bigdata001 cloudera-manager]# cd cm-5.16.2/etc/cloudera-scm-agent/
[root@bigdata001 cloudera-scm-agent]# ll
total 12
-rw-r--r-- 1 1106 4001 8905 May 18  2019 config.ini
[root@bigdata001 cloudera-scm-agent]# vim config.ini

server_host=bigdata001

• 由于我选举第一台机器作为 CM Server，所以下面需要操作第一台机器，指定 CM 的 Mysql 路径

[root@bigdata001 cloudera-scm-agent]# cd ../cloudera-scm-server/
[root@bigdata001 cloudera-scm-server]# ll
total 8
-rw------- 1 1106 4001  714 May 18  2019 db.properties
-rw-r--r-- 1 1106 4001 2229 May 18  2019 log4j.properties
[root@bigdata001 cloudera-scm-server]# vim db.properties 

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=bigdata001
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=123456
com.cloudera.cmf.db.setupType=EXTERNAL

• 所有节点创建⽤户

[root@bigdata001 cloudera-scm-server]# useradd --system --home=/opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
[root@bigdata001 cloudera-scm-server]# chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

• 选取一台机器作为 Parcel 源，并部署，我选取的是第一台作为 Parcel 源

[root@bigdata001 cloudera-scm-server]# mkdir -p /opt/cloudera/parcel-repo
[root@bigdata001 cloudera-scm-server]# cd ~/software/
[root@bigdata001 software]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel /opt/cloudera/parcel-repo/
[root@bigdata001 software]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1 /opt/cloudera/parcel-repo
[root@bigdata001 software]# mv manifest.json /opt/cloudera/parcel-repo/
[root@bigdata001 software]# cd /opt/cloudera/parcel-repo
[root@bigdata001 parcel-repo]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha
[root@bigdata001 parcel-repo]# ll
total 2082872
-rw-r--r-- 1 root root 2132782197 Oct 20 21:16 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel
-rw-r--r-- 1 root root         41 Feb  9 21:37 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha
-rw-r--r-- 1 root root      66804 Oct 20 21:12 manifest.json

6.2 部署 Cloudera Manager

• 创建所有节点的软件安装目录 用户和用户组的权限
• 所有机器都需要执行

[root@bigdata001 cloudera]# mkdir -p /opt/cloudera/parcels
[root@bigdata001 cloudera]# chown -R cloudera-scm:cloudera-scm /opt/cloudera

• 启动 Cloudera Manager Server

[root@bigdata001 cloudera]# /opt/cloudera-manager/cm-5.16.2/etc/init.d/cloudera-scm-server start

• 查看 Log

[root@bigdata001 cloudera-scm-server]# cd /opt/cloudera-manager/cm-5.16.2/log/cloudera-scm-server/
[root@bigdata001 cloudera-scm-server]# tail -200f cloudera-scm-server.log

• 启动所有的 Cloudera Manager Agent

[root@bigdata001 cloudera]# /opt/cloudera-manager/cm-5.16.2/etc/init.d/cloudera-scm-agent start

• 打开阿里云的防火墙
• 打开 WEB 页面

http://115.29.206.70:7180/cmf/login
用户：admin
密码：admin

• 所有机器执行下面命令

[root@bigdata001 cloudera-scm-server]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@bigdata001 cloudera-scm-server]# echo never > /sys/kernel/mm/transparent_hugepage/enabled

• 刷新下检查