CDH6.3.1集群企业真正离线部署

最新推荐文章于 2022-06-27 15:42:21 发布

夙梦-小白

最新推荐文章于 2022-06-27 15:42:21 发布

阅读量810

点赞数 1

文章标签：大数据

本文链接：https://blog.csdn.net/qq_36417221/article/details/105705316

版权

⼀.准备⼯作

1.离线部署主要分为三块:
a.MySQL离线部署
b.CM离线部署
c.Parcel⽂件离线源部署

2.规划:

节点	MySQL部署组件	Parcel⽂件离线源	CM服务进程	⼤数据组件
master01	MySQL	Parcel	Activity Monitor	NN RM DN NM
node01			Alert Publisher Event Server	DN NM
node02			Host Monitor Service Monitor	DN NM

3.下载源:
⽀持的版本⭐: requirements supported versions

CM
cm6.3.1-redhat7.tar.gz
Parcel
CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel
CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1
manifest.json
JDK
https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html
下载jdk-8u181-linux-x64.tar.gz
MySQL

https://dev.mysql.com/downloads/mysql/5.7.html#downloads
下载mysql-5.7.26-el7-x86_64.tar.gz
MySQL jdbc jar

mysql-connector-java-5.1.47.jar
下载完成后要重命名去掉版本号:
mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar

⼆.集群节点初始化

CentOS7.2操作系统，2核16G最低配置(8G 勉强也可以)

当前笔记本或台式机配置hosts⽂件

公⽹地址 机器名称: 
47.74.130.251 master01 
161.117.1.169 node01 
47.88.170.121 node02

设置所有节点的hosts⽂件

私有地址(内⽹地址) 机器名称:
echo "172.21.230.214 master01">> /etc/hosts
echo "172.21.230.216 node01">> /etc/hosts
echo "172.21.230.215 node02">> /etc/hosts

关闭所有节点的防⽕墙及清空规则

systemctl stop firewalld 
systemctl disable firewalld
iptables -F

关闭所有节点的selinux

vi /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled 
设置后需要重启才能⽣效

设置所有节点的时区⼀致及时钟同步

6.1.时区
[root@master01 ~]# date
Wed Oct 23 13:51:31 CST 2019
[root@master01 ~]# timedatectl
 Local time: Wed 2019-10-23 13:51:49 CST
 Universal time: Wed 2019-10-23 05:51:49 UTC
 RTC time: Wed 2019-10-23 13:51:49
 Time zone: Asia/Shanghai (CST, +0800)
 NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
 DST active: n/a
#查看命令帮助，学习⾄关重要，⽆需百度，太
[root@master01 ~]# timedatectl --help
timedatectl [OPTIONS...] COMMAND ...
Query or change system time and date settings.
 -h --help Show this help message
 --version Show package version
 --no-pager Do not pipe output into a pager
 --no-ask-password Do not prompt for password
 -H --host=[USER@]HOST Operate on remote host
 -M --machine=CONTAINER Operate on local container
 --adjust-system-clock Adjust system clock when changing local RTC mode
Commands:
 status Show current time settings
 set-time TIME Set system time
 set-timezone ZONE Set system time zone
 list-timezones Show known time zones
 set-local-rtc BOOL Control whether RTC is in local time
 set-ntp BOOL Control whether NTP is enabled
#查看哪些时区
[root@master01 ~]# timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako
#所有节点设置亚洲上海时区
[root@master01 ~]# timedatectl set-timezone Asia/Shanghai
[root@node01 ~]# timedatectl set-timezone Asia/Shanghai
[root@node02 ~]# timedatectl set-timezone Asia/Shanghai
6.2.时间
#所有节点安装ntp
[root@master01 ~]# yum install -y ntp
#选取ruozedata001为ntp的主节点
[root@master01 ~]# vi /etc/ntp.conf 
#time
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#当外部时间不可⽤时，可使⽤本地硬件时间
server 127.127.1.0 iburst local clock 
#允许哪些⽹段的机器来同步时间
restrict 172.21.230.0 mask 255.255.255.0 nomodify notrap
#开启ntpd及查看状态
[root@master01 ~]# systemctl start ntpd
[root@master01 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
 Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: d
isabled)
 Active: active (running) since Sat 2019-05-11 10:15:00 CST; 11min ago
 Main PID: 18518 (ntpd)
 CGroup: /system.slice/ntpd.service
 !"18518 /usr/sbin/ntpd -u ntp:ntp -g
May 11 10:15:00 master01 systemd[1]: Starting Network Time Service...
May 11 10:15:00 master01 ntpd[18518]: proto: precision = 0.088 usec
May 11 10:15:00 master01 ntpd[18518]: 0.0.0.0 c01d 0d kern kernel time sync en
abled
May 11 10:15:00 master01 systemd[1]: Started Network Time Service.
#验证
[root@master01 ~]# ntpq -p
 remote refid st t when poll reach delay offset jitter
 LOCAL(0) .LOCL. 10 l 726 64 0 0.000 0.000 0.000
#其他从节点停⽌禁⽤ntpd服务
[root@node01 ~]# systemctl stop ntpd
[root@node01 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@node01 ~]# /usr/sbin/ntpdate master01
11 May 10:29:22 ntpdate[9370]: adjust time server 172.19.7.96 offset 0.000867 sec
#每天凌晨同步master01节点时间
[root@node01 ~]# crontab -e
00 00 * * * /usr/sbin/ntpdate master01 
[root@node02 ~]# systemctl stop ntpd
[root@hadoop004 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@hadoop005 ~]# /usr/sbin/ntpdate master01
11 May 10:29:22 ntpdate[9370]: adjust time server 172.19.7.96 offset 0.000867 sec
#每天凌晨同步ruozedata001节点时间
[root@node02 ~]# crontab -e
00 00 * * * /usr/sbin/ntpdate master01

每个节点部署JDK

mkdir /usr/java
tar -xzvf jdk-8u181-linux-x64.gz -C /usr/java/
#切记必须修正所属⽤户及⽤户组
chown -R root:root /usr/java/jdk1.8.0_181
echo "export JAVA_HOME=/usr/java/jdk1.8.0_181" >> /etc/profile
echo "export PATH=/usr/java/jdk1.8.0_181/bin:${PATH}" >> /etc/profile
source /etc/profile
which java

master01节点离线部署MySQL5.7(假如觉得困难哟，就⾃⾏选择RPM部署)

⽂档链接:https://github.com/Hackeruncle/MySQL
视频链接:https://pan.baidu.com/s/1jdM8WeIg8syU0evL1-tDOQ 密码:whic

部署完成后，登录:

master01:mysqladmin:/usr/local/mysql/data:>mysql -uroot -pruozedata
mysql>

创建CDH的元数据库和⽤户、amon服务的数据库及⽤户

create database cmf DEFAULT CHARACTER SET utf8;
create database amon DEFAULT CHARACTER SET utf8;

master01节点部署mysql jdbc jar

mkdir -p /usr/share/java/
重命名不能带版本号
cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar

三.CDH部署

离线部署cm server及agent

1.1.所有节点创建⽬录及解压
mkdir /opt/cloudera-manager
tar -xzvf cm6.3.1-redhat7.tar.gz -C /opt/cloudera-manager/
1.2.选择ruozedata001为cm server，不下载依赖包直接部署
cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
1.3.所有节点(包含ruozedata001)为cm agent，不下载依赖包直接部署
cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
1.4.所有节点修改agent的配置，指向server的节点ruozedata001
sed -i "s/server_host=localhost/server_host=ruozedata001/g" /etc/cloudera-scm-agen
t/config.ini
1.5.主节点修改server的配置:
vi /etc/cloudera-scm-server/db.properties 
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=master01
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=root
com.cloudera.cmf.db.password=123456
com.cloudera.cmf.db.setupType=EXTERNAL

master01节点部署离线parcel源

2.1.安装httpd服务
yum install -y httpd
2.2.部署离线parcel源
$ mkdir -p /var/www/html/cdh6_parcel
[root@master01 CDH6.3.1]# cp CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel /var/
www/html/cdh6_parcel/
[root@master01 CDH6.3.1]# mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /
var/www/html/cdh6_parcel/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha
[root@master01 CDH6.3.1]# mv manifest.json /var/www/html/cdh6_parcel/
$ ll
total 3081664
-rw-r--r-- 1 root root 2083878000 Oct 23 13:44 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7
.parcel
-rw-r--r-- 1 root root 40 Oct 23 13:44 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7
.parcel.sha1
-rw-r--r-- 1 root root 1411444147 Oct 23 13:43 cm6.3.1-redhat7.tar.gz
-rw-r--r-- 1 root root 173271626 Oct 23 13:43 jdk-8u45-linux-x64.gz
-rw-r--r-- 1 root root 33887 Oct 23 13:43 manifest.json
-rw-r--r-- 1 root root 548193637 Oct 23 13:42 mysql-5.7.11-linux-glibc2.5-x86_64.
tar.gz
-rw-r--r-- 1 root root 1007502 Sep 1 12:31 mysql-connector-java-5.1.47.jar
$ mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel /var/www/html/cdh6_parcel/
$ mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /var/www/html/cdh6_parcel/CDH
-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha
$ mv manifest.json /var/www/html/cdh6_parcel/
sha1名称⽂件时，重命名去掉1，不然在部署过程CM认为如上⽂件下载未完整，会持续下载
2.3.启动httpd，window查看
systemctl start httpd

window/mac: http://ruozedata001/cdh6_parcel/ 查看是否可以访问

在这里插入图片描述
master01节点启动Server

3.1.启动server
systemctl start cloudera-scm-server
查看⽇志:
cd /var/log/cloudera-scm-server/
$ ll
total 20
-rw-r----- 1 cloudera-scm cloudera-scm 18265 Oct 23 15:37 cloudera-scm-server.log
-rw-r----- 1 cloudera-scm cloudera-scm 0 Oct 23 15:37 cmf-server-nio.log
-rw-r----- 1 cloudera-scm cloudera-scm 0 Oct 23 15:37 cmf-server-perf.log
$ tail -F cloudera-scm-server.log
有错误就根据错误解决，没有错误，等待1min，出现7180端⼝，表明是成功的。

在这里插入图片描述

等待1min，打开 http://master01:7180 账号密码:admin/admin
假如打不开，去看server的log，根据错误仔细排查错误

所有节点启动Agent

systemctl start cloudera-scm-agent

接下来，全部Web界⾯操作
http://master01:7180/
账号密码:admin/admin
在这里插入图片描述
欢迎使⽤Cloudera Manager–最终⽤户许可条款与条件。勾选

在这里插入图片描述
欢迎使⽤Cloudera Manager–您想要部署哪个版本？选择Cloudera Express免费版本

欢迎:在CM⾥添加⼀个集群

夙梦-小白

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
CDH6.3.1集群企业真正离线部署

⼀.准备⼯作1.离线部署主要分为三块:a.MySQL离线部署b.CM离线部署c.Parcel⽂件离线源部署2.规划:节点MySQL部署组件Parcel⽂件离线源CM服务进程⼤数据组件master01MySQLParcelActivity MonitorNN RM DN NMnode01Alert Publisher Event Serve...
复制链接

扫一扫