离线搭建cdh5.12.2集群

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/lzxlfly/article/details/82056992

一、安装环境准备

1、服务分布一览表

hostname

Server Agent MySQL

NN

DN

RM NM

JN

ZK

DFSZKFC

node01

1    

1

 

1  

 

 

1

node02

  1  

 

  1

1

1

 

node03

  1  

 

1

  1

1

1

 

 node04

  1  

 

1

  1

1

 1

 

node05

  1 1

1(backup)

 

1(backup)  

 

 

1(backup)

2、主机名及host配置

#把HOSTNAME改为node01,其它节点分别改为node02、node03、node04
[root@node01 cm-5.12.2]# vim /etc/sysconfig/network
NETWORKING=yes
#HOSTNAME=localhost.localdomain
HOSTNAME=node01

#配置host
[root@node01 cm-5.12.2]# vim /etc/hosts
127.0.0.1    localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.172.74 node05
192.168.172.74 node04
192.168.172.73 node03
192.168.172.72 node02
192.168.172.71 node01

3、ssh免密钥登录

见之前的一片文章https://blog.csdn.net/lzxlfly/article/details/77916842

4、关闭防火墙

#临时关闭,立即生效,重启失效
service iptables stop
#永久关闭、重启生效
chkconfig iptables off

5、关闭selinux安全服务

#立即生效,重启失效
setenforce 0
#把SELINUX改为disabled,重启生效
[root@node01 cm-5.12.2]# vim /etc/selinux/config 
#SELINUX=enforcing
SELINUX=disabled

SELINUXTYPE=targeted

6、安装jdk,配置环境变量

#安装jdk,官方建议rpm安装
[root@node01 log]# rpm -ivh jdk-8u181-linux-x64.rpm

#查看安装,latest指向jdk1.8.0_181-amd64
[root@node01 log]# ls -l /usr/java/
total 4
lrwxrwxrwx. 1 root root   16 Aug 25 13:01 default -> /usr/java/latest
drwxr-xr-x. 8 root root 4096 Aug 25 13:01 jdk1.8.0_181-amd64
lrwxrwxrwx. 1 root root   28 Aug 25 13:01 latest -> /usr/java/jdk1.8.0_181-amd64

# 配置环境变量
[root@node01 log]# vim /etc/profile

JAVA_HOME=/usr/java/latest
JRE_HOME=$JAVA_HOME/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/jt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

#配置生效
[root@node01 log]# source /etc/profile

7、ntp时间同步

没有安装ntp先 安装ntp服务

#所有节点安装NTP:
yum install ntp -y

#配置开机启动:
chkconfig ntpd on

#检查是否设置成功:
chkconfig --list ntpd (2-5为on状态则成功)

#设置ntp时间同步:
ntpdate -u ntp.sjtu.edu.cn
(210.72.145.8-国家授时中心服务器IP地址)

#启动ntp服务
service ntpd start

8、安装配置mysql

    见上一篇文章 https://blog.csdn.net/lzxlfly/article/details/81990234

二、安装Cloudera Manager

1、下载 Cloudera Manager

去 http://archive.cloudera.com/cdh5/parcels/5.12.2/下载 cloudera-manager-el6-cm5.12.2_x86_64.tar.gz 

分发到其它节点,并解压到/opt/cloudera-manager 

tar -zxvf cloudera-manager-el6-cm5.12.2_x86_64.tar.gz  -C /opt/cloudera-manager

2、创建用户cloudera-scm

#每个节点都要添加
useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3、配置cloudera-scm-agent

#配置node01为server,其它节点亦是如此
[root@node01 log]# vim /opt/cloudera-manager/cm-5.12.2/etc/cloudera-scm-agent/config.ini 

[General]
# Hostname of the CM server.
server_host=node01

4、复制连接mysql驱动

复制mysql jar文件到目录 /usr/share/java/ 下,重命名为 mysql-connector-java.jar(只在安装mysql的节点操作即可)

5、初始化CM-Server数据库 

  进入到  /opt/cloudera-manager/cm-5.12.2/share/cmf/schema,执行脚本,初始化数据库

#数据库类型mysql、数据库名字cm、数据库所在节点node05、用户名cloudera、密码12345678、连接scm-host node01
[root@node01 schema]# ./scm_prepare_database.sh mysql cm -h node05 -ucloudera -p12345678 --scm-host node01 scm scm scm

#看到以下信息才算成功
JAVA_HOME=/usr/java/latest
Verifying that we can write to /opt/cloudera-manager/cm-5.12.2/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cloudera-manager/cm-5.12.2/etc/cloudera-scm-server
Executing:  /usr/java/latest/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.12.2/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager/cm-5.12.2/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

三、配置离线pacels源

1、下载parcels

   去 http://archive.cloudera.com/cdh5/parcels/5.12.2/ 下载到/opt/soft/下

  CH-5.12.2-1.cdh5.12.2.p0.4-el6.parcel、CDH-5.12.2-1.cdh5.12.2.p0.4-el6.parcel.sha1、manifest.json

 CDH的parcel不包过spark2和kafka的,spark2和kafka的可以去这里下载

http://archive.cloudera.com/kafka/parcels/

http://archive.cloudera.com/spark2/parcels/

2、在server制作CDH本地源

#在server节点创建parcel-repo,用于分发到agent
mkdir -p /opt/cloudera/parcel-repo

#把下载的三个文件复制到/opt/cloudera/parcel-repo下
cp /opt/soft/* /opt/cloudera/parcel-repo

#在agent节点创建parcels,接收server的分发
mkdir -p /opt/cloudera/parcels

3、启动server和agent

#在node01上启动server
/opt/cloudera-manager/cm-5.12.2/etc/init.d/cloudera-scm-server start
#在node02、node03、node04、node05上启动agent
/opt/cloudera-manager/cm-5.12.2/etc/init.d/cloudera-scm-agent start

4、登录管理台

http://node01:7180/cmf/login

5、选择免费版,继续配置,接下来简单不再描述

 

阅读更多
换一批

没有更多推荐了,返回首页