CDH 5.8 离线安装

本文介绍在CentOS 6环境下离线安装CDH 5.8(两个节点)。主要步骤如下，

1 安装前配置

(1)配置host name，同步集群每个节点

[root@cent-1 centos]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.16    cent-1.novalocal        cent-1
192.168.0.47    cent-2.novalocal        cent-2
[root@cent-1 centos]# scp /etc/hosts cent-2:/etc/

(2)配置集群节点间免密码，在此不作详细说明

(3)关闭防火墙，每个节点相同

//centos6

service iptables stop
chkconfig iptables off

//centos7

systemctl stop firewalld
systemctl disable firewalld

(4)关闭selinux，每个节点相同

[root@cent-1 ~]# setenforce 0
[root@cent-1 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
[root@cent-1 ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

(5) 检查系统时区，确保为CST时区
如果集群时区非CST，执行以下命令设置为CST时区

cp /usr/share/zoneinfo/PRC /etc/localtime

(6) 安装并打开ntp服务，每个节点相同

[root@cent-1 ~]# yum install -y ntp
[root@cent-1 ~]# service ntpd start
Starting ntpd:                                             [  OK  ]

(7) 检查并调整pid_max(可选)

echo "kernel.pid_max=65535" >> /etc/sysctl.conf
sysctl -p

(8) 检查并调整swap交换分区
swap分区一般至少配置为总内存的一半，关于配置swap的方法可参考https://blog.csdn.net/Post_Yuan/article/details/80831348

2 准备CDH 离线安装包及parcel文件

[root@cent-1 cm5.8.1-el6]# ll
total 689856
drwxrwxrwx. 3 root root      4096 Jan 17 13:23 cent6_dependencies
-rw-r--r--. 1 root root   9366164 Jan 17 13:20 cloudera-manager-agent-5.8.1-1.cm581.p0.7.el6.x86_64.rpm
-rw-r--r--. 1 root root 554964428 Jan 17 13:20 cloudera-manager-daemons-5.8.1-1.cm581.p0.7.el6.x86_64.rpm
-rw-r--r--. 1 root root      8448 Jan 17 13:20 cloudera-manager-server-5.8.1-1.cm581.p0.7.el6.x86_64.rpm
-rw-r--r--. 1 root root     10156 Jan 17 13:20 cloudera-manager-server-db-2-5.8.1-1.cm581.p0.7.el6.x86_64.rpm
-rw-r--r--. 1 root root 142039186 Jan 17 13:20 oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

[root@cent-1 5.8.0-el6]# ll
total 1453684
-rw-r--r--. 1 root root 1488562357 Jan 17 13:21 CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel
-rw-r--r--. 1 root root         41 Jan 17 13:21 CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha

3 createrepo创建yum本地源

[root@cent-1 cm5.8.1-el6]# yum install -y createrepo

[root@cent-1 cm5.8.1-el6]# createrepo .
Spawning worker 0 with 65 pkgs
Workers Finished
Gathering worker results

Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

4 生成临时HTTP服务并准备本地yum repo文件

[root@cent-1 cm5.8.1-el6]# python -m SimpleHTTPServer 8910 &
[1] 10858
[root@cent-1 cm5.8.1-el6]# Serving HTTP on 0.0.0.0 port 8910 ...

[root@cent-1 cm5.8.1-el6]# ps -ef | grep 8910
root     10858  1888  0 13:28 pts/0    00:00:00 python -m SimpleHTTPServer 8910
root     10868  1888  0 13:28 pts/0    00:00:00 grep 8910

[root@cent-1 cm5.8.1-el6]# cat cm_local.repo
[cm_local]
baseurl=http://cent-1:8910
enabled=1
gpgcheck=0
[root@cent-1 cm5.8.1-el6]# mv cm_local.repo /etc/yum.repos.d/

5 从本地yum源安装CDH安装包

[root@cent-1 ~]# yum install cloudera-manager-server cloudera-manager-server-db-2 cloudera-manager-daemons oracle-j2sdk1.7 -y

[root@cent-1 ~]# rpm -qa | grep cloudera
cloudera-manager-daemons-5.8.1-1.cm581.p0.7.el6.x86_64
cloudera-manager-server-db-2-5.8.1-1.cm581.p0.7.el6.x86_64
cloudera-manager-server-5.8.1-1.cm581.p0.7.el6.x86_64

6 复制CDH parcel文件到/opt/cloudera/parcel-repo/下

[root@cent-1 5.8.0-el6]# cp CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel* /opt/cloudera/parcel-repo/

7 启动CDH 服务，并检查7180端口是否成功监听

[root@cent-1 ~]# service cloudera-scm-server-db start
[root@cent-1 ~]# service cloudera-scm-server start

[root@cent-1 ~]# service cloudera-scm-server-db status
pg_ctl: server is running (PID: 11162)
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"
[root@cent-1 ~]# service cloudera-scm-server status
cloudera-scm-server (pid  11203) is running...

[root@cent-1 ~]# netstat -nlp | grep 7180
tcp        0      0 0.0.0.0:7180                0.0.0.0:*                   LISTEN      11205/java

8 打开CDH Manager网页部署Hadoop集群，登录默认用户 admin/admin

9 根据提示配置如下

10 至此，CDH Hadoop集群安装完毕，最后需要关闭本地源

[root@cent-1 cm5.8.1-el6]# ps -ef | grep 8910
root     10858  1888  0 13:28 pts/0    00:00:03 python -m SimpleHTTPServer 8910
root     23231  1888  0 14:52 pts/0    00:00:00 grep 8910
[root@cent-1 cm5.8.1-el6]# kill -9 10858
[root@cent-1 cm5.8.1-el6]# ps -ef | grep 8910
root     23286  1888  0 14:52 pts/0    00:00:00 grep 8910
[1]+  Killed                  python -m SimpleHTTPServer 8910
[root@cent-1 cm5.8.1-el6]# rm -rf /etc/yum.repos.d/cm_local.repo