Centos 7 CDH 5 部署

申明：该文章步骤只能参考，不建议参照该文档进行安装，如若参考本安装教程未能成功安装集群，本人概不负责，建议参考官方安装手册，（https://www.cloudera.com/documentation/enterprise/latest/topics/installation.html）。

 

一、安装之前先说遇到的几个坑

1. 、CDH不支持openjdk,只支持oracle jdkl，如果本地不是需要卸载后重装
2. 、MYSQL JDBC驱动 官方拷贝路径 cp mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar ，这个路径是错误的，cp mysql-connector-java-5.1.47-bin.jar /usr/share/cmf/lib/mysql-connector-java-5.1.47
3. 、免ssh无法登陆的问题，除了网上经常说的.ssh，及authorized_keys的权限问题，/root根目录的权限不对的话也会导致无法正常免密码登录，具体错误查看/var/log/secure日志进行排错。

 

二、主机环境配置

 

1、修改主机名

sudo hostnamectl set-hostname master.cluster.com

sudo hostnamectl set-hostname slave1.cluster.com

sudo hostnamectl set-hostname slave2.cluster.com

echo "192.168.199.128 master.cluster.com master">>/etc/hosts

echo "192.168.199.129 slave1.cluster.com slave1">>/etc/hosts

echo "192.168.199.130 slave2.cluster.com slave2">>/etc/hosts

echo "HOSTNAME= master.cluster.com">>/etc/sysconfig/network

echo "HOSTNAME= slave1.cluster.com">>/etc/sysconfig/network

echo "HOSTNAME= slave2.cluster.com">>/etc/sysconfig/network

2、免SSH登陆

在三个节点依次执行

ssh-keygen -t rsa

在主节点执行以下命令

scp slave1:/root/.ssh/id_rsa.pub slave1.pub

scp slave2:/root/.ssh/id_rsa.pub slave2.pub

 

cat id_rsa.pub >>/root/.ssh/authorized_keys

cat slave1.pub >>/root/.ssh/authorized_keys

cat slave2.pub >>/root/.ssh/authorized_keys

 

scp /root/.ssh/authorized_keys slave1:/root/.ssh

scp /root/.ssh/authorized_keys slave2:/root/.ssh

3、防火墙及selinux

在三个节点依次执行

systemctl stop firewalld

systemctl disable firewalld

setenforce 0

sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

 

三、安装Cloudera Manager Server

 

1、卸载自带的JAVA

rpm -e java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 --nodeps

rpm -e java-1.7.0-openjdk-1.7.0.191-2.6.15.4.el7_5.x86_64 --nodeps

2、下载相关rpm包

http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.0/RPMS/x86_64/

在master节点执行如下步骤

yum --nogpgcheck localinstall cloudera-manager-daemons

yum --nogpgcheck localinstall cloudera-manager-server

在slave节点执行如下步骤:

yum --nogpgcheck localinstall cloudera-manager-daemons

yum --nogpgcheck localinstall cloudera-manager-agent

3、安装数据库

在主节点执行

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm

sudo yum update

sudo yum install mysql-server

sudo systemctl start mysqld

/usr/bin/mysql_secure_installation 进行配置密码配置，如下：

 

4、MySQL JDBC Driver 安装

主节点执行

下载 http://www.mysql.com/downloads/connector/j/5.1.html 

tar zxvf mysql-connector-java-5.1.46.tar.gz

#注意：官网该步骤拷贝的路径是错误的 cp mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar

正确拷贝路径：cp mysql-connector-java-5.1.47-bin.jar /usr/share/cmf/lib/mysql-connector-java-5.1.47

 

5、安装各软件的DB元数据库

主节点执行

CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY '123456';

 

6、配置Cloudera数据库

配置脚本如下：

/usr/share/cmf/schema/scm_prepare_database.sh [options] <databaseType> <databaseName> <databaseUser> <password>

/usr/share/cmf/schema/scm_prepare_database.sh mysql scm root 123456

 

四 、安装 CDH

1、下载CDH相关包

提前下载 相关包http://archive.cloudera.com/cdh5/parcels/5.15.1/manifest.json

并放在master节点的/opt/cloudera/parcel-repo目录下，下载文件如下

2、打开http://192.168.199.128:7180/,图形界面安装全部next

 

 

五、参数优化

 

 /proc/sys/vm/swappiness，默认值是30，也就是说当内存使用超过100-30， %70的时候就会使用swap分区，将导致大量的页的换进换出。

 

临时更改：sysctl vm.swappiness=10

永久生效：echo "sysctl vm.swappiness=10">>/etc/sysctl.conf