cdh集群安装

环境说明

Cloudera Manager：5.13.3
CDH: 5.13.3
CentOS 7.3  x64
JDK 1.8.0_152
MySQL 5.7.20
关闭防火墙和SELinux (所有节点)
配置时间同步（所有节点）

#下载CM安装包(主节点)
wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.13.3_x86_64.tar.gz
wget http://archive.cloudera.com/cdh5/parcels/5.13.3/CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel
wget http://archive.cloudera.com/cdh5/parcels/5.13.3/CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1
wget http://archive.cloudera.com/cdh5/parcels/5.13.3/manifest.json

#Mysql安装包及JDBC驱动(主节点)生产安装时只需安装驱动即可
wget http://ftp.ntu.edu.tw/MySQL/Downloads/Connector-J/mysql-connector-java-5.1.45.tar.gz

一、安装

1、安装依赖包

yum install -y  psmic cmake gcc gcc-c++ ncurses-devel bison   libxml2 libxml2-devel  bzip2-devel  libssl* python-devel  libaio* bzip2 libxml2-python
yum install -y chkconfig bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb
yum install -y make automake autoconf libtool perl  mod_ssl  python-psycopg2  postgresql-server openssl-devel MySQL-python

二、配置环境

1. 网络配置 (所有节点)

#修改主机名, 切记主机名不能出现下划线
#修改主机名(/etc/hostname):
hostnamectl set-hostname  cdh-master
hostnamectl set-hostname  cdh-slave1
hostnamectl set-hostname  cdh-slave2
hostnamectl set-hostname  cdh-slave3
hostnamectl set-hostname  cdh-slave4
hostnamectl set-hostname  cdh-slave5

#修改 /etc/sysconfig/netroot, 使之生效 source
vim /etc/sysconfig/netroot    
NETROOTING=yes
HOSTNAME=cdh-xxx

#修改 /etc/hosts 加入
vim /etc/hosts
192.168.168.83 cdh-master
192.168.168.84 cdh-slave1
192.168.168.85 cdh-slave2
192.168.168.86 cdh-slave3
192.168.168.87 cdh-slave4
192.168.168.88 cdh-slave5

2. 打通SSH，设置ssh无密码登陆, 集群内节点能够互相无密码访问 (所有节点)

ssh-keygen 
ssh-copy-id -i id_rsa.pub root@192.168.168.*

3. 安装jdk（所有节点）

mkdir /usr/java
tar zxf /root/jdk1.8.0_111.tar.gz -C /usr/java

#配置环境变量
vim /etc/profile
#增加如下内容:
JAVA_HOME=/usr/java/jdk1.8.0_111
JRE_HOME=/usr/java/jdk1.8.0_111/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH

#刷新配置
source /etc/profile

4. 安装配置MySql（主节点）

(1)解压安装包

tar zxf mysql-5.6.39-linux-glibc2.12-x86_64.tar.gz -C /opt/

(2)修改文件目录名称

mv /opt/mysql-5.6.39-linux-glibc2.12-x86_64/ mysql

(3)创建数据目录

mkdir sqldata
useradd -M -s /sbin/nologin mysql
#授权
cd /data/
chown -R mysql.mysql mysql

(4)配置环境变量

vim  /etc/profile
PATH=$PATH:/opt/mysql/bin
source /etc/profile

(5)初始化

yum install libaio* -y
scripts/mysql_install_db --user=root --basedir=/opt/mysql --datadir=/opt/mysql/sqldata
chmod -R 777 sqldata

(6)复制配置文件

cp support-files/my-default.cnf /etc/my.cnf
vim /etc/my.cnf
max_allowed_packet=1024M
basedir = /opt/mysql/
datadir = /opt/mysql/sqldata

(7)复制启动脚本

cp support-files/mysql.server /etc/init.d/mysqld
chkconfig --add mysqld
chkconfig mysqld on
service mysqld start

(8)登录

mysqladmin -uroot  password 123abcABC
mysql -uroot -p123abcABC

(9)授权用户root使用密码passwd从任意主机连接到mysql服务器

grant all privileges on *.* to 'root'@'%' identified by '123abcABC' with grant option;
flush privileges;

(10)建数据库（用于后边组件安装）

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

三、安装cdh

1. 安装Cloudera Manager Server 和Agent

mkdir /opt/cloudera-manager
tar zxf cloudera-manager-centos7-cm5.13.3_x86_64.tar.gz
mv cloudera  cm-5.13.3  /opt/cloudera-manager
mkdir /var/cloudera-scm-server（所有节点）

2、Mysql驱动放到cm的lib下(所有节点)

scp mysql-connector-java-5.1.45.tar.gz root@192.168.168.*:/root
tar zxf mysql-connector-java-5.1.45.tar.gz
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar /usr/share/java/mysql-connector-java.jar
cp mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar  /opt/cloudera-manager/cm-5.13.3/share/cmf/lib/

3、初始化CM5的数据库(主节点)

/opt/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p scm scm
#查看配置文件，发现信息已写入：
vim /data/cm-5.13.3/etc/cloudera-scm-server/db.properties

4、Agent配置(主节点)

vim /opt/cloudera-manager/cm-5.13.3/etc/cloudera-scm-agent/config.ini
server_host=cdh-master
cloudera_mysql_connector_jar=/usr/share/java/mysql-connector-java.jar

5、同步Agent到其他节点(主节点)

scp -r /opt/cloudera-manager/cm-5.13.3 root@192.168.168.*:/opt/cloudera-manager

6、创建cloudera-scm用户(所有节点)

useradd --system --home=/opt/cloudera-manager/cm-5.13.3/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

7、准备Parcels(主节点)

mkdir -p /opt/cloudera/parcel-repo/
cp CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel /opt/cloudera/parcel-repo/
cp CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha
cp manifest.json /opt/cloudera/parcel-repo/

#所有节点
chown -R  cloudera-scm:cloudera-scm /opt/cloudera/
chown cloudera-scm:cloudera-scm /var/cloudera-scm-server
chown cloudera-scm:cloudera-scm /opt/cloudera-manager 

vim /etc/sysctl.conf
vm.swappiness = 10
sysctl –p

#所有节点执行
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

#开机执行，添加上面的语句
chmod +x /etc/rc.d/rc.local
vim /etc/rc.local

9、启动相关脚本

/opt/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-server start    (主节点)
/opt/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-agent start     (所有节点)
#同样可以使用stop, restart

三、CDH5安装配置

1. 登录

http://172.21.161.32:7180/cmf/login, 用户名和密码是 admin/admin