Cloudera CDH 安装详细说明

目的

很多云厂商提供了PaaS服务，可以直接使用Hadoop生态里的组件。从成本考虑，也从更大的自由度考虑，自己安装Cloudera CDH 构建Hadoop生态也是一种选择。

准备工作

最低配置建议至少6台服务器：
Cloudera Manager Server *1
Master Server *1 （NameNode、HMaster、ResourceManager等）
Master HA Server *1 （NameNode，SecondaryNameNode 等）
Node Servers *3 （DataNode、RegionServer、NodeManager等）

系统： Centos 7

操作系统初始化

6台Server都初始化

mkdir /data
mkdir /data/log
mkdir /data/log/cloudera-scm-firehose
mkdir /data/log/cloudera-scm-eventserver
mkdir /data/log/cloudera-scm-alertpublisher
mkdir /data/lib/cloudera-scm-server -p

yum install net-tools sysstat* deltarpm lrzsz ntp wget vim gcc -y

yum -y install epel-release
yum install yum-axelget -y

sed -i "s/enforcing/disabled/g"  /etc/selinux/config

reboot

安装准备

选择一：建立本地yum源

mkdir /tools
cd /tools
axel -n 10 -o /tools/ http://archive-primary.cloudera.com/cm5/repo-as-tarball/5.15.0/cm5.15.0-centos7.tar.gz
tar -zxvf cm5.15.0-centos7.tar.gz 

yum install httpd -y
cd /var/www/html  
mkdir -p cm5/redhat/7/x86_64
cd cm5/redhat/7/x86_64
mv /tools/cm . 
chmod -R ugo+rX cm

systemctl start httpd
systemctl enable httpd.service

cd /etc/yum.repos.d/

上传以下repo文件
[需要根据服务地址修改内容]

[myrepo]  
name=myrepo  
baseurl=http://服务器ip/cm5/redhat/7/x86_64/cm/5.15.0/
enabled=1  
gpgcheck=0

yum clean all
yum makecache

选择二：使用远程cloudera源

cd /etc/yum.repos.d/

上传以下repo文件

[cloudera-manager]
name=Cloudera Manager
baseurl=http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5/
gpgkey = http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck=1

yum clean all
yum makecache

安装

#安装Cloudera Manager Server 端 (Cloudera Manager Server)

-----------------预先下载-----------------------------

axel -n 10 -o /data/tools/  https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.0/RPMS/x86_64/cloudera-manager-agent-5.15.0-1.cm5150.p0.62.el7.x86_64.rpm
axel -n 10 -o /data/tools/  https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.0/RPMS/x86_64/cloudera-manager-daemons-5.15.0-1.cm5150.p0.62.el7.x86_64.rpm
axel -n 10 -o /data/tools/  https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.0/RPMS/x86_64/cloudera-manager-server-5.15.0-1.cm5150.p0.62.el7.x86_64.rpm
axel -n 10 -o /data/tools/  https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.0/RPMS/x86_64/cloudera-manager-server-db-2-5.15.0-1.cm5150.p0.62.el7.x86_64.rpm
axel -n 10 -o /data/tools/  https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.0/RPMS/x86_64/enterprise-debuginfo-5.15.0-1.cm5150.p0.62.el7.x86_64.rpm
axel -n 10 -o /data/tools/  https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.0/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

cp /data/tools/*.rpm /var/cache/yum/x86_64/7/cloudera-manager/packages

---

yum -y install oracle-j2sdk1.7.x86_64
yum -y install cloudera-manager-server
yum -y install cloudera-manager-server-db-2
cd /data/tools
axel -n 10 -o /data/tools/ http://archive.cloudera.com/cm5/installer/5.15.0/cloudera-manager-installer.bin
chmod 777 cloudera-manager-installer.bin
./cloudera-manager-installer.bin

可选操作

mv /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.properties.bak

安装Cloudera Manager agent 端 (所有servers)

------预先下载------ 同Cloudera Manager server

yum -y install oracle-j2sdk1.7.x86_64 
yum -y install cloudera-manager-agent 

准备安装CDH

mkdir /data/cloudera/parcels -p
mkdir /data/cloudera/parcel-repo/ -p
chown cloudera-scm.cloudera-scm /data -R

#离线下载 Parcel CDH 5.15.0-1.cdh5.15.0.p0.21
#只需要cloudera manager server 下载
#目录看配置，默认是/opt/cloudera/parcels

axel -n 10 -o /data/tools/  http://archive.cloudera.com/cdh5/parcels/5.15.0.21/CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel
axel -n 10 -o /data/tools/  http://archive.cloudera.com/cdh5/parcels/5.15.0.21/CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1

cp /data/tools/CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1 /data/tools/CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha

cp /data/tools/CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel* /data/cloudera/parcel-repo/
chown cloudera-scm.cloudera-scm /data/cloudera/parcel-repo/*

配置scm到Mysql数据库

准备数据库
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_mysql.html#cmig_topic_5_5
https://www.cloudera.com/documentation/enterprise/latest/topics/prepare_cm_database.html

下载mysql-connector-java.jar

mkdir /usr/share/java/
cp mysql-connector-java.jar /usr/share/java/

新建mysql数据库

Service	Database	User
Cloudera Manager Server	scm	scm
Activity Monitor	amon	amon
Reports Manager	rman	rman
Hue	hue	hue
Hive Metastore Server	metastore	hive
Sentry Server	sentry	sentry
Cloudera Navigator Audit Server	nav	nav
Cloudera Navigator Metadata Server	navms	navms
Oozie	oozie	oozie

修改db.properties

cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.properties.bak
vi /etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=mysqldb:3306
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=scm_pd
com.cloudera.cmf.db.setupType=EXTERNAL

/usr/share/cmf/schema/scm_prepare_database.sh mysql -h mysqldb scm scm scm_pd
/etc/init.d/cloudera-scm-server restart

访问 http://cloudera manager servier ip:7180/
admin/admin