以下步骤亲测有效无误!!!!!
一、 基础环境准备
1)软件清单
Centos7(64位)服务器3台,地址如下:
jdk1.8
Mysql5.7 (必须要5.7的版本!!)
mysql-connector-java-5.1.42-bin.jar
CDH包:https://ro-bucharest-repo.bigstepcloud.com/cloudera-repos/cdh5/parcels/5.15.2/
CM包:https://ro-bucharest-repo.bigstepcloud.com/cloudera-repos/cm5/redhat/7/x86_64/cm/5.15.2/RPMS/x86_64/
2)服务器地址
192.168.56.1 (此地址映射你自己的服务器地址)
192.168.56.2 (此地址映射你自己的服务器地址)
192.168.56.3 (此地址映射你自己的服务器地址)
3)修改hostname
vim /etc/hostname
修改主机名为cdh*后进行以下操作
service network restart
4)配置hosts
vim /etc/hosts
192.168.56.1 cdh1
192.168.56.2 cdh2
192.168.56.3 cdh3
5)免密访问
ssh-keygen -t rsa
//三台服务器全都执行以下命令
ssh-copy-id root@cdh1
ssh-copy-id root@cdh2
ssh-copy-id root@cdh1
6)关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
7)关闭SELINUX
setenforce 0
vim /etc/sysconfig/selinux
SELINUX=disabled
8)配置NTP
yum -y install ntp
yum install nfs-utils rpcbind
systemctl start rpcbind.service
systemctl enable rpcbind.service
systemctl start ntpd
systemctl enable ntpd
在cdh1节点
vim /etc/ntp.conf
server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org
restrict 0.cn.pool.ntp.org nomodify notrap noquery
restrict 1.cn.pool.ntp.org nomodify notrap noquery
restrict 2.cn.pool.ntp.org nomodify notrap noquery
restrict 3.cn.pool.ntp.org nomodify notrap noquery
server 127.127.1.0
fudge 127.127.1.0 stratum 10
systemctl start ntpd.service
systemctl enable ntpd.service
ntpstat
在cdh[2-3]节点
vim /etc/ntp.conf
server cdh1 prefer
systemctl start ntpd.service
systemctl enable ntpd.service
ntpdate -u cdh1
ntpstat
9)安装JDK
mkdir /usr/local/java
cp /home/ jdk-8u131-linux-x64.gz /usr/local/java/
cd /usr/local/java/
tar xvf jdk-8u131-linux-x64.gz
rm jdk-8u131-linux-x64.gz
scp -r /usr/local/java/ root@cdh2:/usr/local/
scp -r /usr/local/java/ root@cdh3:/usr/local/
设置环境变量
vi /etc/profile
export JAVA_HOME=/usr/local/jdk
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH
java -version
10)安装MySQL
删除自带数据库
rpm -qa | grep mariadb
rpm -e –nodeps 包名
rm -rf /etc/my.cnf
安装请参考以下链接:
https://blog.csdn.net/baidu_32872293/article/details/80557668
创建数据库并使本地root用户有访问权限
mysql -u root -p
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database am DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
flush privileges;
安装CDH
安装CM所有包
修改agent配置
vim /etc/cloudera-scm-agent/config.ini
server_host=cdh1
cdh1节点
cp /home/ mysql-connector-java-5.1.47-bin.jar /usr/share/cmf/lib/
所有节点
cp /home/ mysql-connector-java-5.1.47-bin.jar /usr/share/java
初始化脚本配置数据库
/usr/share/cmf/schema/scm_prepare_database.sh mysql -h cdh1 -uroot -p000000 --scm-host cdh1 scm root 000000
将安装包文件复制到默认安装路径
cp /home/CDH-5.8.5-1.cdh5.8.5.p0.5-el7.parcel /opt/cloudera/parcel-repo/
cp /home/CDH-5.8.5-1.cdh5.8.5.p0.5-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.8.5-1.cdh5.8.5.p0.5-el7.parcel.sha
cp /home/manifest.json /opt/cloudera/parcel-repo/
cdh1启动服务器端:
/opt/cm-5.8.5/etc/init.d/cloudera-scm-server start
cdh[1-3] 启动Agent:
/opt/cm-5.8.5/etc/init.d/cloudera-scm-agent start
启动需要2-5分钟
地址:http://localhost:7180
用户:admin
密码:admin
常见问题汇总
1、检查主机正确性警告解决方式
在三台服务器中执行以下命令
sysctl -w vm.swappiness=10
vim /etc/sysctl.conf
vm.swappiness=10
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
2、数据库设置hue数据库无法连接
在cdh2和cdh3节点执行一下命令
scp -r /usr/lib64/mysql/libmysqlclient.so.18 root@cdh1:/usr/lib64
或者参考
https://blog.csdn.net/ckg8933/article/details/83379279
3、hive缺少mysql驱动
cdh1执行
cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hive/lib/
4、oozie缺少mysql驱动
cdh1执行
cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/oozie/lib/
cp /home/mysql-connector-java-5.1.42-bin.jar /var/lib/oozie/mysql-connector-jave.jar
chmod 777 /var/lib/oozie/mysql-connector-jave.jar
5、hue数据库测试连接失败
https://blog.csdn.net/ckg8933/article/details/83379279
6、数据库测试连接失败
https://blog.csdn.net/levy_cui/article/details/51142843