Centos7在线安装CDH6

以下步骤亲测有效无误！！！！！

一、 基础环境准备

1）软件清单
Centos7(64位)服务器3台，地址如下：
jdk1.8
Mysql5.7 （必须要5.7的版本！！）
mysql-connector-java-5.1.42-bin.jar
cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm（https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/）
cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm（https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/）
cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm（https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/）
oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm（https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/）
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel（https://archive.cloudera.com/cdh6/6.3.2/parcels/）
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1（https://archive.cloudera.com/cdh6/6.3.2/parcels/）
manifest.json（https://archive.cloudera.com/cdh6/6.3.2/parcels/）

2）服务器地址
192.168.56.1 （此地址映射你自己的服务器地址）
192.168.56.2 （此地址映射你自己的服务器地址）
192.168.56.3 （此地址映射你自己的服务器地址）

3）修改hostname

vim /etc/hostname

修改主机名为cdh*后进行以下操作

service network restart

4）配置hosts

vim /etc/hosts
#必须先把127.0.0.1的行全部注释
192.168.56.1 cdh1
192.168.56.2 cdh2
192.168.56.3 cdh3

5）免密访问

ssh-keygen -t rsa
//三台服务器全都执行以下命令
ssh-copy-id  root@cdh1
ssh-copy-id  root@cdh2
ssh-copy-id  root@cdh1

6）关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

7）关闭SELINUX

setenforce 0
vim /etc/sysconfig/selinux
SELINUX=disabled

8）配置NTP

yum -y install ntp
systemctl start ntpd 
systemctl enable ntpd 
在cdh1节点
vim /etc/ntp.conf
#删掉其他server
server 127.127.1.0
fudge 127.127.1.0 stratum 10

systemctl restart ntpd.service
systemctl enable ntpd.service
ntpstat
在cdh[2-3]节点
vim /etc/ntp.conf
 #删掉其他server
server cdh1 prefer

systemctl restart ntpd.service
systemctl enable ntpd.service
ntpdate -u cdh1
ntpstat

9）安装JDK

mkdir /usr/local/java
cp /home/ jdk-8u131-linux-x64.gz /usr/local/java/
cd /usr/local/java/
tar xvf jdk-8u131-linux-x64.gz
rm jdk-8u131-linux-x64.gz
scp -r /usr/local/java/ root@cdh2:/usr/local/
scp -r /usr/local/java/ root@cdh3:/usr/local/

设置环境变量

vim /etc/profile
export JAVA_HOME=/usr/local/jdk
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH

java -version

10）安装MySQL

删除自带数据库

rpm -qa | grep mariadb
rpm -e –nodeps 包名
rm -rf /etc/my.cnf

下载依赖并创建用户和组

yum -y install perl perl-devel autoconf
groupadd mysql
useradd -g mysql mysql
id mysql

安装请参考以下链接：

https://blog.csdn.net/baidu_32872293/article/details/80557668
https://www.cnblogs.com/downmoon/p/11896158.html

设置数据库允许远程登陆

mysql -u root -p
use mysql;
update user set host='%' where user='root';

若报错可能是已经设置好了，用下列语句查看

select host from user where user = 'root';
flush privileges;

更改数据库字符驱动

set character_set_client=utf8;
set character_set_connection=utf8;
set character_set_results=utf8;
set character_set_server=utf8;
set character_set_system=utf8;

查看修改结果

show variables like 'character_set_%';

关闭数据库安全模式，使数据库可以增删改

SET SQL_SAFE_UPDATES=0;

创建数据库并使本地root用户有访问权限

mysql -u root -p
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database am DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database rm DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database scm DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all privileges on *.* to 'root'@'cdh1' identified by '000000' with grant option;
flush privileges;

安装CDH Server（master节点）

提前准备：

将mysql-connector-java-5.1.42-bin.jar改名为mysql-connector-java.jar放入/usr/share/java目录下

数据库修改/etc/my.cnf

[mysqld]
validate_password=off
wait_timeout=31536000
interactive_timeout=31536000

安装cloudera的rpm

yum install oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

yum install cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

parcel初始化
新建parcel目录

mkdir -p /opt/cloudera/parcel-repo

将安装包文件复制到默认安装路径

cp /home/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /opt/cloudera/parcel-repo/
cp /home/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
cp /home/manifest.json /opt/cloudera/parcel-repo/
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
chmod 777 /opt/cloudera/parcel-repo/*

执行创建数据库命令

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm -hlocalhost  -uroot -p'000000' --scm-host localhost scm scm scm

重启cloudera-scm-server

service cloudera-scm-server restart

地址：http://localhost:7180
用户：admin
密码：admin

安装CDH Agent(每个节点)

提前准备：

将mysql-connector-java-5.1.42-bin.jar改名为mysql-connector-java.jar放入/usr/share/java目录下

安装cloudera的rpm

yum install oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

yum install  cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

修改agent配置

vim /etc/cloudera-scm-agent/config.ini
server_host=cdh1

启动agent

systemctl start cloudera-scm-agent

常见问题汇总

1、检查主机正确性警告解决方式

在三台服务器中执行以下命令

sysctl -w vm.swappiness=10
vim /etc/sysctl.conf
vm.swappiness=10

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

2、数据库设置hue数据库无法连接

在cdh2和cdh3节点执行一下命令

scp -r /usr/lib64/mysql/libmysqlclient.so.18 root@cdh1:/usr/lib64

或者参考
https://blog.csdn.net/ckg8933/article/details/83379279

3、hive缺少mysql驱动

cdh1执行

cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hive/lib/

4、oozie缺少mysql驱动

cdh1执行

cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/oozie/lib/
cp /home/mysql-connector-java-5.1.42-bin.jar /var/lib/oozie/mysql-connector-jave.jar
chmod 777 /var/lib/oozie/mysql-connector-jave.jar

5、hue数据库测试连接失败

https://blog.csdn.net/ckg8933/article/details/83379279

6、数据库测试连接失败

https://blog.csdn.net/levy_cui/article/details/51142843

8、hdfs文件夹权限问题

chmod 777 -R nn
chmod 777 -R snn
chmod o-w -R dn
chmod o-w -R var

9、缺少libmysqlclient.so.18()

下载安装此组件
https://download.csdn.net/download/lufengbo123/12195518
或者
sudo wget -O /usr/lib/libmysqlclient.so.18 http://files.directadmin.com/services/es_7.0_64/libmysqlclient.so.18

10、hbase使用中出现各种异常

删除zookeeper下hbase目录，然后重启hbase即可

如果还是出现Master is initializing报错，则在zookeeper中删除hbase目录后，停止hbase集群，使用 hdfs dfs -rmr /hbase即可
删除hdfs文件后，在hbase前端新建hbase根目录然后重启即可。

11、启动前

看看各个组件内部是否需要创建文件夹和数据库

12、hue中使用hbase报错

Hbase设置一下选项

hue中添加以下选项
[hbase]
thrift_transport=buffered