CentOS7安装cdh6.0.1

JDK

卸载自带java

rpm -qa |grep java
#删除自带的java
yum remove java*

安装jdk（每个节点rpm安装）

rpm -ivh jdk-8u261-linux-x64.rpm

在/etc/profile中加入：

vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_261-amd64
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin

使配置生效（每个节点）

source /etc/profile

1.配置hosts

vi /etc/hosts

192.168.1.1 n1
192.168.1.2 n2
192.168.1.3 n3
192.168.1.4 n4
192.168.1.5 n5
192.168.1.6 n6

2.禁用防火墙

systemctl stop firewalld
#禁止开机自启
systemctl disable firewalld

3.设置SELinux模式

setenforce 0     表示关闭selinux防火墙。 

 vi /etc/selinux/config    
 SELINUX=disabled

4.启用NTP服务

安装
yum install ntp

将n1服务器作为ntpserver

【n1服务器】
 vi /etc/ntp.conf
注释掉0~3server
server 127.127.1.0
fudge 127.127.1.0 stratum 11
加入本地时间服务器，主要是 127.127.1.0，由于是顶层服务，也就是说没有上级时间服务器了，
这里就是使用n1服务器的时间为准。

【其他服务器】
vi /etc/ntp.conf
注释掉0~3server
server 192.168.1.1
fudge 127.127.1.0 stratum 11 #这个是需要11，因为不是顶级的
------------------------------------------------------
接着，重启或者启动ntpd服务：
systemctl restart ntpd

开机自启
systemctl enable ntpd

查看ntp服务状态
ntpq -p

将系统时钟同步到NTP服务器
ntpdate -u n1

5.配置集群免密登录

生成秘钥
ssh-keygen
localhost免密码：cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
其他服务器免密码：ssh-copy-id root@xxx.xxx.xxx.xxx
测试 ssh xxx.xxx.xxx.xxx 退出 exit

6.安装和配置数据库

安装参照：https://blog.csdn.net/m0_37859032/article/details/107856559

mysql -uroot -p123456

create database scm default character set utf8 default collate utf8_general_ci;
grant all on scm.* to 'scm'@'%' identified by '123456';
create database amon default character set utf8 default collate utf8_general_ci;
grant all on amon.* to 'amon'@'%' identified by '123456';
create database rman default character set utf8 default collate utf8_general_ci;
grant all on rman.* to 'rman'@'%' identified by '123456';
create database hue default character set utf8 default collate utf8_general_ci;
grant all on hue.* to 'hue'@'%' identified by '123456';
create database metastore default character set utf8 default collate utf8_general_ci;
grant all on metastore.* to 'hive'@'%' identified by '123456';
create database sentry default character set utf8 default collate utf8_general_ci;
grant all on sentry.* to 'sentry'@'%' identified by '123456';
create database nav default character set utf8 default collate utf8_general_ci;
grant all on nav.* to 'nav'@'%' identified by '123456';
create database navms default character set utf8 default collate utf8_general_ci;
grant all on navms.* to 'navms'@'%' identified by '123456';
create database oozie default character set utf8 default collate utf8_general_ci;
grant all on oozie.* to 'oozie'@'%' identified by '123456';
flush privileges;

7.安装 MySQL JDBC 驱动(所有节点)

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz 
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
cd .. 
rm -rf mysql-connector-java-5.1.46*
复制驱动到其他节点
scp /usr/share/java/mysql-connector-java.jar root@n2:/usr/share/java/

8.禁用透明大页面压缩

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

并将上面的两条命令写入开机自启动
vi /etc/rc.local

9.优化交换分区

echo "vm.swappiness = 10" >> /etc/sysctl.conf
sysctl -p

10.配置 Cloudera Manager 仓库

• 下载 cloudera-manager.repo：

wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/

cm6版本地址：https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cm_6_version_download.html

• 导入 repository signing GPG key：

rpm --import https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPM-GPG-KEY-cloudera

11.安装 Cloudera Manager Server

使用仓库安装会比较慢，建议先把需要的rpm下载下来，进行离线安装或者建私有仓库，主要下面的三个软件包：

cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm

cloudera-manager-daemons 必须安装的。

在cm服务器上：

yum install cloudera-manager-daemons cloudera-manager-server
#或者离线安装
yum install cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm -y

在其他服务器上：

yum install cloudera-manager-daemons
#或者离线安装
yum install cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm -y

12.设置Cloudera Manager数据库

rm -rf /etc/cloudera-scm-server/db.properties*
/opt/cloudera/cm/schema/scm_prepare_database.sh  mysql  scm scm 123456

13.配置CDH的软件包 parcels

把CDH的parcels包下载到CM主服务器上

cd /opt/cloudera/parcel-repo
wget https://archive.cloudera.com/cdh6/6.0.1/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
wget https://archive.cloudera.com/cdh6/6.0.1/parcels/manifest.json
#在manifest.json文件中，找到对应版本的秘钥，复制到.sha文件中。
echo "2e650f1f1ea020a3efc98a231b85c2df1a50b030" > CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha
#修改属主属组
chown cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo/*

14.启动CM

启动 Cloudera Manager Server：
systemctl start cloudera-scm-server
查看日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

启动成功
浏览器访问 http://<server_host>:7180
登录：用户名密码 admin

问题

1.Install Agents -> 无法检测 Cloudera Manager Server

首先判断cloudera-scm-agent在节点服务状态是run
在节点服务器执行以下命令：

mv /usr/bin/host /usr/bin/host.bak

2.切换用户su hdfs出现This account is currently not available.
看看 hdfs的帐号信息

[root@n1 home]#  cat /etc/passwd|grep hdfs
hdfs:x:985:978:Hadoop HDFS:/var/lib/hadoop-hdfs:/sbin/nologin

发现它的shell是“/sbin /nologin”，需要将其改成“/bin/bash”

[root@n1 home]# vi /etc/passwd 

cloudera-scm:x:987:981:Cloudera Manager:/var/lib/cloudera-scm-server:/bin/bash
flume:x:986:979:Flume:/var/lib/flume-ng:/bin/bash
hdfs:x:985:978:Hadoop HDFS:/var/lib/hadoop-hdfs:/bin/bash
solr:x:984:977:Solr:/var/lib/solr:/bin/bash
sentry:x:983:976:Sentry:/var/lib/sentry:/bin/bash
hue:x:982:975:Hue:/usr/lib/hue:/bin/bash
zookeeper:x:981:974:ZooKeeper:/var/lib/zookeeper:/bin/bash
mapred:x:980:973:Hadoop MapReduce:/var/lib/hadoop-mapreduce:/bin/bash
httpfs:x:979:972:Hadoop HTTPFS:/var/lib/hadoop-httpfs:/bin/bash
sqoop:x:978:971:Sqoop:/var/lib/sqoop:/bin/bash
hive:x:977:970:Hive:/var/lib/hive:/bin/bash
kafka:x:976:969:Kafka:/var/lib/kafka:/bin/bash
kms:x:975:968:Hadoop KMS:/var/lib/hadoop-kms:/bin/bash
yarn:x:974:967:Hadoop Yarn:/var/lib/hadoop-yarn:/bin/bash
oozie:x:973:966:Oozie User:/var/lib/oozie:/bin/bash
kudu:x:972:965:Kudu:/var/lib/kudu:/bin/bash
hbase:x:971:964:HBase:/var/lib/hbase:/bin/bash
impala:x:970:963:Impala:/var/lib/impala:/bin/bash
spark:x:969:962:Spark:/var/lib/spark:/bin/bash

[root@n1 home]# su hdfs
[hdfs@n1 home]$ 

关闭集群

1.关闭集群所有组件
2.关闭Cloudera Management Service
3.关闭所有cloudera-scm-agent节点

#在所有节点上执行
service cloudera-scm-agent stop
# 查看是否关闭
service cloudera-scm-agent status

4.关闭cloudera-scm-server节点

#在cloudera-scm-server节点上执行
service cloudera-scm-server stop
 #查看是否关闭
service cloudera-scm-server status

启动集群

1.启动cloudera-scm-server

systemctl start cloudera-scm-server

查看日志

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

2.在所有节点启动cloudera-scm-agent

systemctl start cloudera-scm-agent

3.启动 Cloudera Management Service
4.启动所有组件
5.集群启动完毕

常用

zk状态：

/opt/cloudera/parcels/CDH/lib/zookeeper/bin/zkServer.sh status
 /opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/zookeeper/bin/zkServer.sh status

agent路径相关：

/etc/cloudera-scm-agent/config.ini
/var/lib/cloudera-scm-agent