CDH集群规划

Cluster1集群规划

	版本
操作系统	Redhat7.2
JDK	1.8
mysql	5.7.20
CM	6.3.1
CDH	6.2.1

ip	hostname
10.162.5.181(SERVER)	dsj-4t-323
10.162.5.182	dsj-4t-324
10.162.5.183	dsj-4t-325
10.162.5.184	dsj-4t-326
10.162.5.185	dsj-4t-327
10.162.5.186	dsj-4t-328
10.162.5.187	dsj-4t-329
10.162.5.188	dsj-4t-330
10.162.5.189	dsj-4t-331

Cluster2集群规划

	版本
JDK	1.8
mysql	5.7.29
CM	6.3.1
CDH	6.2.1

ip	hostname
10.162.5.190(SERVER)	dsj-4t-332
10.162.5.191	dsj-4t-333
10.162.5.192	dsj-4t-334
10.162.5.193	dsj-4t-335
10.162.5.194	dsj-4t-336

安装CM过程中遇到的问题

安装agent时检测不到信号:

确保9001与9000端口没有被占用

netstat -tunlp | grep 9001
sudo kill -9 <PID>

注意：不要先杀死所有主机的9001的进程再统一安装agent，因为一些进程啥杀死后过一段时间还会自己启动，要逐个主机的安装agent。

出现ProtocolError:

ps -ef | grep supervisord
kill -9 <PID>
ps aux | grep super
kill -9 <PID>
sudo service cloudera-scm-agent restart

安装CDH过程中遇到的问题

1. 所有CDH的安装位置

cd /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib

2. 查看zookeeper数据

cd /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/zookeeper/bin/
sudo bash zkCli.sh -server 127.0.0.1:2181

3. 修改kafka的ID后还要修改meta.propertie中的id

cd /var/local/kafka/data
sudo vim meta.propertie

4. 安装HDFS后NFS Gateway启动失败

先来到安装了NFS Gateway的主机上，启动rpcbind服务

sudo service rpcbind status
sudo service rpcbind start

5. 安装HDFS时出现错误

Failed to add storage directory [DISK]file:/mnt/sd01/dfs/dn
java.io.IOException: Incompatible clusterIDs in /mnt/sd01/dfs/dn: namenode clusterID = cluster22; datanode clusterID = cluster13

这种情况是所有datanode无法启动，需要去所有安装datanode的主机上删除current文件

rm -rf /mnt/sd01/dfs/dn/current/
rm -rf /mnt/sd02/dfs/dn/current/
rm -rf /mnt/sd03/dfs/dn/current/
rm -rf /mnt/sd04/dfs/dn/current/
rm -rf /mnt/sd05/dfs/dn/current/
rm -rf /mnt/sd06/dfs/dn/current/
rm -rf /mnt/sd07/dfs/dn/current/
rm -rf /mnt/sd08/dfs/dn/current/
rm -rf /mnt/sd09/dfs/dn/current/
rm -rf /mnt/sd10/dfs/dn/current/
rm -rf /mnt/sd11/dfs/dn/current/

6. 安装HIve是测试数据库连接时报错：Logon denied for user/password. Able to find the database server and database, but logon request was rejected

root用户无法连接新建的hive数据库，要赋权一下

mysql -uroot -p
grant all privileges on *.* to 'root'@'10.162.5.190' identified by 'HXa#@2018QdoP' with grant option;
flush privileges;

7.修改配置文件在web上能够上传下载文件到HDFS：

8.从本地上传文件到HDFS权限不够，在linux执行：

sudo groupadd supergroup
sudo usermod -a -G supergroup root
hdfs dfsadmin -refreshUserToGroupsMappings
grep 'supergroup:' /etc/group
hadoop fs  -moveFromLocal  ./1.txt  /