cdh 集群部署

1.软件下载准备
下载CM软件包:
wget http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.1/RPMS/x86_64/cloudera-manager-agent-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
wget http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.1/RPMS/x86_64/cloudera-manager-server-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
wget http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.1/RPMS/x86_64/cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
wget http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.1/RPMS/x86_64/enterprise-debuginfo-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
wget http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.1/RPMS/x86_64/cloudera-manager-daemons-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
wget http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.1/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
下载cloudera-manager安装文件:
wget http://archive.cloudera.com/cm5/installer/5.15.1/cloudera-manager-installer.bin
下载parcel（文件比较大，2G）
wget http://archive.cloudera.com/cdh5/parcels/5.15.2/CDH-5.15.2-1.cdh5.15.2.p0.3-el7.parcel.sha1
wget http://archive.cloudera.com/cdh5/parcels/5.15.2/CDH-5.15.2-1.cdh5.15.2.p0.3-el7.parcel
下载rpm仓库文件
wget http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo
修改仓库文件cloudera-manager.repo，把版本号加上5.15.1
[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 7 x86_64
name=Cloudera Manager
baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.1/
gpgkey =https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1
2.host配置
# 集群配置使用内部ip,不使用公网ip
ip1 host1
ip2 host2
ip3 host3
# ping 这个网址,获取此ip
ip archive.cloudera.com

3.关闭selinux(可能机子本身就是关闭状态)
vim /etc/sysconfig/selinux
SELINUX=disabled
重启生效
重启后检查n
sestatus -v
显示以下设置成功
SELinux status: disabled

4.主节点到从节点设置免密登录(主节点)
ssh-keygen -t rsa
ssh-copy-id cdh-slave001
ssh-copy-id cdh-slave002

5.安装配置NTP同步(保证集群机子间时间同步)
安装ntp
sudo yum install -y ntp
配置ntp
sudo vi /etc/ntp.conf
修改为以下内容
driftfile /var/lib/ntp/drift
pidfile /var/run/ntpd.pid
logfile /var/log/ntp.log
# Access Control Support
restrict cdh-master
server cdh-master
配置开机启动
sudo systemctl enable ntpd
重启ntp
sudo systemctl restart ntpd
查询同步情况(两种方式):
ntpq -p #显示和cdh-master的同步情况
ntpdc -c loopinfo # offset很小,就是已经同步了

6.安装jdk1.8(主+agent)
安装jdk
sudo tar -xzf jdk1.8.0_101-linux-x64.tar.gz -C /usr/local
配置环境变量如下
export JAVA_HOME=/usr/local/jdk1.8.0_191
export CLASSPATH=.:$CLASSPTAH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin
更新配置文件
source /etc/profile
7.复制yum安装文件(主+agent)
sudo cp /home/yue.jin/cdh/cloudera-manager.repo /etc/yum.repos.d/
8.安装软件包(主+agent)
检查主+agent文件是否缺漏
主机中文件:
/home/yue.jin/cdh/CDH-5.15.2-1.cdh5.15.2.p0.3-el7.parcel
/home/yue.jin/cdh/CDH-5.15.2-1.cdh5.15.2.p0.3-el7.parcel.sha1
/home/yue.jin/cdh/cloudera-manager.repo
/home/yue.jin/cdh/cloudera-manager-agent-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
/home/yue.jin/cdh/cloudera-manager-daemons-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
/home/yue.jin/cdh/cloudera-manager-installer.bin
/home/yue.jin/cdh/cloudera-manager-server-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
/home/yue.jin/cdh/cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
/home/yue.jin/cdh/enterprise-debuginfo-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
/home/yue.jin/cdh/jdk-8u191-linux-x64.tar.gz
/home/yue.jin/cdh/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
agent中文件:
/home/yue.jin/cdh/cloudera-manager.repo
/home/yue.jin/cdh/cloudera-manager-agent-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
/home/yue.jin/cdh/cloudera-manager-daemons-5.15.1-1.cm5151.p0.3.el7.x86_64.rpm
/home/yue.jin/cdh/jdk-8u191-linux-x64.tar.gz
sudo yum localinstall -y --nogpgcheck /home/yue.jin/cdh/*.rpm
9.安装完后检查(主+agent)
所有节点检查
yum list | grep cloudera
主节点检查
yum list|grep oracle-j2sdk

主显示如下:
    [yue.jin@ssp-dev-server002 ~]$ yum list | grep cloudera
    cloudera-manager-agent.x86_64           5.15.1-1.cm5151.p0.3.el7       installed
    cloudera-manager-daemons.x86_64         5.15.1-1.cm5151.p0.3.el7       installed
    cloudera-manager-repository.noarch      5.0-1                          installed
    cloudera-manager-server.x86_64          5.15.1-1.cm5151.p0.3.el7       installed
    cloudera-manager-server-db-2.x86_64     5.15.1-1.cm5151.p0.3.el7       installed
    jdk.x86_64                            2000:1.6.0_31-fcs              cloudera-manager
agent显示如下:
    [yue.jin@ssp-dev-server003 ~]$ yum list | grep cloudera
    cloudera-manager-agent.x86_64           5.15.1-1.cm5151.p0.3.el7       installed
    cloudera-manager-daemons.x86_64         5.15.1-1.cm5151.p0.3.el7       installed
    cloudera-manager-server.x86_64          5.15.1-1.cm5151.p0.3.el7       cloudera-manager
    cloudera-manager-server-db-2.x86_64     5.15.1-1.cm5151.p0.3.el7       cloudera-manager
    jdk.x86_64                              2000:1.6.0_31-fcs              cloudera-manager

10.修改目录权限(主+agent)
sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-agent/ -R

11.重启agent(主+agent)
sudo systemctl restart cloudera-scm-agent
12.修改用户权限(主+agent)
sudo echo ‘cloudera-scm ALL=(ALL) NOPASSWD: ALL’ >> /etc/sudoers
13.此步骤可以不配置,不影响使用,不配置的话在状态监测状态时候有notice
sudo echo never > /sys/kernel/mm/transparent_hugepage/defrag
sudo echo never > /sys/kernel/mm/transparent_hugepage/enabled
sudo echo ‘echo never > /sys/kernel/mm/transparent_hugepage/defrag’ >> /etc/rc.local
sudo echo ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled’ >> /etc/rc.local
sudo echo 0 > /proc/sys/vm/swappiness
sudo sysctl vm/swappiness
sudo echo ‘vm.swappiness = 0’>> /etc/sysctl.conf
sudo sysctl -p
14. 修改agent的jdk
sudo echo ‘export JAVA_HOME=/usr/local/jdk1.8.0_191’ >> /etc/default/cloudera-scm-agent
sudo systemctl restart cloudera-scm-agent
15.预置parcel文件,安装时间将会减少(主)
sudo cp /home/yue.jin/cdh/CDH-5.15.2-1.cdh5.15.2.p0.3-el7.parcel /opt/cloudera/parcel-repo/
sudo cp /home/yue.jin/cdh/CDH-5.15.2-1.cdh5.15.2.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/
sudo mv /opt/cloudera/parcel-repo/CDH-5.15.2-1.cdh5.15.2.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.15.2-1.cdh5.15.2.p0.3-el7.parcel.sha
sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
sudo chmod 0640 /opt/cloudera/parcel-repo/*

16.配置外置数据库,如果配置本地数据库可以不配置本步骤(主)
使用mysql作为数据库，提示 JDBC driver cannot be found. Unable to find the JDBC database jar on host
当报驱动找不到，需要手工拷贝 mysql-connector-java-5.1.6-bin.jar 到 mysql主机上的CM5 的目录
如下目录参考：/usr/share/cmf/lib # 默认目录

sudo cp /home/yue.jin/cdh/mysql-connector-java-5.1.38.jar /usr/share/java/
sudo mv /usr/share/java/mysql-connector-java-5.1.38.jar /usr/share/java/mysql-connector-java.jar
sudo cp /home/yue.jin/cdh/mysql-connector-java-5.1.38.jar /usr/share/cmf/lib/
sudo mv /usr/share/cmf/lib/mysql-connector-java-5.1.38.jar /usr/share/cmf/lib/mysql-connector-java.jar

17.启动server安装(主)
删除预置配置
chmod u+x /home/yue.jin/cdh/cloudera-manager-installer.bin
sudo rm -rf /etc/cloudera-scm-server/db.properties
sudo /home/yue.jin/cdh/cloudera-manager-installer.bin
18.测试安装
远程访问 7180 查询安装情况(server启动一分钟之后才可以启动)
curl http://127.0.0.1:7180/
http://cdh-master:7180/
19.修改ClouderaManager的jdk
sudo echo ‘export JAVA_HOME=/usr/local/jdk1.8.0_191’ >> /etc/default/cloudera-scm-server
sudo vi /etc/default/cloudera-scm-server
export JAVA_HOME=/usr/local/jdk1.8.0_191
sudo systemctl restart cloudera-scm-server
0.安装过程中安装失败执行卸载,要保证卸载干净:
主机卸载
控制台停止并移除所有安装的服务
注销并移除Parcles(主+agent)
sudo rm -rf /opt/cloudera
卸载Cloudera Manager Server(主)
sudo /usr/share/cmf/uninstall-cloudera-manager.sh
##注意：如果集群中没有安装uninstall-cloudera-manager.sh，使用下面的命令卸载： a、停止Cloudera Manager Server及其数据库
sudo service cloudera-scm-server stop
sudo service cloudera-scm-server-db stop
b、卸载Cloudera Manager Server及其数据库
sudo yum remove cloudera-manager-server -y
sudo yum remove cloudera-manager-server-db -y
卸载CDH及Clouder Manager Agent(主+agent)
sudo systemctl stop cloudera-scm-agent
sudo yum remove -y 'cloudera-manager-’
sudo yum clean all
删除所有的Cloudera Manager数据，若考虑以后访问，需要将这些数据备份，若使用了内嵌的数据库PostgreSQL数据库，数据存储在/var/lib/cloudear-scm-server-db(主+agent)
sudo rm -rf /var/cache/yum /usr/share/cmf /var/lib/cloudera /var/cache/yum/x86_64/var/run/cloudera* /tmp/.scm_prepare_node.lock /tmp/cmf* /tmp/scm* /etc/yum.repos.d/cloudera* /var/lib/cloudear-scm-server-db

agent卸载
    sudo rm -rf /opt/cloudera
    sudo systemctl stop cloudera-scm-agent
    sudo yum remove -y 'cloudera-manager-*'
    sudo yum clean all
    sudo  rm -rf /var/cache/yum /usr/share/cmf   /var/lib/cloudera*   /var/cache/yum/x86_64/var/run/cloudera* /tmp/.scm_prepare_node.lock /tmp/cmf* /tmp/scm* /etc/yum.repos.d/cloudera*

问题解决:
1. No portmap or rpcbind service is running on this host. Please start portmap or rpcbind service before attempting to start the NFS Gateway role on this host.
出错的机子执行:sudo systemctl restart rpcbind
2. Unable to access datadir, exiting abnormally
保证 zookeeper datadir下 version-2
3. 各个端口:https://www.cnblogs.com/xiqing/p/9645619.html
集群机组均开启端口
hdfs 8020 控制台 (50070)
hive 10000
hue 8889
impala 21050

4.    No portmap or rpcbind service is running on this host
    service rpcbind restart        
    
5.FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=root, access=WRITE, inode="/data/${ip}/20180713":hdfs:supergroup:
    export HADOOP_USER_NAME=hdfs
    
6.    hbase 不能创建不存在的表
    https://www.cnblogs.com/davidwang456/p/6814648.html
    
7.    主机状态不良
    # rm -rf /var/lib/cloudera-scm-agent/cm_guid      --删除文件
    # /etc/init.d/cloudera-scm-agent restart       --重启服务
8. requires of libmysqlclient.so.18()(64bit)
    wget http://www.percona.com/redir/downloads/Percona-XtraDB-Cluster/5.5.37-25.10/RPM/rhel6/x86_64/Percona-XtraDB-Cluster-shared-55-5.5.37-25.10.756.el6.x86_64.rpm
    rpm -ivh Percona-XtraDB-Cluster-shared-55-5.5.37-25.10.756.el6.x86_64.rpm
9. 正在获取锁
    ps aux|grep scm_prepare_node 删除所有显示的进程
    sudo rm -rf /tmp/scm_prepare_node.*