cdh3.6.1安装笔记

一、配置hosts文件，实现主机名与ip的映射
1、vim /etc/hosts
2.
106.12.19.239 hadoop125
106.12.183.117 hadoop123
106.12.37.122 hadoop122
设置hostname
hostnamectl set-hostname hadoop125
hostnamectl set-hostname hadoop123
hostnamectl set-hostname hadoop122
查看：
hostname
关闭防火墙与selinux文件
1.systemctl stop firewalld
2.systemctl disable firewalld.service
3.关闭selinux 输入vim /etc/selinux/config,将SELINUX=disabled
4.

临时关闭 selinux

setenforce 0

永久关闭 selinux

sed -i “s/^SELINUX=enforcing/SELINUX=disabled/g” /etc/selinux/config
设置 SSH 免密码登录（主节点）
1.ssh-keygen -t rsa
2.cd ~/.ssh
3.# 将公钥导入到认证文件
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
4.# 认证文件合并，输入 yes 回车再输入服务器密码
ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop123
ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop122
部署 NTP 服务器进行时间同步

安装 ntp

yum install -y ntp

启动ntp

systemctl start ntpd

开机自动启动

systemctl enable ntpd

查看ntp状态

systemctl status ntpd
#在master修改ntp.conf 输入vim /etc/ntp.conf ,注释其他时间服务器，添加为本机
server 127.127.1.0
fudge 127.127.1.0 stratum8
#在其他节点(hadoop123\hadoop122)修改ntp.conf文件 输入 在26行添加server hadoop125 iburst,注释其他时间服务器
在所有节点启动ntp服务
systemctl start ntpd
systemctl enable ntpd
二、安装jdk

创建文件夹

mkdir -p /usr/java/

上传 JDK8 压缩文件

解压

tar -zxvf jdk-8u212-linux-x64.tar.gz

配置当前用户环境变量

vim ~/.bash_profile

在文件中添加如下命令，记得切换自己文件路径

export JAVA_HOME=/usr/java/jdk1.8.0_212
export PATH=$JAV{A}_{H}OME/bin:$PATH

立即生效

source ~/.bash_profile

三、Cloudera Manager 安装
3.1配置本地 Cloudera Manager 仓库

创建本地 CM 仓库文件夹

mkdir -p /var/www/html/cloudera-repos/cm6/

上传下载好的 CM 安装包放入 /var/www/html/cloudera-repos/cm6/ 目录

安装 httpd 和 createrepo

yum -y install httpd createrepo

启动 httpd 服务并设置为开机自启动

systemctl start httpd && systemctl enable httpd

生成 RPM，注意最后必须有 .

cd /var/www/html/cloudera-repos/cm6/ && createrepo .

通过浏览器访问了：默认端口 80

http://106.12.19.239/cloudera-repos/cm6/

3.2、配置本地 CDH 仓库

创建本地 CDH 仓库文件夹

mkdir -p /var/www/html/cloudera-repos/cdh6/

上传下载好的 CDH 安装包放入 /var/www/html/cloudera-repos/cdh6/ 目录

生成 RPM，注意最后必须有 .

cd /var/www/html/cloudera-repos/cdh6/ && createrepo .

通过浏览器访问了：默认端口80

http://106.12.19.239/cloudera-repos/cdh6/

3.3构建 yum 源

所有机器都需要配置

vim /etc/yum.repos.d/cloudera-repo.repo

拷贝如下内容

[cloudera-repo]
name=cloudera-repo
baseurl=http://hadoop125/cloudera-repos/cm6/
enabled=1
gpgcheck=0

清理

yum clean all
yum makecache

3.4MySQL安装（主节点）

1、将Mysql所需的rpm包上传到hadoop125节点上

创建 mysql 文件夹

mkdir -p /var/www/html/cloudera-repos/mysql5/

进入目录

cd /var/www/html/cloudera-repos/mysql5/
#创建目录
mkdir -p /usr/share/java/
#复制mysql-connect到/usr/share/java/,如果没有该文件夹可以自行创建
cp mysql-connector-java.jar /usr/share/java/

卸载系统自带的数据库

rpm -qa|grep mariadb | xargs -I {} rpm -e --nodeps {}

按照以下步骤安装mysql数据库

rpm -ivh mysql-community-common-5.7.26-1.el7.x86_64.rpm

rpm -ivh mysql-community-libs-5.7.26-1.el7.x86_64.rpm

rpm -ivh mysql-community-devel-5.7.26-1.el7.x86_64.rpm

rpm -ivh mysql-community-client-5.7.26-1.el7.x86_64.rpm

rpm -ivh mysql-community-server-5.7.26-1.el7.x86_64.rpm

rpm -ivh mysql-community-libs-compat-5.7.26-1.el7.x86_64.rpm

开启数据库服务

systemctl start mysqld

systemctl enable mysqld

获取mysql初始化密码

grep ‘temporary password’ /var/log/mysqld.log

修改Mysql密码

mysql -uroot -p 输入上面获取到的密码

alter user root@localhost identified by ‘2wsx#EDC’;

flush privileges;

#创建所需的数据库并授权
create database metastore default character set utf8;
create user ‘hive’@‘%’ identified by ‘2wsx#EDC’;
create database hive default character set utf8;
grant all privileges on hive.* to ‘hive’@‘%’;
grant all privileges on metastore.* to ‘hive’@‘%’;
create user ‘amon’@‘%’ identified by ‘2wsx#EDC’;
create database amon default character set utf8;
grant all privileges on amon.* to ‘amon’@‘%’;
create user ‘rman’@‘%’ identified by ‘2wsx#EDC’;
create database rman default character set utf8;
grant all privileges on rman.* to ‘rman’@‘%’;
create user ‘sentry’@‘%’ identified by ‘2wsx#EDC’;
create database sentry default character set utf8;
grant all privileges on sentry.* to ‘sentry’@‘%’;
create user ‘nav’@‘%’ identified by ‘2wsx#EDC’;
create database nav default character set utf8;
grant all privileges on nav.* to ‘nav’@‘%’;
create user ‘navms’@‘%’ identified by ‘2wsx#EDC’;
create database navms default character set utf8;
grant all privileges on navms.* to ‘navms’@‘%’;
create user ‘scm’@‘%’ identified by ‘2wsx#EDC’;
create database scm default character set utf8;
grant all privileges on scm.* to ‘scm’@‘%’;
create user ‘oos’@‘%’ identified by ‘2wsx#EDC’;
create database oos default character set utf8;
grant all privileges on oos.* to ‘oos’@‘%’;
create user ‘hue’@‘%’ identified by ‘2wsx#EDC’;
create database hue default character set utf8;
grant all privileges on hue.* to ‘hue’@‘%’;
GRANT ALL PRIVILEGES ON . TO scm@“%” IDENTIFIED BY “2wsx#EDC”;
flush privileges;
#验证
show databases;
quit;
3.5安装 CM Server 和 Agent

主节点
yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

所有从节点

yum install cloudera-manager-daemons cloudera-manager-agent

3.6、建立CM的数据库（主节点）

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

3.7、启动 CM Server

主节点

启动服务

systemctl restart cloudera-scm-server
systemctl restart cloudera-scm-agent

查看服务的状态

systemctl status cloudera-scm-server
systemctl status cloudera-scm-agent

查看系统日志

journalctl -xe

默认日志存放地方

/var/log/cloudera-scm-server

从节点

启动服务

systemctl start cloudera-scm-agent

查看服务的状态

systemctl status cloudera-scm-agent

4、CDH优化

静止透明大页

1、所有主机输入以下命令

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

2、所有主机的/etc/rc.local中也需要加入这两条命令

虚拟内存设置

查看虚拟内存需求率 默认60

cat /proc/sys/vm/swappiness

Cloudera 建议将 vm.swappiness 修改为 0

sysctl -w vm.swappiness=0
echo “vm.swappiness = 0” >> /etc/sysctl.conf

卸载cdh

#卸载cloudera manager server(以下称为cm server)以及cloudera manager agent(以下称为cm agent)相关服务

（1）在所有节点停止cm agent服务

$ systemctl stop supervisord
$ systemctl stop cloudera-scm-agent

（2）在主节点（代指安装cm server的节点）停止cm server服务

$ systemctl stop cloudera-scm-server

(3) 在所有节点卸载cloudera manager相关服务
$ yum remove ‘cloudera-manager-*’
#删除残留数据目录

$ sudo umount cm_processes
$ sudo rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera* /opt/cloudera
$ sudo rm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper /var/lib/hive /var/lib/yarn /var/lib/impala /var/lib/kudu /var/lib/kafka /var/lib/spark /var/lib/sentry /var/lib/hbase
$ sudo rm -Rf /dfs /mapred /yarn
$ sudo rm -Rf /etc/hadoop* /etc/hbase* /etc/hive* /etc/flume* /etc/hive* /etc/kafka* /etc/impala* /etc/hue* /etc/sqoop* /etc/spark* /etc/zookeeper* /etc/kudu*
$ sudo rm /tmp/.scm_prepare_node.lock
$ sudo rm -Rf /etc/cloudera-scm*
#删除数据库（mysql为例）
mysql>

drop database scm;
drop database amon;
drop database rman;
drop database hue;
drop database metastore;
drop database sentry;
drop database nav;
drop database navms;
drop database oos;
drop database hive;
drop database sys;
flush privileges;

注意：在安装CDH时创建了一系列数据库，例如hive、oozie、cm server用到的数据库，这些都要一并删除，具体数据库名称以实际情况为准

卸载mysql

1、 rpm -qa | grep -i mysq
2、 yum -y remove mysql-*
3、 find / -name mysql
4、rm -rf /etc/my.cnf
5、 rm -rf /root/.mysql_sercret

主机运行状况不良

删除agent目录下面的cm_guid文件，并重启失败节点的agent服务恢复。

[root@esgyn001 ~]# cd /var/lib/cloudera-scm-agent/
cm_guid response.avro uuid
[root@esgyn001 cloudera-scm-agent]# rm -rf cm_guid
[root@esgyn001 cloudera-scm-agent]# systemctl restart cloudera-scm-agent

kudu权限不够

cd /var/log/

chown -R kudu:kudu kudu

修改hdfs文件夹权限

hdfs dfs -chmod -R 777 /liu

kudu 报错：Tried to update clock beyond the max. error.

解决：所有机器重启ntp服务
检查主机，开启透明大页