[教程]CentOS 7 CDH 5.16 安装教程

[教程] CentOS 7 CDH 5.16 安装教程

一、CDH5.16下载地址

1、cm 地址

http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz

2、cdh包的地址

http://archive.cloudera.com/cdh5/parcels/5.16.1.3/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.16.1.3/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.16.1.3/manifest.json

3、其他环境

java1.8以上，mysql驱动5.4

二、配置环境

1、机器环境

ip	host	节点
192.168.239.1	hadoop01	主节点
192.168.239.2	hadoop02	子节点1
192.168.239.3	hadoop03	子节点2

所有机器分别修改本机的hostname

echo "hadoop01" > /etc/hostname
echo "hadoop02" > /etc/hostname
echo "hadoop03" > /etc/hostname

2、修改机器HOST（所有节点）

vi /etc/hosts
192.168.239.1 hadoop01
192.168.239.2 hadoop02
192.168.239.3 hadoop03

3、关闭防火墙（所有节点）

所有机器分别执行

systemctl stop firewalld
systemctl disable firewalld
sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config

4、配置JDK（所有节点）

删除openjdk

rpm -qa|grep java
rpm -e --nodeps java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64

配置JAVA

复制jdk并且修改权限

mv jdk1.8.0_211 /usr/lib/
chmod 777 -R jdk1.8.0_211/

修改环境变量

vi /etc/profile

加入以下
export JAVA_HOME=/usr/lib/jdk1.8.0_211
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME CLASSPATH PATH

source /etc/profile

5、无密码登录（所有节点）

生成公钥

ssh-keygen -t rsa

每台机器执行复制公钥实现无密码登录

ssh-copy-id hadoop01
ssh-copy-id hadoop02
ssh-copy-id hadoop03

6、NPT配置

所有节点安装ntp服务

yum -y install ntp

主节点同步一个时间服务器 ntp.api.gz

vi /etc/ntp.conf
server ntp.api.gz iburst

子节点同步为主节点时间服务器

vi /etc/ntp.conf
server hadoop01 iburst

重启npt服务

systemctl restart ntpd
systemctl enable ntpd

7、下载配置缺少的包

yum  -y  install psmisc MySQL-python at bc bind-libs bind-utils cups-client cups-libs cyrus-sasl-gssapi cyrus-sasl-plain ed fuse fuse-libs httpd httpd-tools keyutils-libs-devel krb5-devel libcom_err-devel libselinux-devel libsepol-devel libverto-devel mailcap noarch mailx mod_ssl openssl-devel pcre-devel postgresql-libs python-psycopg2 redhat-lsb-core redhat-lsb-submod-security  x86_64 spax time zlib-devel

修改部分配置

chmod +x /etc/rc.d/rc.local
echo "echo 0 > /proc/sys/vm/swappiness" >>/etc/rc.d/rc.local
echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>/etc/rc.d/rc.local
echo 0 > /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag

rcp 修改

yum -y install rpcbind
systemctl start rpcbind
echo "systemctl start rpcbind" >> /etc/rc.d/rc.local

三、配置管理数据库 mysql

1、安装数据库

某个几点安装mysql,这里我使用节点三安装，使用mariadb

yum -y install mariadb-server

修改配置加入默认字符集utf8

vi /etc/my.cnf
	[mysqld]
    datadir=/var/lib/mysql
    socket=/var/lib/mysql/mysql.sock
    symbolic-links=0
    character-set-server=utf8     \\添加的行
	
    [mysqld_safe]
    log-error=/var/log/mariadb/mariadb.log
    pid-file=/var/run/mariadb/mariadb.pid
    !includedir /etc/my.cnf.d

重启

systemctl restart mariadb 
systemctl enable mariadb

2、初始化数据库

根据提示操作Y或者N

mysql_secure_installation

3、创建数据库

mysql -uroot -p
#执行sql创建cdh所有需要的所有数据库
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

4、修改数据库权限

grant all on *.* to root@"%" Identified by "rootmc";
如果mysql不在本地要把scm的权限也加进去，
grant all on *.* to scm@"%" Identified by "scm";
exit

四、传包解压（所有节点）

把cm安装包复制到/opt目录下

cd /opt
tar vfx cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz

复制mysql Jar包到环境里

cp mysql-connector-java-5.1.30-bin.jar /usr/share/java/mysql-connector-java.jar
cp /usr/share/java/mysql-connector-java.jar /opt/cm-5.16.1/share/cmf/lib/

把cdh的parcel放到相应文件夹下

mv manifest.json CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/

把sha1 改名 sha

mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha

五、初始化cm

1、初始化CM库

在主节点 把cdh管理的表自动加载到数据库里面

/opt/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql cm -hhadoop03 -uroot -prootmc --scm-host hadoop03 scm scm scm

-hhadoop03 -uroot -prootmc  为mysql的连接信息
--scm-host hadoop03 scm scm scm 为要创建的数据库scm,用户名scm,密码scm

显示 All done, your SCM database is configured correctly! 表示成功
,如果初始化报错。查看日志，有时候会由于scm的用户没有权限登录mysql

2、创建系统用户（所有节点）

useradd --system --home=/opt/cm-5.16.1/run/cloudera-scm-server  --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

修改cm配置文件

sed -i "s/server_host=localhost/server_host=hadoop01/" /opt/cm-5.16.1/etc/cloudera-scm-agent/config.ini

创建允许所有需要文件夹

mkdir /opt/cm-5.16.1/run/cloudera-scm-agent

六、启动

主服务器启动server 子服务器启动agent

/opt/cm-5.16.1/etc/init.d/cloudera-scm-server start
/opt/cm-5.16.1/etc/init.d/cloudera-scm-agent start

访问
192.168.239.1:7180

账户admin 密码admin

七、 补充

设置开机自启

1.复制启动脚本

主节点

cp cloudera-scm-server /etc/init.d/
cp cloudera-scm-agent /etc/init.d/

其他节点

cp cloudera-scm-agent /etc/init.d/

2.修改文件

主节点修改server配置文件路径

vi /etc/init.d/cloudera-scm-server
export JAVA_HOME=xxxxxxx

CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
修改成
CMF_DEFAULTS=${CMF_DEFAULTS:-/opt/cm-5.16.1/etc/default}

所有节点修改agent配置文件路径

vi /etc/init.d/cloudera-scm-agent
CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
修改成
CMF_DEFAULTS=${CMF_DEFAULTS:-/opt/cm-5.16.1/etc/default}