CDH集群搭建与管理
自建yum仓库
由于官方服务器太慢,并且集群规模够大的话会造成断线安装不成功,除非使用一些外网通道vpn专线,所以本次搭建采用yum离线方式。
具体命令如下:
yum install httpd -y
systemctl restart httpd
systemctl enable httpd
cd /var/www/html
mkdir CDH6.2
cp 安装包 CDH6.2/
createrepo CDH6.2
对于这些文件都是校验文件,不可忽视,否则无法安装成功。
添加新的内容:
name=cdh_repo
baseurl=http://192.168.13.20/repo
enabled=1
gpgcheck=0
安装mysql并创建元数据库
wget http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/mysql57-community-release-el7-7.noarch.rpm
rpm -ivh mysql57-community-release-el7-7.noarch.rpm
yum -y install mysql-community-client-5.7.18-1.el7 mysqlcommunity-server-5.7.18-1.el7
systemctl start mysqld
grep 'temporary password' /var/log/mysqld.log
mysql -u root -p
首先我们修改root的密码。接着创建scm、rman、
hive、oozie、hue、sentry6套库和账号,分别用于Cloudera Manager、
Reports Manager、hive、oozie、hue、sentry存储数据。
set password for root@'localhost'=password(’<password>');
create database scm default character set utf8;
CREATE USER 'scm'@'%' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON scm.* TO 'scm'@'%';
create database rman default character set utf8;
CREATE USER 'rman'@'%' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON rman.* TO 'rman'@'%';
create database hive default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%’;
create database oozie default character set utf8;
CREATE USER 'oozie'@'%' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%';
create database hue default character set utf8;
CREATE USER 'hue'@'%' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%';
create database sentry default character set utf8;
CREATE USER 'sentry'@'%' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON sentry.* TO 'sentry'@'%';
flush privileges;
exit
环境准备
• 修改ip地址
vim /etc/sysconfig/network-script/ifcfg-eth0
• 修改主机名
vi /etc/hostname
填入hadoop1
• 修改本机DNS
vi /etc/hosts
192.168.173.71 hadoop1
192.168.173.72 hadoop2
192.168.173.73 hadoop3
192.168.173.74 hadoop4
192.168.173.75 hadoop5
将虚拟内存的交换设为最小,但不禁用(重启后生效)
echo “vm.swappiness=1” >> /etc/sysctl.conf
• 关闭透明大页(重启后生效)
vi /etc/rc.d/rc.local
加入以下代码
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
保存退出,执行
chmod +x /etc/rc.d/rc.local
修改配置并安装软件
• 关闭SELinux(重启后生效)
vi /etc/selinux/config
改为SELINUX=disabled
• 禁用防火墙(重启后生效)
systemctl disable firewalld
配置chrony服务(时钟同步服务)
我们每台服务器都可以共享访问外网,因此这步不需要。如果是内网
集群,需要修改/etc/chrony.conf指向内网ntp服务器。
• 重新挂载数据盘所在挂载点(目前实验环境只需要重新挂载/)
vi /etc/fstab
找到/dev/mapper/centos-root / xfs defaults 0 0
在defaults后加,noatime
mount -o remount /
安装JDK1.8,并替换JCE文件(Kerberos等需要加密的组件会用到)
在自己电脑上从Oracle官网下载jdk-8u144-linux-x64.rpm,并上传
到root家目录下。下载jce_policy-8.zip,解压后将两个.jar文件上
传到root家目录下
yum -y install jdk*.rpm
cd ~
vi .bash_profile
增加一行JAVA_HOME=/usr
source .bash_profile
mv *policy.jar /usr/java/jdk1.8.0_144/jre/lib/security
安装mysql connector
yum -y install wget
wget http://mirrors.sohu.com/mysql/Connector-J/mysqlconnector-java-5.1.41.tar.gz
tar zxvf mysql-connector-java-5.1.41.tar.gz
cp mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-
bin.jar /usr/share/java/
cd /usr/share/java/
ln -s mysql-connector-java-5.1.41-bin.jar mysql-connectorjava.jar
注:五台机器都要做
安装cm
yum -y install cloudera-manager-daemons cloudera-managerserver
修改以下CM的启动配置,默认分配的JVM堆内存是2G,某些情况下会出
现OOM的情况,我们将其调大到4G。
vi /etc/default/cloudera-scm-server
#将-Xmx2G修改为-Xmx4G
/usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm
'<password>'
#初始化数据库
systemctl start cloudera-scm-server
配置集群
默认用户名密码:admin/admin
由于是非生产环境,某些节点的角色进程运行失败报错,通过窗口化即可添加节点,授权管理等操作,实时监控节点健康信息