大数据平台CDH 6.2.0 环境搭建
一、概述
Cloudera版本(Cloudera’s Distribution Including Apache Hadoop,简称“CDH”),基于Web的用户界面,支持大多数Hadoop组件,包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop,简化了大数据平台的安装、使用难度。
因为公司测试环境,因为操作人员多,莫名奇妙出现各种问题。所以在测试环境搭建CDH平台,方便平时开发测试使用。本文记录了本次操作的一些步骤,方便日后查阅。
节点部署
s50 | 192.168.1.50 | |
---|---|---|
s51 | 192.168.1.51 | |
s52 | 192.168.1.52 | |
s53 | 192.168.1.53 |
二、安装部署
2.2.1 基础环境部署
-
关闭防火墙
systemctl stop firewalld.service && systemctl disable firewalld.service
-
关闭SELinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config setenforce 0
-
配置时间同步
chrony既可作时间服务器服务端,也可作客户端。chrony性能比ntp要好很多,且chrony配置简单、管理方便。
-
禁用透明大页面压缩,CDH配置需要
echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled 并将上面的两条命令写入开机自启动/etc/rc.local.
-
优化交换分区
echo "vm.swappiness = 10" >> /etc/sysctl.conf sysctl -p
-
免密登录
ssh-keygen -t rsa & cat id_rsa.pub >> authorized_keys & chmod 700 ~/.ssh chmod 644 ~/.ssh/authorized_keys # 分发秘钥() ssh-copy-id -i s51
-
配置映射关系
问题:在每个节点的etc/hosts文件中添加“127.0.0.1 localhost”映射 ,socket.gaierror: [Errno -2] Name or service not known
解决方案:
127.0.0.1 localhost
配置hosts:
192.168.1.50 s50.online.cn s50 192.168.1.51 s51.online.cn s51 192.168.1.52 s52.online.cn s52 192.168.1.53 s53.online.cn s53
-
配置时间同步
1. 安装ntpdate工具 # yum -y install ntp ntpdate 2. 设置系统时间与网络时间同步 # ntpdate cn.pool.ntp.org 3. 将系统时间写入硬件时间 # hwclock --systohc 4.强制系统时间写入CMOS中防止重启失效 # hwclock -w 或# clock -w
三. 安装 CM 和 CDH
-
配置 Cloudera Manager 仓库(所有节点)
wget https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/ rpm --import https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera
-
安装 CM Server 和 Agent
资料下载地址
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/
主节点(cloudera-manager-daemons, cloudera-manager-agent, cloudera-manager-server )
# yum localinstall cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm -y # yum localinstall cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm -y # yum localinstall cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm -y
从节点 (cloudera-manager-daemons cloudera-manager-agent )
# yum localinstall cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm -y # yum localinstall cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm -y
-
安装数据库
- 主节点安装数据库(s50)
yum install mariadb* -y
-
配置数据库
可选步骤。根据官方推荐的配置,编辑文件/etc/my.cnf,修改成如下内容:
vim /etc/my.cnf [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock transaction-isolation = READ-COMMITTED symbolic-links = 0 key_buffer = 16M key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 max_connections = 550 log_bin=/var/lib/mysql/mysql_binary_log server_id=1 binlog_format = mixed read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M [mysqld_safe] log-error=/var/log/mariadb/mariadb.log pid-file=/var/run/mariadb/mariadb.pid !includedir /etc/my.cnf.d
-
启动服务
systemctl enable mariadb && systemctl restart mariadb
-
配置
/usr/bin/mysql_secure_installation
-
按照下面提示输入。
[...] Enter current password for root (enter for none): OK, successfully used password, moving on... [...] Set root password? [Y/n] Y New password: Re-enter new password: [...] Remove anonymous users? [Y/n] Y [...] Disallow root login remotely? [Y/n] N [...] Remove test database and access to it [Y/n] Y [...] Reload privilege tables now? [Y/n] Y [...] All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB!
-
安装 MySQL JDBC 驱动(所有节点)
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz tar xf mysql-connector-java-5.1.46.tar.gz mkdir -p /usr/share/java/ cd mysql-connector-java-5.1.46 cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
-
为 Cloudera 各软件创建数据库
mysql -uroot -p
执行以下sql (下边数据库配置的时候用到)
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';
-
设置 Cloudera Manager 数据库
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm //接着,输入scm数据库密码(123456)
-
安装CM6.2时,请装官方提供的JDK:oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
1.第一种方式:
因为已经配置好repo仓库所以yum时会到192.168.88.100/cm6.2目录下找到oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm进行安装yum -y install oracle-j2sdk1.8-1.8.0+update181-1.x86_64
默认安装在 /usr/java/jdk1.8.0_181-cloudera
vim /etc/profile 文件中内容如下JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera CLASSPATH=.:$JAVA_HOME/lib/tools.jar PATH=$JAVA_HOME/bin:$PATH export JAVA_HOME CLASSPATH PATH
重新加载profile配置文件,让配置文件生效:source /etc/profile
检查是否已配置好新的JDK:java -version -
安装 CDH
CM安装成功之后,接下来我们就可以通过CM安装CDH的方式构建企业大数据平台。所以首先需要把CDH的parcels包下载到CM主服务器上。同样的,我们为了加速我们的安装,我们可以把需要下载的软件包提前下载下来,也可以创建CDH私有仓库。
下载CDH的软件包 parcels(s50)
# cd /opt/cloudera/parcel-repo
# wget https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
# wget https://archive.cloudera.com/cdh6/6.2.0/parcels/manifest.json
生成一个sha文件。
sha1sum CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel | awk '{ print $1 }' > CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha
修改属主属组
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
-
启动 Cloudera Manager Server
# systemctl start cloudera-scm-server
如果启动中有什么问题,可以查看日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
ps -ef | grep cloudera-scm-server 查看是否启动 systemctl status cloudera-scm-server 显示 Active: active (running)
检查端口是否监听
netstat -lnpt | grep 7180
通过 id:7180/cmf/login 访问CM
-
问题求解
-
提示cloudera安装报错 socket.gaierror: [Errno -2] Name or service not known
1. 确保hosts ,hostname映射正确,以及在hosts文件配置中有127.0.0.1 localhost 2. 删除/usr/bin/host文件即可解决问题
-
启动提示:不良 : 当从 Java 进程查看时该主机的主机名称和规范名称不一致
问题原因是在配置hostname的时候配置了域名,服务器根据/etc/resolv.conf DNF解析是,提示找不到改域名。
处理方式:在每台服务器上添加本地域名解析:(根据上述hosts配置,需要添加如下配置,domain 是本地域名的意思)echo "domain online.cn" >> /etc/resolv.conf
配置完之后重启或者等五分钟缓存时间在重试,问题解决!
3.不良 : 该主机与 Cloudera Manager Server 失去联系的时间过长。 该主机未与 Host Monitor 建立联系。
解决办法:
1. 首先查看该主机NTP服务是否启动:https://www.cnblogs.com/sisimi/p/7698452.html
$ systemctl start ntpd
$ systemctl enable ntpd
$ systemctl status ntpd
如果已经启动,请查看客户端服务是否启动:/opt/cm-5.14.0/etc/init.d/cloudera-scm-agent start
启动之后主机一切正常。
2. 将问题主机卸载之后再重新安装
1. 注意将host 保持一致
2. 删除 rm -rf /usr/bin/host
3. 删除cm_guid
是因为节点上次安装没有成功,所以这个agent的id用的是上一套集群的id。所以需要删除cm_guid文件再次安装。他就能创建新的cm_guid文件了。
解决错误
cd /var/lib/cloudera-scm-agent/
删除掉cm.guid 这个文件。然后重新安装集群即可。
-
初始化 Cloudera Manager
http://192.168.1.50:7180,用户名和密码默认都是admin。
具体初始化请参照(第一篇)参考资料。
搭建完成!
参考资料:
https://xuchao918.github.io/2019/05/31/Centos7%E5%AE%89%E8%A3%85%E5%A4%A7%E6%95%B0%E6%8D%AE%E5%B9%B3%E5%8F%B0CDH-6-2/
https://blog.51cto.com/wzlinux/2321433
https://www.jianshu.com/p/106739236db4
https://blog.csdn.net/wh211212/article/details/78743191
http://www.ishenping.com/ArtInfo/321643.html