centos7脚本离线化安装CDH6.3版本(ansible闪亮登场~~~)

centos7脚本离线化安装CDH6.3版本(ansible闪亮登场~~~)

CDH好不好,毋庸置疑,具体体现在该平台是闭源的。逻辑是这样的:既然敢闭源,那么,说明这平台确实好用,闭源都不怕挣不到钱。当然,作为大数据平台管理平台之一的CDH好像也有底气闭源啦。

软件开或者闭源,都只是它的一种运营模式,在此,不过多评价。 

CDH有三种使用形式,第一,白嫖,功能砍了很多,第二,免费试用期限,免费期内和收费的版本一样功能,过期了砍成白嫖功能。第三,付费版本,所有功能都开放,如何选择,不在本文讨论范围,文章末尾会有给出详细图解。

那么,本文主要是如何自动化的脚本安装CDH6.3版本,在使用脚本前,我想,有些工作还是手动做吧,毕竟脚本不能写的太庞大,臃肿了。

基本的服务器信息概述:

实验环境服务器基本信息概述
IP地址CPU内存硬盘域名
192.168.88.112c4core8G100Ghdp1.com/主机名hdp-1
192.168.88.122c4core8G100Ghdp2.com/主机名hdp-2
192.168.88.132c4core8G100Ghdp3.com/主机名hdp-3

说明:CDH安装的时候非常吃内存,8G是勉强可以跑,推荐是16G,虚拟机设置16G的话,宿主机可能先不行了。硬盘大一些,是怕安装的时候磁盘满造成意外停止。

文件下载地址:

链接:https://pan.baidu.com/s/12wNH03PZhqtSHNIwP8LgfA 
提取码:cdhf 

 

一,CDH脚本安装前的环境搭建工作

(1)关闭防火墙和selinux,以防止不必要的干扰。

关闭防火墙:systemctl  disable firewalld && systemctl stop firewalld

关闭selinux: vim /etc/selinux/config 文件,修改成disabled

(2)ntpd时间服务器的搭建(时间服务器搭建完毕并不能立即使用,需要等待若干分钟,搭建写在脚本内并不合适)

(3)服务器之间的免密登陆

(4)本地仓库的搭建(基础软件安装仓库,一般使用centos7的系统安装ISO格式文件搭建,基础命令指得是gcc,openssl-devel这些基本的软件)

(5)编译安装MySQL5.7

编译安装MySQL也是脚本自动安装,请参照https://blog.csdn.net/alwaysbefine/article/details/118440336安装,并将后续初始化部分执行完成。

(6)进入MySQL,执行以下SQL语句,建立库和用户

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

grant all privileges on scm.* to scm@'localhost' identified by 'scm';

grant all privileges on scm.* to scm@'%' identified by 'scm';

flush privileges;

注意,我这里MySQL的scm用户设置的密码是scm,当然不建议这么简单的密码,请按实际需要更改,脚本内初始化数据库也要做相应更改哦。

(7)执行cdh安装脚本,该脚本放置在主服务器192.168.88.11上。

安装cdh的脚本:

#!/bin/bash
# author zsk_john
NGINX_HOME=`find / -name ansible_offline.tar.gz`
tar zxf $NGINX_HOME
rpm -ivh ansible_offline/* --nodeps
cp -f /root/CDH-offline/ansible-hosts /etc/ansible/hosts
ansible all -m copy -a'src=~/host.txt dest=~/'
ansible all -m copy -a'src=~/freelogin.sh  dest=~/'
ansible all -m shell -a 'bash ~/freelogin.sh'
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/
PATH=.$PATH:$JAVA_HOME/bin
CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar
export CLASSPATH PATH JAVA_HOME
export PATH=./usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/java/jdk1.8.0_181-cloudera//bin:/usr/local/src/cmake/bin:/usr/local/mysql/bin


ansible all -m copy -a 'src=~/CDH-offline/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm  dest=~/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm'
ansible all -m copy -a 'src=~/CDH-offline/enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm  dest=~/enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm'
ansible all -m copy -a 'src=~/CDH-offline/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm  dest=~/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm'
ansible all -m copy -a 'src=~/CDH-offline/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm  dest=~/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm'
ansible all -m yum -a 'state=installed name=MySQL-python,bind-utils,psmisc,cyrus-sasl-plain,cyrus-sasl-gssapi,portmap,lsb,httpd,mod_ssl,openssl-devel,python-psycopg2'
sleep 120
ansible all -m shell -a'rpm -ivh ~/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm'
ansible all -m shell -a'rpm -ivh ~/enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm'
ansible all -m shell -a'rpm -ivh ~/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm'
ansible all -m shell -a'rpm -ivh ~/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm'
ansible all -m shell -a 'systemctl enable cloudera-scm-agent'
ansible all -m shell -a 'systemctl start  cloudera-scm-agent'
ansible all -m copy -a 'src=/etc/cloudera-scm-agent/config.ini dest=/etc/cloudera-scm-agent/config.ini'
ansible all -m shell -a 'mkdir -p /usr/share/java'
ansible all -m copy -a 'src=~/CDH-offline/mysql-connector-java-5.1.47.jar dest=/usr/share/java/mysql-connector-java.jar'
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
systemctl enable cloudera-scm-server
systemctl start cloudera-scm-server
ansible all -m shell -a 'systemctl restart  cloudera-scm-agent'
#初始化数据库,最后那个scm 是前面指定的scm用户密码,这里写成固定的,可根据需要更改哦
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm
systemctl restart cloudera-scm-server
mv ~/CDH-offline/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* /opt/cloudera/parcel-repo/

2021-07-10 17:43:42,768 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
2021-07-10 17:43:42,771 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Discovered parcel on CM server: CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
2021-07-10 17:43:42,771 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Created torrent file: /opt/cloudera/parcel-repo/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.torrent
2021-07-10 17:43:42,778 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Creating single-file torrent for CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel...
2021-07-10 17:43:42,782 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashing data from CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel with 8 threads (3972 pieces)...
2021-07-10 17:43:43,384 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 10% complete
2021-07-10 17:43:43,803 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 20% complete
2021-07-10 17:43:44,284 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 30% complete
2021-07-10 17:43:44,687 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 40% complete
2021-07-10 17:43:45,120 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 50% complete
2021-07-10 17:43:45,532 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 60% complete
2021-07-10 17:43:45,952 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 70% complete
2021-07-10 17:43:46,370 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 80% complete
2021-07-10 17:43:46,766 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 90% complete
2021-07-10 17:43:47,203 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashed 1 file(s) (2082186246 bytes) in 3972 pieces (3972 expected) in 4417.8ms.
2021-07-10 17:43:47,210 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Single-file torrent information:
2021-07-10 17:43:47,210 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Torrent name: CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
2021-07-10 17:43:47,210 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Announced at: Seems to be trackerless
2021-07-10 17:43:47,210 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Created on..: Sat Jul 10 17:43:42 CST 2021

通过以上步骤,我们的CDH的web端应该就已经可以正常工作了。检测是否是正常的有如下方法:

[root@master ~]# netstat -antup | grep 7180
tcp        0      0 0.0.0.0:7180            0.0.0.0:*               LISTEN      1027/java           
tcp        0      0 192.168.0.16:7180       192.168.0.111:54364     ESTABLISHED 1027/java           
[root@master ~]# jps
1027 Main
4117 Jps

可以看到服务端口已开。 

mysql> use scm;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+--------------------------------+
| Tables_in_scm                  |
+--------------------------------+
| audits                         |
| auth_roles                     |
| auth_roles_scopes              |
| client_configs                 |
| client_configs_to_hosts        |
| cluster_activated_releases     |
| cluster_activated_releases_aud |
| cluster_managed_releases       |
| cluster_undistributed_releases |
| clusters                       |
| clusters_aud                   |
| cm_peers                       |
| cm_version                     |
| command_schedules              |
| commands                       |
| commands_detail                |
| config_containers              |
| configs                        |
| configs_aud                    |
| credentials                    |
| data_context_to_base_service   |
| data_contexts                  |
| diagnostics_events             |
| external_accounts              |
| external_accounts_aud          |
| external_mapping_auth_roles    |
| external_mappings              |
| global_settings                |
| host_template_to_role_conf_grp |
| host_templates                 |
| hosts                          |
| hosts_aud                      |
| metrics                        |
| parcel_components              |
| parcels                        |
| process_active_releases        |
| processes                      |
| processes_detail               |
| releases                       |
| releases_aud                   |
| revisions                      |
| role_config_groups             |
| role_config_groups_aud         |
| role_staleness_status          |
| roles                          |
| roles_aud                      |
| schema_version                 |
| services                       |
| services_aud                   |
| snapshot_policies              |
| user_auth_roles                |
| user_roles                     |
| user_settings                  |
| users                          |
+--------------------------------+
54 rows in set (0.00 sec)

 可以看到scm这个数据库有54张表,如果表的数目不对,请删除库重新初始化。

可以看到端口被激活就证明没有问题了。此时,我们的agent也就是客户端并没有做任何改动,这样服务器是认不到的,也就是192.168.88.11是认不到的。因此,需要修改文件   

vim /etc/cloudera-scm-agent/config.ini

[General]
# Hostname of the CM server.
server_host=localhost

# Port that the CM server is listening on.
server_port=7182

更改为:

[General]
# Hostname of the CM server.
server_host=192.168.88.11

# Port that the CM server is listening on.
server_port=7182

然后重启客户端:

在192.168.88.11这个服务器上执行:ansible all -m shell -a 'systemctl restart  cloudera-scm-agent'   这条命令即可。

二,web端的操作

经过 第一大步的操作,如果严格安装执行,应该web端已经可以进入了。

 修改 vim /etc/cloudera-scm-agent/config.ini  将服务器地址指定好

[General]
# Hostname of the CM server.
server_host=192.168.88.11

# Port that the CM server is listening on.
server_port=7182

 

下图是按照CDH了,这里需要注意,前面的免密是为了这一步而做得工作,

在客户端,比如192.168.88.12或者13,我们进入这个目录会看到很多文件:

[root@hdp-2 parcels]# du -sh CDH-6.3.2-1.cdh6.3.2.p0.1605554/
3.0G	CDH-6.3.2-1.cdh6.3.2.p0.1605554/
[root@hdp-2 parcels]# pwd
/opt/cloudera/parcels

最后的检测确认,发现需要修改内核参数如下(3台服务器都修改):

echo never > /sys/kernel/mm/transparent_hugepage/defrag

echo never > /sys/kernel/mm/transparent_hugepage/enabled

永久修改:

/etc/sysctl.conf 文件里添加如下参数:

vm.swappiness=10

 然后在每个服务器上执行命令:sysctl -p 使之生效即可。

解决方法:

删除rm -f /opt/cloudera-manager/cm-5.11.1/lib/cloudera-scm-agent/cm_guid ,重启agent,命令为:systemctl restart cloudera-scm-agent

 所有的建库命令为:

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
grant all privileges on scm.* to scm@'localhost' identified by '密码';
grant all privileges on scm.* to scm@'%' identified by '密码';
grant all privileges on hive.* to hive@'localhost' identified by '密码';
grant all privileges on hive.* to hive@'%' identified by '密码';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
grant all privileges on hue.* to hue@'%' identified by '密码';
grant all privileges on hue.* to hue@'localhost' identified by '密码2';
CREATE DATABASE rm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
grant all privileges on rm.* to rm@'localhost' identified by '密码';
grant all privileges on rm.* to rm@'%' identified by '密码';
flush privileges;

 

  • 1
    点赞
  • 6
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

晚风_END

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值