Centos7+CDP7.1.1安装部署

组件版本
Cloudera Manager7.1.1
Cloudera Runtime7.1.1.0
OpenJDK1.8.0_252
CentOS7.4.1708
MySQL5.7
IPHostname内存CPU磁盘
10.180.249.2manager.cdp16G161T
10.180.249.3master.cdp16G16500G
10.180.249.4worker.cdp16G16500G

1.搭建步骤

1.1 hosts 配置(所有主机)

cat >> /etc/hosts <<EOF
10.180.249.2 manager.cdp
10.180.249.3 master.cdp
10.180.249.4 worker.cdp
EOF

1.2 配置 SSH 免密(manager.cdp)

ssh-keygen -t rsa # 一路回车
ssh-copy-id manager.cdp
ssh-copy-id master.cdp
ssh-copy-id worker.cdp

1.3 关闭 Selinux(所有主机)

setenforce 0
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/g’ /etc/selinux/config

1.4 关闭防火墙(所有主机)

systemctl stop firewalld
systemctl disable firewalld

1.5 安装 ntpd 服务(所有主机)

yum install -y ntp ntpdate

manager.cdp 节点
vim /etc/ntp.conf
// 注释 server
restrict 10.180.249.254 mask 255.255.254.0 nomodify notrap
server 127.127.1.0
fudge 127.127.1.0 stratum 8
// 注:10.180.249.254 和 255.255.254.0是集群所在网段的网关和子网掩码

master.cdp 节点
vim /etc/ntp.conf
// 注释 server
restrict 10.180.249.254 mask 255.255.254.0 nomodify notrap
server manager.cdp prefer
server 127.127.1.0
fudge 127.127.1.0 stratum 9

worker.cdp 节点
vim /etc/ntp.conf
// 注释 server
server manager.cdp prefer
server master.cdp

在manager启动ntp之后,其余各节点启动ntp服务之前,master和worker节点执行命令,同步manager时间
启动 manager.cdp 节点ntpd服务
systemctl start ntpd
systemctl enable ntpd

在master.cdp、worker.cdp 节点进行时间同步
ntpdate manager.cdp

启动master.cdp、worker.cdp 节点ntpd服务
systemctl start ntpd
systemctl enable ntpd

查看ntp状态
ntpq -p
’*’ 表示当前使用的时钟源,’+’ 表示这些源可作为 NTP 源

1.6 设置swap(所有主机)

cat >> /etc/sysctl.conf <<EOF
vm.swappiness=1
EOF

sysctl vm.swappiness=1

1.7 关闭透明大页(所有主机)

echo ‘echo never > /sys/kernel/mm/transparent_hugepage/defrag’ >> /etc/rc.local
echo ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled’ >> /etc/rc.local

vim /etc/rc.d/rc.local

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
        echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
        echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

# 保存退出之后:
chmod +x /etc/rc.d/rc.local

1.8 设置 limits(所有主机)

cat >> /etc/security/limits.conf <<EOF
hdfs - nofile 32768
mapred - nofile 32768
hbase - nofile 32768
hdfs - noproc 32768
mapred - noproc 32768
hbase - noproc 32768
EOF

1.9 安装MySQL(manager.cdp)

下载yum源的rpm包:

wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm

安装rpm包:

rpm -Uvh mysql57-community-release-el7-11.noarch.rpm

检查有mysql源:

yum repolist enabled | grep mysql

在这里插入图片描述
安装MySQL5.7:

yum install -y mysql-community-server

首先启动MySQL:

systemctl start mysqld
systemctl status mysqld

MySQL开始正常运行,不过要想进入MySQL还得先找出此时root用户的密码,通过如下命令可以在日志文件中找出密码:

grep “password” /var/log/mysqld.log

修改root默认密码:

mysql -uroot -p
set global validate_password_policy=LOW;
set password=password(“bigdata123”);

1.9.1 启动并配置MySQL

停止Mysql服务:

systemctl stop mysqld

将旧的InnoDB日志/var/lib/mysql/ib_logfile0和/var/lib/mysql/ib_logfile1移动到备份目录:

[root@manager ~]# pwd
/root
[root@manager ~]# mkdir mysql_backup
[root@manager ~]# mv /var/lib/mysql/ib_logfile* /root/mysql_backup/

然后修改mysql的配置文件/etc/my.cnf:

vi /etc/my.cnf

# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html

[mysqld]
#设置编码为utf-8
character_set_server=utf8
init_connect='SET NAMES utf8'
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

key_buffer_size = 32M                                                                                                        
max_allowed_packet = 32M                                                                                                     
thread_stack = 256K                                                                                                          
thread_cache_size = 64                                                                                                       
query_cache_limit = 8M                                                                                                       
query_cache_size = 64M                                                                                                       
query_cache_type = 1                                                                                                         
                                                                                                                             
max_connections = 550                                                                                                        
#expire_logs_days = 10                                                                                                       
#max_binlog_size = 100M                

#log_bin should be on a disk with enough free space.                                                                         
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your                                                 
#system and chown the specified folder to the mysql user.                                                                    
log_bin=/var/lib/mysql/mysql_binary_log                                                                                      

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

设置mysql自启动:

systemctl enable mysqld
systemctl start mysqld

配置数据库:

sudo /usr/bin/mysql_secure_installation

[root@manager ~]# sudo /usr/bin/mysql_secure_installation

Securing the MySQL server deployment.

Enter password for user root: 
The 'validate_password' plugin is installed on the server.
The subsequent steps will run with the existing configuration
of the plugin.
Using existing password for root.

Estimated strength of the password: 50 
Change the password for root ? ((Press y|Y for Yes, any other key for No) : N

 ... skipping.
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.


Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : n

 ... skipping.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.


Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
 - Dropping test database...
Success.

 - Removing privileges on test database...
Success.

Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.

All done!

1.9.2 创建数据库

mysql -uroot -p
set global validate_password_policy=LOW;

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'bigdata123';

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'bigdata123';

CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'bigdata123';

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'bigdata123';

CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'bigdata123';

CREATE DATABASE ranger DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON ranger.* TO 'rangeradmin'@'%' IDENTIFIED BY 'bigdata123';

CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'bigdata123';

CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'bigdata123';

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'bigdata123';
Refresh;

1.10 安装jdbc驱动(所有节点)

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java
tar zxvf mysql-connector-java-5.1.46.tar.gz

scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar master.cdp:/usr/share/java/mysql-connector-java.jar

scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar worker.cdp:/usr/share/java/mysql-connector-java.jar

1.11 安装JDK(所有节点)

yum -y install java-1.8.0-openjdk.x86_64 java-1.8.0-openjdk-headless.x86_64 java-1.8.0-openjdk-devel.x86_64

1.12 配置本地源(manager.cdp)

下载CM7.1.1的安装包,地址为(manager.cdp):

https://archive.cloudera.com/cm7/7.1.1/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-7.1.1-3274282.el7.x86_64.rpm
https://archive.cloudera.com/cm7/7.1.1/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-7.1.1-3274282.el7.x86_64.rpm
https://archive.cloudera.com/cm7/7.1.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-7.1.1-3274282.el7.x86_64.rpm
https://archive.cloudera.com/cm7/7.1.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-7.1.1-3274282.el7.x86_64.rpm
https://archive.cloudera.com/cm7/7.1.1/redhat7/yum/RPMS/x86_64/enterprise-debuginfo-7.1.1-3274282.el7.x86_64.rpm
https://archive.cloudera.com/cm7/7.1.1/redhat7/yum/RPMS/x86_64/openjdk8-8.0+232_9-cloudera.x86_64.rpm
https://archive.cloudera.com/cm7/7.1.1/redhat7/yum/RPM-GPG-KEY-cloudera
https://archive.cloudera.com/cm7/7.1.1/allkeys.asc

批量下载命令为(manager.cdp):

wget -nd -r -l1 --no-parent https://archive.cloudera.com/cm7/7.1.1/redhat7/yum/RPMS/x86_64/  
wget https://archive.cloudera.com/cm7/7.1.1/redhat7/yum/RPM-GPG-KEY-cloudera
wget https://archive.cloudera.com/cm7/7.1.1/allkeys.asc

下载 Cloudera Runtime7.1.1的安装包,地址为(manager.cdp):

https://archive.cloudera.com/cdh7/7.1.1.0/parcels/CDH-7.1.1-1.cdh7.1.1.p0.3266817-el7.parcel
https://archive.cloudera.com/cdh7/7.1.1.0/parcels/CDH-7.1.1-1.cdh7.1.1.p0.3266817-el7.parcel.sha256
https://archive.cloudera.com/cdh7/7.1.1.0/parcels/manifest.json

批量下载命令为(manager.cdp):

wget -nd -r -l1 --no-parent https://archive.cloudera.com/cdh7/7.1.1.0/parcels/

安装工具包(manager.cdp):

yum install httpd createrepo -y
systemctl start httpd
systemctl enable httpd
systemctl status httpd

mkdir -p /var/www/html/cm7/7.1.1/
cd /var/www/html/cm7/7.1.1/
上传rpm包
createrepo .
在这里插入图片描述
在这里插入图片描述

mkdir -p /var/www/html/cdh7/7.1.1.0
cd /var/www/html/cdh7/7.1.1.0
上传rpm包
在这里插入图片描述
在这里插入图片描述

1.13 制作Cloudera Manager的repo源(所有节点)

cd /etc/yum.repos.d
vim cm7.repo
[cm7]
name=cm7
baseurl=http://manager.cdp/cm7/7.1.1
gpgcheck=0
enabled=1

2.安装 Cloudera Manager Server

2.1 通过yum安装Cloudera Manager Server(manager.cdp)

yum -y install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

2.2 初始化数据库

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm bigdata123
在这里插入图片描述

2.3 启动Cloudera Manager Server

systemctl start cloudera-scm-server.service
systemctl status cloudera-scm-server.service

2.4 检查服务

sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

当您看到此日志条目时,Cloudera Manager管理控制台已准备就绪:

INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
在这里插入图片描述

2.5 检查端口是否监听

netstat -nltp | grep 7180
在这里插入图片描述

3.访问CM 创建集群

通过 http://manager.cdp:7180/ 访问CM
在这里插入图片描述
使用admin/admin用户登录系统
在这里插入图片描述

3.1 CDH集群安装向导

  1. 使用admin/admin登录到CM:
    在这里插入图片描述
    这时有两个选项,一个是上传企业订阅的license,一个是使用60天试用,试用到期后无法访问Cloudera Manager的管理控制台。
    我们这里先选择60天试用。弹出试用协议,如下图所示,选择继续,进行下一步:
    在这里插入图片描述
    选择继续,进行下一步:
    在这里插入图片描述

  2. 点击“继续”,输入集群的名称,也可以使用默认的”Cluster 1”:
    在这里插入图片描述

  3. 输入主机ip或者名称,点击搜索找到主机后点击继续:
    在这里插入图片描述

  4. 点击继续:
    在这里插入图片描述

  5. “CDH and other software”中选择“Parcel Repositories & Network Settings”,点击“ 更多选项”,点击“-”删除其它所有地址,输入http://manager.cdp/cdh7/7.1.1.0/,点击“保存更改”:
    在这里插入图片描述
    点击Save & Verify Configuration,验证通过,点击关闭:
    在这里插入图片描述
    在这里插入图片描述

  6. 点击“继续”,进入下一步安装jdk. 选择Install a Cloudera-provided version of OpenJDK,点击继续:
    在这里插入图片描述

  7. 点击“继续”,进入下一步配置ssh账号密码,确保所有机器的root密码都一致,然后输入root的密码即可:
    在这里插入图片描述

  8. 点击“继续”,进入下一步,安装Cloudera Manager相关到各个节点:
    在这里插入图片描述
    等待agent安装完毕后,自动跳转到下一步开始安装parcel

  9. 点击“继续”,进入下一步安装cdh到各个节点:
    在这里插入图片描述

  10. 自动进入下一步主机检查和网络检查,确保所有检查项均通过:
    在这里插入图片描述

需要手工点击进行网络性能和主机检查:
在这里插入图片描述

如果有错误或者黄色警告,查看“显示检查器结果”,并逐项解决,然后“重新运行”检查,直到所有的检查都通过,否则没办法点击继续下一步:
在这里插入图片描述
点击完成进入服务安装向导。

3.2 集群设置安装向导

  1. 选择需要安装的服务:
    在这里插入图片描述
    自定义服务中可以看到所有组件,可以根据自己的需求来选择:
    在这里插入图片描述

  2. 点击“继续”,进入集群角色分配,一台机器作为管理节点,另外三台机器作为DataNode:
    在这里插入图片描述
    注意: Cloudera Management Service中的Activity Monitor现在已经基本上不用,可以不安装该服务。Telemetry Publisher是遥感服务,用于Workload XM通信,如果没有计划使用Workload XM,则不需要安装该服务。ZooKeeper至少安装3节点,需要为奇数节点数。

  3. 点击“继续”,进入下一步,测试数据库连接:
    在这里插入图片描述
    测试都成功后才能点击继续:
    在这里插入图片描述

  4. 点击“继续”,进入参数设置,此处使用默认参数,根据实际情况进行目录修改:
    在这里插入图片描述

  5. 点击“继续”,进入各个服务启动:
    在这里插入图片描述

  6. 安装成功,点击继续:
    在这里插入图片描述

  7. 安装成功后进入home管理界面,系统会自动恢复成没有错误的状态:
    在这里插入图片描述

4.组件版本检查

通过Hosts->All Hosts进入All Hosts页面,然后执行Inspect All Hosts:
在这里插入图片描述
执行完成后点击Show Inspector Results查看结果:
在这里插入图片描述
检查结果:
在这里插入图片描述
版本汇总:
在这里插入图片描述

参考:
https://docs.cloudera.com/cloudera-manager/7.1.1/index.html
https://cloud.tencent.com/developer/article/1638154

评论 23
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

@TangXin

路漫漫其修远兮,吾将上下而求索

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值