Linux运维之RHCS集群套件——ISCSI/SCSI实现存储共享、CLVM/GFS2集群文件系统部署

前言：

此内容承接上一篇博客：
Linux运维之RHCS集群套件——Luci/Ricci实现Web高可用集群

一、搭建环境

两台redht6.5的虚拟机。

• server1 配置luci集群管理界面，用来配置和管理集群。
• server1、server2 集群节点，配置ricci。
• server3 配置共享存储，增添一块共享磁盘

二、搭建步骤

1、server3上安装共享存储所需要的软件（高级yum源）

yum install scsi-* -y

2、在server1、server2上安装客户端iscsi

yum install iscsi-* -y

3、在server3上配置共享策略，开启服务，查看

vim /etc/tgt/targets.conf 

<target iqn.2019-04.com.example:server.target1>
    backing-store /dev/sdb		#确定共享的磁盘
</target>

/etc/init.d/tgtd start	#启动服务

tgt-admin -s

4、在server1、server2上查看并激活共享的存储

iscsiadm -m discovery -t st -p 172.25.75.3		#查看共享存储
iscsiadm -m node -l			#激活共享存储
fdisk -l

5、在server1、server2上下载mysql

[root@server1 ~]# yum install mysql-server -y
[root@server2 ~]# yum install mysql-server -y

6、格式化共享设备（只在一台设备上操作）

[root@server1 ~]# mkfs.ext4 /dev/sdb

7、在server1、server2上更改mysql目录的权限

server1:

[root@server1 ~]# mount /dev/sdb /var/lib/mysql/
[root@server1 ~]# ll -d /var/lib/mysql/
drwxr-xr-x 3 root root 4096 Apr 22 14:20 /var/lib/mysql/
[root@server1 ~]# chown mysql.mysql /var/lib/mysql/
[root@server1 ~]# ll -d /var/lib/mysql/
drwxr-xr-x 3 mysql mysql 4096 Apr 22 14:20 /var/lib/mysql/
[root@server1 ~]# 

8、检测服务能否正常开启

[root@server1 ~]# /etc/init.d/mysqld start
[root@server1 ~]# /etc/init.d/mysqld stop
[root@server1 ~]# umount /dev/sdb

9、将apache的单线程关掉，并且把apache服务组资源停掉

将Run Exclusive的勾去掉，并且重新submit。

也可以用命令：

clusvcadm -d apache

10、浏览器配置（先ip后挂载再httpd服务）

1、在Failover Domains中添加dbfail

2、在Resources中添加IP Address,File System,Script资源。

3、在Service Groups中添加sql服务组

4、开启sql

11、测试

[root@server2 ~]# mysql -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.71 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SHOW DATABASES;
+---------------------+
| Database            |
+---------------------+
| information_schema  |
| #mysql50#lost+found |
| mysql               |
| test                |
+---------------------+
4 rows in set (0.00 sec)

mysql> USE mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> QUIT
Bye

clusvcadm -e apache		#启动apache服务组
clusvcadm -r apache -m server2	#将apache服务组切到server2
clusvcadm -d apache		#关闭apache服务组

三、CLVM/GFS2集群文件系统部署

前言：

首先将上个实验的apache、sql服务组的资源停掉

[root@server1 ~]# clusvcadm -d apache
Local machine disabling service:apache...Success
[root@server1 ~]# clusvcadm -d sql
Local machine disabling service:sql...Success
[root@server1 ~]# clustat 
Cluster Status for HA @ Mon Apr 22 15:31:15 2019
Member Status: Quorate

 Member Name                                     ID   Status
 ------ ----                                     ---- ------
 server1                                             1 Online, Local, rgmanager
 server2                                             2 Online, rgmanager

 Service Name                           Owner (Last)                           State         
 ------- ----                           ----- ------                           -----         
 service:apache                         (server1)                              disabled      
 service:sql                            (server2)                              disabled      
[root@server1 ~]# 

1、设置CLVM服务开机自启动

lvmconf --enable-cluster
cat /etc/lvm/lvm.conf | grep locking_type		#为3则自启动

2、划分物理卷、逻辑卷组、逻辑卷

pvcreate /dev/sdb
vgcreate clustervg /dev/sdb
lvcreate -L 4G -n demo clustervg
lvs

3、格式化逻辑卷为GFS2集群文件系统

[root@server1 ~]# mkfs.gfs2 -t HA:mygfs2 -p lock_dlm -j 2 /dev/clustervg/demo 
This will destroy any data on /dev/clustervg/demo.
It appears to contain: symbolic link to `../dm-2'

Are you sure you want to proceed? [y/n] y

Device:                    /dev/clustervg/demo
Blocksize:                 4096
Device Size                8.00 GB (2097152 blocks)
Filesystem Size:           8.00 GB (2097150 blocks)
Journals:                  2
Resource Groups:           32
Locking Protocol:          "lock_dlm"
Lock Table:                "HA:mygfs2"
UUID:                      7c3da294-a84f-2f87-3592-c04263b70fa2

 mkfs.gfs2 
 	-j #: 指定日志区域的个数，有几个就能够被几个节点所挂载； 
 	-J #: 指定日志区域的大小，默认为128MB; 
 	-p {lock_dlm|lock_nolock}：所使用的锁协议名称，集群需要使用lock_dlm； 
 	-t <name>: 锁表的名称
 			   格式为clustername:fsname, 
 			   clustername为当前节点所在的集群的名称
 			   fsname文件系统名称，自定义，要在当前集群惟一，这也叫锁表名称，表明是分布式锁的范围。

[root@server1 ~]# gfs2_tool sb /dev/clustervg/demo all
mh_magic = 0x01161970
mh_type = 1
mh_format = 100
sb_fs_format = 1801
sb_multihost_format = 1900
sb_bsize = 4096
sb_bsize_shift = 12
no_formal_ino = 2
no_addr = 23
no_formal_ino = 1
no_addr = 22
sb_lockproto = lock_dlm
sb_locktable = HA:mygfs2
uuid = 7c3da294-a84f-2f87-3592-c04263b70fa2

4、挂载设备，更改目录的权限

mount /dev/clustervg/demo /var/lib/mysql/
ll -d /var/lib/mysql/
chown mysql.mysql /var/lib/mysql/
blkid

5、测试多点挂载

server1：

mount /dev/clustervg/demo /var/lib/mysql/
[root@server1 mysql]# pwd
/var/lib/mysql
[root@server1 mysql]# cp /etc/passwd .
[root@server1 mysql]# ls
ibdata1  ib_logfile0  ib_logfile1  mysql  passwd  test
[root@server1 mysql]# 

server2:

[root@server2 ~]# cd /var/lib/mysql/
[root@server2 mysql]# ls
ibdata1  ib_logfile0  ib_logfile1  mysql  passwd  test
[root@server2 mysql]# 

6、将组文件永久挂载在/var/lib/mysql

vim /etc/fstab

UUID="7c3da294-a84f-2f87-3592-c04263b70fa2"     /var/lib/mysql  gfs2    _netdev 0 0		#UUID可以通过blkid查看
mount -a

7、更改浏览器设置

server1、server2：

更改之前先卸载逻辑卷：
	umount /dev/clustervg/demo

1、先在Service Groups—sql中删除 Filesystem：

删除后，一定要Submit。

2、删除集群资源中的dbdata

3、重新添加集群资源

4、在Service Groups——sql中添加更改后的gfs2的dbdata：

8、测试

启动sql服务组