mysql服务器的HA集群之corosync+drbd+pacemaker实现

最新推荐文章于 2021-02-07 07:14:23 发布

jiangchuanze

最新推荐文章于 2021-02-07 07:14:23 发布

阅读量699

点赞数

本文属转载，请尊重他人的劳动成果，转载时请务必以超链接形式标明文章原始出处、作者信息和本声明。否则将追究法律责任。 http://xjzhujunjie.blog.51cto.com/3582724/886317

案例应用：红帽企业群集和存储管理之

mysql服务器的HA集群之corosync+drbd+pacemaker实现

案例应用企业需求：

某公司需要通过HA实现服务器的高可用性，即通过corosync+drbd+

pacemaker实现mysql服务器的高可用集群。

案例应用拓扑图：

案例应用实现步骤：

一．准备工作：

1.1 修改node1.junjie.com主机名，IP地址和系统时间

[root@node1 ~]# hostname
node1.junjie.com
[root@node1 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1.junjie.com
[root@node1 ~]# hwclock -s
[root@node1 ~]# date
Tue Feb 7 20:24:22 CST 2012
[root@node1 ~]#
[root@node1 ~]# setup

[root@node1 ~]# service network restart
Shutting down interface eth0: [ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: [ OK ]
[root@node1 ~]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:0C:29:AE:83:D1
inet addr:192.168.101.81 Bcast:192.168.101.255 Mask:255.255.255.0
1.2 修改node2.junjie.com主机名，IP地址和系统时间
[root@node2 ~]# hostname
node2.junjie.com
[root@node2 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node2.junjie.com
[root@node2 ~]# hwclock -s
[root@node2 ~]# date
Tue Feb 7 20:25:49 CST 2012
[root@node2 ~]#
[root@node2 ~]# setup

[root@node2 ~]# service network restart
Shutting down interface eth0: [ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: [ OK ]
[root@node2 ~]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:0C:29:D1:D4:32
inet addr:192.168.101.82 Bcast:192.168.101.255 Mask:255.255.255.0
1.3 在node1和node2上配置hosts文件和ssh密钥信息
有利于以后在一个节点对另一节点直接操作。
1.3.1 在node1上修改/etc/hosts文件
[root@node1 ~]# echo "192.168.101.81 node1.junjie.com node1" >>/etc/hosts
[root@node1 ~]# echo "192.168.101.82 node2.junjie.com node2" >>/etc/hosts
1.3.1 在node1上修改/etc/hosts文件
[root@node2 ~]# echo "192.168.101.81 node1.junjie.com node1" >>/etc/hosts
[root@node2 ~]# echo "192.168.101.82 node2.junjie.com node2" >>/etc/hosts
1.3.3 在node1上配置ssh密钥信息
[root@node1 ~]# ssh-keygen -t rsa #一直输入空格
[root@node1 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@node2.junjie.com
1.3.4 在node2上配置ssh密钥信息
[root@node2 ~]# ssh-keygen -t rsa #一直输入空格
[root@node2 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@node1.junjie.com
[root@node2 ~]# ssh node1 'ifconfig' #将看到node2上的信息
The authenticity of host 'node1 (192.168.101.81)' can't be established.
RSA key fingerprint is d4:f1:06:3b:a0:81:fd:85:65:20:9e:a1:ee:46:a6:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1' (RSA) to the list of known hosts.
eth0 Link encap:Ethernet HWaddr 00:0C:29:AE:83:D1
inet addr:192.168.101.81 Bcast:192.168.101.255 Mask:255.255.255.0

1.4 下载相关软件包：（这里我将下载的软件包放在/root/ha/下面了，共18个）

资源的下载地址：http://down.51cto.com/data/402802

[root@node1 ~]# cd ha/
[root@node1 ha]#
cluster-glue-1.0.6-1.6.el5.i386.rpm
cluster-glue-libs-1.0.6-1.6.el5.i386.rpm
corosync-1.2.7-1.1.el5.i386.rpm
corosynclib-1.2.7-1.1.el5.i386.rpm
drbd83-8.3.8-1.el5.centos.i386.rpm
heartbeat-3.0.3-2.3.el5.i386.rpm
heartbeat-libs-3.0.3-2.3.el5.i386.rpm
kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
ldirectord-1.0.1-1.el5.i386.rpm
libesmtp-1.0.4-5.el5.i386.rpm
mysql-5.5.15-linux2.6-i686.tar.gz
openais-1.1.3-1.6.el5.i386.rpm
openaislib-1.1.3-1.6.el5.i386.rpm
pacemaker-1.1.5-1.1.el5.i386.rpm
pacemaker-cts-1.1.5-1.1.el5.i386.rpm
pacemaker-libs-1.1.5-1.1.el5.i386.rpm
perl-TimeDate-1.16-5.el5.noarch.rpm
resource-agents-1.0.4-1.1.el5.i386.rpm
这里不需要ldirectord-1.0.1-1.el5.i386.rpm，故将其删除
[root@node1 ha]# rm ldirectord-1.0.1-1.el5.i386.rpm
rm: remove regular file `ldirectord-1.0.1-1.el5.i386.rpm'? y
[root@node1 ha]# ssh node2 'mkdir /root/ha'
[root@node1 ha]# scp *.rpm node2:/root/ha/#将相关软件包移动到node2上
[root@node1 ha]# scp *.rpm node2:/root/ha/
cluster-glue-1.0.6-1.6.el5.i386.rpm 100% 265KB 265.0KB/s 00:00
cluster-glue-libs-1.0.6-1.6.el5.i386.rpm 100% 130KB 130.1KB/s 00:00
corosync-1.2.7-1.1.el5.i386.rpm 100% 166KB 166.1KB/s 00:00
corosynclib-1.2.7-1.1.el5.i386.rpm 100% 155KB 154.8KB/s 00:00
drbd83-8.3.8-1.el5.centos.i386.rpm 100% 217KB 216.7KB/s 00:00
heartbeat-3.0.3-2.3.el5.i386.rpm 100% 162KB 161.7KB/s 00:00
heartbeat-libs-3.0.3-2.3.el5.i386.rpm 100% 283KB 282.8KB/s 00:00
kmod-drbd83-8.3.8-1.el5.centos.i686.rpm 100% 123KB 123.0KB/s 00:00
libesmtp-1.0.4-5.el5.i386.rpm 100% 59KB 59.0KB/s 00:00
openais-1.1.3-1.6.el5.i386.rpm 100% 202KB 202.2KB/s 00:00
openaislib-1.1.3-1.6.el5.i386.rpm 100% 92KB 92.4KB/s 00:00
pacemaker-1.1.5-1.1.el5.i386.rpm 100% 778KB 778.1KB/s 00:00
pacemaker-cts-1.1.5-1.1.el5.i386.rpm 100% 203KB 203.1KB/s 00:00
pacemaker-libs-1.1.5-1.1.el5.i386.rpm 100% 324KB 324.2KB/s 00:00
perl-TimeDate-1.16-5.el5.noarch.rpm 100% 32KB 32.1KB/s 00:00
resource-agents-1.0.4-1.1.el5.i386.rpm 100% 380KB 379.5KB/s 00:00
[root@node1 ha]# scp mysql-5.5.15-linux2.6-i686.tar.gz node2:/root/ha/
mysql-5.5.15-linux2.6-i686.tar.gz 100% 155MB 10.3MB/s 00:15
1.5 配置本地yum数据库

[root@node1 ha]#
[root@node1 ha]# mkdir /mnt/cdrom/
[root@node1 ha]# mount /dev/cdrom /mnt/cdrom/
mount: block device /dev/cdrom is write-protected, mounting read-only
[root@node1 ha]# yum list all
[root@node1 ha]#
[root@node1 ha]# scp /etc/yum.repos.d/server.repo node2:/etc/yum.repos.d/
server.repo 100% 647 0.6KB/s 00:00
[root@node1 ha]#
[root@node1 ha]# ssh node2 'mkdir /mnt/cdrom/'
[root@node1 ha]# ssh node2 'mount /dev/cdrom /mnt/cdrom/'
[root@node1 ha]# ssh node2 'yum list all'
1.6 在node1上新增磁盘
[root@node1 ha]# cd
[root@node1 ~]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1318 10482412+ 83 Linux
/dev/sda3 1319 1579 2096482+ 82 Linux swap / Solaris
[root@node1 ~]# fdisk /dev/sda
p/n/p//+1000M/p/w
[root@node1 ~]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1318 10482412+ 83 Linux
/dev/sda3 1319 1579 2096482+ 82 Linux swap / Solaris
/dev/sda4 1580 1702 987997+ 83 Linux
[root@node1 ~]# partprobe /dev/sda
[root@node1 ~]# cat /proc/partitions
major minor #blocks name

8 0 20971520 sda
8 1 104391 sda1
8 2 10482412 sda2
8 3 2096482 sda3
8 4 987997 sda4
[root@node1 ~]#
1.7 在node2上新增磁盘
[root@node2 ha]# cd
[root@node2 ~]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1318 10482412+ 83 Linux
/dev/sda3 1319 1579 2096482+ 82 Linux swap / Solaris
[root@node2 ~]# fdisk /dev/sda
p/n/p//+1000M/p/w
[root@node2 ~]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1318 10482412+ 83 Linux
/dev/sda3 1319 1579 2096482+ 82 Linux swap / Solaris
/dev/sda4 1580 1702 987997+ 83 Linux
[root@node2 ~]# partprobe /dev/sda
[root@node2 ~]# cat /proc/partitions
major minor #blocks name

8 0 20971520 sda
8 1 104391 sda1
8 2 10482412 sda2
8 3 2096482 sda3
8 4 987997 sda4
[root@node2 ~]#

二、DRBD安装配置步骤

在node1和node2做以下操作：

我下载的软件包是：（我放在/root/ha/下了）

drbd83-8.3.8-1.el5.centos.i386.rpm

kmod-drbd83-8.3.8-1.el5.centos.i686.rpm

2.1、安装DRBD 套件

[root@node1 ~]# cd ha/
[root@node1 ha]# ls
[root@node1 ha]# yum localinstall -y drbd83-8.3.8-1.el5.centos.i386.rpm kmod-drbd83-8.3.8-1.el5.centos.i686.rpm –nogpgcheck
...
Installed:
drbd83.i386 0:8.3.8-1.el5.centos kmod-drbd83.i686 0:8.3.8-1.el5.centos

Complete!
[root@node1 ha]# cd

[root@node2 ~]# cd ha/
[root@node2 ha]# ls
[root@node2 ha]# yum localinstall -y drbd83-8.3.8-1.el5.centos.i386.rpm kmod-drbd83-8.3.8-1.el5.centos.i686.rpm –nogpgcheck
...
Installed:
drbd83.i386 0:8.3.8-1.el5.centos kmod-drbd83.i686 0:8.3.8-1.el5.centos

Complete!
[root@node2 ha]# cd
2.2、加载DRBD 模块

[root@node1 ~]# modprobe drbd
[root@node1 ~]# lsmod |grep drbd
drbd 228528 0
[root@node1 ~]#
[root@node1 ~]# ssh node2 'modprobe drbd'
[root@node1 ~]# ssh node2 'lsmod |grep drbd'
drbd 228528 0
[root@node1 ~]#
2.3、修改配置文件

drbd.conf配置文件DRBD运行时,会读取一个配置文件/etc/drbd.conf.这个文件里描述了DRBD设备与硬盘分区的映射关系
2.3.1 在node1上以下配置
[root@node1 ~]# cp /usr/share/doc/drbd83-8.3.8/drbd.conf /etc/
cp: overwrite `/etc/drbd.conf'? y
[root@node1 ~]# cat /etc/drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";
include "drbd.d/*.res";
[root@node1 ~]# cd /etc/drbd.d/
[root@node1 drbd.d]# ll
total 4
-rwxr-xr-x 1 root root 1418 Jun 4 2010 global_common.conf
[root@node1 drbd.d]# cp global_common.conf global_common.conf.bak
#修改全局配置文件（说明略）
[root@node1 drbd.d]# vim global_common.conf
global {
usage-count yes;
}
common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
}
startup {
wfc-timeout 120;
degr-wfc-timeout 100;
}
disk {
on-io-error detach;
}
net {
cram-hmac-alg "sha1";
shared-secret "mydrbd123";
}
syncer {
rate 100M;
}
}
#修改资源配置文件（说明略）
[root@node1 drbd.d]# vim /etc/drbd.d/mysql.res

resource mysql {
on node1.junjie.com {
device /dev/drbd0;
disk /dev/sda4;
address 192.168.101.81:7898;
meta-disk internal;
}
on node2.junjie.com {
device /dev/drbd0;
disk /dev/sda4;
address 192.168.101.82:7898;
meta-disk internal;
}
}
2.3.2 复制配置到node2上：
[root@node1 drbd.d]# scp /etc/drbd.conf node2:/etc/
drbd.conf 100% 133 0.1KB/s 00:00
[root@node1 drbd.d]# scp /etc/drbd.d/* node2:/etc/drbd.d/
global_common.conf 100% 682 0.7KB/s 00:00
global_common.conf.bak 100% 1418 1.4KB/s 00:00
mysql.res 100% 468 0.5KB/s 00:00
2.4、检测配置文件, 创建nfs 的资源

//分别在node1和node2上初始化定义的mysql的资源
//检测配置文件（两次执行如下命令)
在node1上：
[root@node1 drbd.d]# drbdadm adjust mysql
--== Thank you for participating in the global usage survey ==--
The server's response is:

0: Failure: (119) No valid meta-data signature found.

==> Use 'drbdadm create-md res' to initialize meta-data area. <==

Command 'drbdsetup 0 disk /dev/sda4 /dev/sda4 internal --set-defaults --create-device --on-io-error=detach' terminated with exit code 10

[root@node1 drbd.d]# drbdadm adjust mysql
drbdsetup 0 show:5: delay-probe-volume 0k => 0k out of range [4..1048576]k.
[root@node1 drbd.d]# drbdadm create-md mysql
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
[root@node1 drbd.d]# ll /dev/drbd0
brw-r----- 1 root disk 147, 0 Feb 7 20:44 /dev/drbd0
在node2上：
[root@node2 ~]# drbdadm create-md mysql

--== Thank you for participating in the global usage survey ==--
The server's response is:

Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.

[root@node2 ~]# ll /dev/drbd0
brw-rw---- 1 root root 147, 0 Feb 7 20:47 /dev/drbd0
2.5 启动DRBD服务，查看DRBD状态

[root@node1 drbd.d]# service drbd start
Starting DRBD resources: [
mysql
Found valid meta data in the expected location, 1011703808 bytes into /dev/sda4.
d(mysql) s(mysql) n(mysql) ]......

[root@node2 ~]# service drbd start
Starting DRBD resources: [
mysql
Found valid meta data in the expected location, 1011703808 bytes into /dev/sda4.
d(mysql) s(mysql) n(mysql) ].

[root@node1 drbd.d]# service drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
m:res cs ro ds p mounted fstype
0:mysql Connected Secondary/Secondary Inconsistent/Inconsistent C
[root@node1 drbd.d]# drbd-overview
0:mysql Connected Secondary/Secondary Inconsistent/Inconsistent C r----

[root@node2 ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
m:res cs ro ds p mounted fstype
0:mysql Connected Secondary/Secondary Inconsistent/Inconsistent C
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary Inconsistent/Inconsistent C r----
2.6 设置DRBD的主节点：

从上面的信息中可以看出此时两个节点均处于Secondary状态。于是，我们接下来需要将其中一个节点设置为Primary，这里将node1设置为主节点，故要在node1上执行如下命令：可以看到文件同步过程。
[root@node1 drbd.d]# drbdadm -- --overwrite-data-of-peer primary mysql

[root@node1 drbd.d]# drbd-overview
0:mysql SyncSource Primary/Secondary UpToDate/Inconsistent C r----
[===>................] sync'ed: 23.6% (761048/987928)K delay_probe: 20
[root@node1 drbd.d]# drbd-overview
0:mysql SyncSource Primary/Secondary UpToDate/Inconsistent C r----
[========>...........] sync'ed: 47.6% (522296/987928)K delay_probe: 42
[root@node1 drbd.d]#
[root@node1 drbd.d]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 drbd.d]#
[root@node1 drbd.d]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
ns:987928 nr:0 dw:0 dr:987928 al:0 bm:61 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
[root@node1 drbd.d]#

[root@node2 ~]# drbd-overview
0:mysql SyncTarget Secondary/Primary Inconsistent/UpToDate C r----
[=====>..............] sync'ed: 32.7% (670680/987928)K queue_delay: 0.1 ms
[root@node2 ~]# drbd-overview
0:mysql SyncTarget Secondary/Primary Inconsistent/UpToDate C r----
[===========>........] sync'ed: 61.6% (384536/987928)K queue_delay: 11.6 ms
[root@node2 ~]#
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Primary UpToDate/UpToDate C r----
[root@node2 ~]#
[root@node2 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
ns:0 nr:987928 dw:987928 dr:0 al:0 bm:61 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
注：Primary/Secondary 说明当前节点为主节点；Secondary/Primary 说明当前节点为从节点。使用：watch -n 1 'cat /proc/drbd'查看同步过程！
2.7 创建文件系统（只可以在primary节点上进行，这里在node1上创建）

[root@node1 drbd.d]# mkfs -t ext3 /dev/drbd0
[root@node1 drbd.d]# mkdir -pv /mnt/mysqldata
mkdir: created directory `/mnt/mysqldata'
[root@node1 drbd.d]# ssh node2 'mkdir -pv /mnt/mysqldata'
mkdir: created directory `/mnt/mysqldata'
[root@node1 drbd.d]# mount /dev/drbd0 /mnt/mysqldata/
[root@node1 drbd.d]#
[root@node1 drbd.d]# cd /mnt/mysqldata/
[root@node1 mysqldata]# ll
total 16
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node1 mysqldata]# echo "123" >f1
[root@node1 mysqldata]# touch f2
[root@node1 mysqldata]# ll
total 20
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node1 mysqldata]#
2.8将node2设置为primary节点

[root@node1 mysqldata]# cd
[root@node1 ~]# umount /mnt/mysqldata/
[root@node1 ~]# drbdadm secondary mysql
[root@node1 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#
将node2设置为primary节点
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# drbdadm primary mysql
[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# mount /dev/drbd0 /mnt/mysqldata/
[root@node2 ~]# cd /mnt/mysqldata/
[root@node2 mysqldata]# ll
total 20
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
此处看到在node1上创建的文件在这里都显示了，
最后将node2的/mnt/mysqldata/卸载
[root@node2 mysqldata]# cd
[root@node2 ~]# umount /mnt/mysqldata/

至此DRBD配置成功！！！

三、mysql安装与配置

3.1 将node1设为主节点，node2设为备份节点

[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# drbdadm secondary mysql
[root@node2 ~]# drbdadm role mysql
Secondary/Secondary
[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]#

[root@node1 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# drbdadm primary mysql
[root@node1 ~]# drbdadm role mysql
Primary/Secondary
[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#
3.2 在node1.junjie.com上安装配置mysql

[root@node1 ~]# groupadd -r mysql
[root@node1 ~]# useradd -g mysql -r mysql
[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# mount /dev/drbd0 /mnt/mysqldata/
[root@node1 ~]# mkdir -pv /mnt/mysqldata/data
mkdir: created directory `/mnt/mysqldata/data'
[root@node1 ~]# chown -R mysql.mysql /mnt/mysqldata/data/
[root@node1 ~]# ll /mnt/mysqldata/
total 24
drwxr-xr-x 2 mysql mysql 4096 Feb 7 21:48 data
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node1 ~]#cd ha/
[root@node1 ha]# tar -zxvf mysql-5.5.15-linux2.6-i686.tar.gz -C /usr/local/
[root@node1 ~]# cd /usr/local/
[root@node1 src]# ln -sv mysql-5.5.15-linux2.6-i686/ mysql
create symbolic link `mysql' to `mysql-5.5.15-linux2.6-i686/'
[root@node1 src]# ll
[root@node1 src]# cd mysql
[root@node1 mysql]# chown -R mysql:mysql .
[root@node1 mysql]# scripts/mysql_install_db --user=mysql --datadir=/mnt/mysqldata/data/
[root@node1 mysql]# chown -R root .
[root@node1 mysql]# cp support-files/my-large.cnf /etc/my.cnf
[root@node1 mysql]# vim /etc/my.cnf
39 thread_concurrency = 2
40 datadir = /mnt/mysqldata/data/ #指定mysql数据文件的存放位置(添加)
为mysql提供sysv服务脚本，使其能使用service命令：
[root@node1 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@node1 mysql]# scp /etc/my.cnf node2:/etc/
my.cnf 100% 4696 4.6KB/s 00:00
[root@node1 mysql]# scp /etc/rc.d/init.d/mysqld node2:/etc/rc.d/init.d/
mysqld 100% 10KB 10.4KB/s 00:00
[root@node1 mysql]#
[root@node1 mysql]# chkconfig --add mysqld
[root@node1 mysql]# chkconfig mysqld off
[root@node1 mysql]# chkconfig --list mysqld
mysqld 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@node1 mysql]# service mysqld start
Starting MySQL..... [ OK ]
[root@node1 mysql]# l1 /mnt/mysqldata/data/
total 28744
-rw-rw---- 1 mysql mysql 5242880 Feb 7 22:00 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 Feb 7 21:59 ib_logfile1
-rw-rw---- 1 mysql mysql 18874368 Feb 7 22:00 ibdata1
drwx------ 2 mysql root 4096 Feb 7 21:53 mysql
-rw-rw---- 1 mysql mysql 126 Feb 7 22:00 mysql-bin.000001
-rw-rw---- 1 mysql mysql 19 Feb 7 21:59 mysql-bin.index
-rw-rw---- 1 mysql root 2127 Feb 7 22:00 node1.junjie.com.err
drwx------ 2 mysql mysql 4096 Feb 7 21:53 performance_schema
drwx------ 2 mysql root 4096 Feb 7 21:53 test
[root@node1 mysql]# service mysqld stop
Shutting down MySQL. [ OK ]
[root@node1 mysql]#
为了使用mysql的安装符合系统使用规范，并将其开发组件导出给系统使用，这里还需要进行如下步骤：

输出mysql的man手册至man命令的查找路径：添加如下行即可：
[root@node1 mysql]# vim /etc/man.config
48行添加 MANPATH /usr/local/mysql/man
输出mysql的头文件至系统头文件路径/usr/include，这可以通过简单的创建链接实现：
[root@node1 mysql]# ln -sv /usr/local/mysql/include /usr/include/mysql
create symbolic link `/usr/include/mysql' to `/usr/local/mysql/include'
[root@node1 mysql]#
输出mysql的库文件给系统库查找路径：（文件只要是在/etc/ld.so.conf.d/下并且后缀是.conf就可以）而后让系统重新载入系统库
[root@node1 mysql]# echo '/usr/local/mysql/lib' >> /etc/ld.so.conf.d/mysql.conf
[root@node1 mysql]# ldconfig -v |grep mysql
/usr/local/mysql/lib:
libmysqlclient.so.18 -> libmysqlclient_r.so.18.0.0
修改PATH环境变量，让系统所有用户可以直接使用mysql的相关命令:
[root@node1 mysql]# vim /etc/profile
59 PATH=$PATH:/usr/local/mysql/bin #添加
[root@node1 mysql]# . /etc/profile
[root@node1 mysql]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin
卸载drbd设备：
[root@node1 mysql]# umount /mnt/mysqldata/
3.3 将node2设为主节点，node1设为备份节点

[root@node1 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]# drbdadm secondary mysql
[root@node1 ~]# drbdadm role mysql
Secondary/Secondary
[root@node1 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node1 ~]#

[root@node2 ~]# drbd-overview
0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]# drbdadm primary mysql
[root@node2 ~]# drbdadm role mysql
Primary/Secondary
[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]#
3.4 在node2.junjie.com上安装配置mysql

添加用户和组：
[root@node2 ~]# groupadd -r mysql
[root@node2 ~]# useradd -g mysql -r mysql
[root@node2 ~]# drbd-overview
0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----
[root@node2 ~]#
挂载drbd设备：
[root@node2 ~]# mount /dev/drbd0 /mnt/mysqldata/
[root@node2 ~]# ll /mnt/mysqldata/
total 24
drwxr-xr-x 5 mysql mysql 4096 Feb 7 22:00 data
-rw-r--r-- 1 root root 4 Feb 7 21:28 f1
-rw-r--r-- 1 root root 0 Feb 7 21:28 f2
drwx------ 2 root root 16384 Feb 7 21:26 lost+found
[root@node2 ~]# ll /mnt/mysqldata/data/
total 28744
-rw-rw---- 1 mysql mysql 5242880 Feb 7 22:00 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 Feb 7 21:59 ib_logfile1
-rw-rw---- 1 mysql mysql 18874368 Feb 7 22:00 ibdata1
drwx------ 2 mysql root 4096 Feb 7 21:53 mysql
-rw-rw---- 1 mysql mysql 126 Feb 7 22:00 mysql-bin.000001
-rw-rw---- 1 mysql mysql 19 Feb 7 21:59 mysql-bin.index
-rw-rw---- 1 mysql root 2127 Feb 7 22:00 node1.junjie.com.err
drwx------ 2 mysql mysql 4096 Feb 7 21:53 performance_schema
drwx------ 2 mysql root 4096 Feb 7 21:53 test
[root@node2 ~]#
安装mysql：
[root@node2 ~]# cd ha/
[root@node2 ha]# tar -zxvf mysql-5.5.15-linux2.6-i686.tar.gz -C /usr/local/
[root@node2 ha]# cd /usr/local/
[root@node2 local]# ln -sv mysql-5.5.15-linux2.6-i686/ mysql
create symbolic link `mysql' to `mysql-5.5.15-linux2.6-i686/'
[root@node2 local]# cd mysql
一定不能对数据库进行初始化，因为我们在node1上已经初始化了：
[root@node2 mysql]# chown -R root:mysql .
mysql主配置文件和sysc服务脚本已经从node1复制过来了，不用在添加。
管理mysql服务：
[root@node2 mysql]# chkconfig --add mysqld
[root@node2 mysql]# chkconfig mysqld off
[root@node2 mysql]# chkconfig --list mysqld
mysqld 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@node2 mysql]#
启动mysql服务：
[root@node2 mysql]# service mysqld start
Starting MySQL....... [ OK ]
[root@node2 mysql]# ll /mnt/mysqldata/data/
total 28756
-rw-rw---- 1 mysql mysql 5242880 Feb 7 22:29 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 Feb 7 21:59 ib_logfile1
-rw-rw---- 1 mysql mysql 18874368 Feb 7 22:00 ibdata1
drwx------ 2 mysql root 4096 Feb 7 21:53 mysql
-rw-rw---- 1 mysql mysql 126 Feb 7 22:00 mysql-bin.000001
-rw-rw---- 1 mysql mysql 107 Feb 7 22:29 mysql-bin.000002
-rw-rw---- 1 mysql mysql 38 Feb 7 22:29 mysql-bin.index
-rw-rw---- 1 mysql root 2127 Feb 7 22:00 node1.junjie.com.err
-rw-rw---- 1 mysql root 941 Feb 7 22:29 node2.junjie.com.err
-rw-rw---- 1 mysql mysql 5 Feb 7 22:29 node2.junjie.com.pid
drwx------ 2 mysql mysql 4096 Feb 7 21:53 performance_schema
drwx------ 2 mysql root 4096 Feb 7 21:53 test
测试之后关闭服务：
[root@node2 mysql]# service mysqld stop
Shutting down MySQL. [ OK ]
[root@node2 mysql]#
为了使用mysql的安装符合系统使用规范，并将其开发组件导出给系统使用，这里还需要进行如下步骤：

输出mysql的man手册至man命令的查找路径：添加如下行即可：
[root@node2 mysql]# vim /etc/man.config
48行添加 MANPATH /usr/local/mysql/man
输出mysql的头文件至系统头文件路径/usr/include，这可以通过简单的创建链接实现：
[root@node2 mysql]# ln -sv /usr/local/mysql/include /usr/include/mysql
create symbolic link `/usr/include/mysql' to `/usr/local/mysql/include'
[root@node2 mysql]#
输出mysql的库文件给系统库查找路径：（文件只要是在/etc/ld.so.conf.d/下并且后缀是.conf就可以）而后让系统重新载入系统库
[root@node2 mysql]# echo '/usr/local/mysql/lib' >> /etc/ld.so.conf.d/mysql.conf
[root@node2 mysql]# ldconfig -v |grep mysql
/usr/local/mysql/lib:
libmysqlclient.so.18 -> libmysqlclient_r.so.18.0.0
修改PATH环境变量，让系统所有用户可以直接使用mysql的相关命令:
[root@node2 mysql]# vim /etc/profile
59 PATH=$PATH:/usr/local/mysql/bin #添加
[root@node2 mysql]# . /etc/profile
[root@node2 mysql]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin
卸载drbd设备：
[root@node2 mysql]# umount /mnt/mysqldata/
四、corosync+pacemaker的安装和配置

4.1安装有关软件

[root@node1 ~]# cd ha/
[root@node1 ha]# yum localinstall -y *.rpm --nogpgcheck
…
Installed:
cluster-glue.i386 0:1.0.6-1.6.el5 cluster-glue-libs.i386 0:1.0.6-1.6.el5
corosync.i386 0:1.2.7-1.1.el5 corosynclib.i386 0:1.2.7-1.1.el5
heartbeat.i386 0:3.0.3-2.3.el5 heartbeat-libs.i386 0:3.0.3-2.3.el5
libesmtp.i386 0:1.0.4-5.el5 openais.i386 0:1.1.3-1.6.el5
openaislib.i386 0:1.1.3-1.6.el5 pacemaker.i386 0:1.1.5-1.1.el5
pacemaker-cts.i386 0:1.1.5-1.1.el5 pacemaker-libs.i386 0:1.1.5-1.1.el5
perl-TimeDate.noarch 1:1.16-5.el5 resource-agents.i386 0:1.0.4-1.1.el5

Dependency Installed:
libibverbs.i386 0:1.1.2-4.el5 librdmacm.i386 0:1.0.8-5.el5
libtool-ltdl.i386 0:1.5.22-6.1 lm_sensors.i386 0:2.10.7-4.el5
openhpi-libs.i386 0:2.14.0-5.el5 openib.noarch 0:1.4.1-3.el5

Complete!
[root@node1 ha]#

[root@node2 ~]# cd ha/
[root@node2 ha]# yum localinstall -y *.rpm --nogpgcheck
…
Installed:
cluster-glue.i386 0:1.0.6-1.6.el5 cluster-glue-libs.i386 0:1.0.6-1.6.el5
corosync.i386 0:1.2.7-1.1.el5 corosynclib.i386 0:1.2.7-1.1.el5
heartbeat.i386 0:3.0.3-2.3.el5 heartbeat-libs.i386 0:3.0.3-2.3.el5
libesmtp.i386 0:1.0.4-5.el5 openais.i386 0:1.1.3-1.6.el5
openaislib.i386 0:1.1.3-1.6.el5 pacemaker.i386 0:1.1.5-1.1.el5
pacemaker-cts.i386 0:1.1.5-1.1.el5 pacemaker-libs.i386 0:1.1.5-1.1.el5
perl-TimeDate.noarch 1:1.16-5.el5 resource-agents.i386 0:1.0.4-1.1.el5

Dependency Installed:
libibverbs.i386 0:1.1.2-4.el5 librdmacm.i386 0:1.0.8-5.el5
libtool-ltdl.i386 0:1.5.22-6.1 lm_sensors.i386 0:2.10.7-4.el5
openhpi-libs.i386 0:2.14.0-5.el5 openib.noarch 0:1.4.1-3.el5

Complete!
[root@node2 ha]#
4.2 对node1和node2节点进行相应的配置

1：切换到主配置文件的目录
[root@node1 ha]# cd /etc/corosync/
[root@node1 corosync]# ll
total 20
-rw-r--r-- 1 root root 5384 Jul 28 2010 amf.conf.example
-rw-r--r-- 1 root root 436 Jul 28 2010 corosync.conf.example
drwxr-xr-x 2 root root 4096 Jul 28 2010 service.d
drwxr-xr-x 2 root root 4096 Jul 28 2010 uidgid.d
[root@node1 corosync]# cp corosync.conf.example corosync.conf
[root@node1 corosync]# vim corosync.conf
10 bindnetaddr: 192.168.101.0 #修改此行
#添加以下几行
33 service {
34 ver: 0
35 name: pacemaker
36 use_mgmtd: yes
37 }
38 aisexec {
39 user: root
40 group: root
41 }
2:创建cluster目录
[root@node1 corosync]# mkdir -pv /var/log/cluster
mkdir: created directory `/var/log/cluster'
3:为了便面其他主机加入该集群，需要认证，生成一authkey
[root@node1 corosync]# corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.
[root@node1 corosync]# ll
total 28
-rw-r--r-- 1 root root 5384 Jul 28 2010 amf.conf.example
-r-------- 1 root root 128 Feb 7 22:47 authkey
-rw-r--r-- 1 root root 555 Feb 7 22:45 corosync.conf
-rw-r--r-- 1 root root 436 Jul 28 2010 corosync.conf.example
drwxr-xr-x 2 root root 4096 Jul 28 2010 service.d
drwxr-xr-x 2 root root 4096 Jul 28 2010 uidgid.d
[root@node1 corosync]#
4:将node1节点上的文件拷贝到节点node2上面（记住要带-p)
[root@node1 corosync]# scp -p authkey corosync.conf node2:/etc/corosync/
authkey 100% 128 0.1KB/s 00:00
corosync.conf 100% 561 0.6KB/s 00:00
[root@node1 corosync]# ssh node2 'mkdir -pv /var/log/cluster'
mkdir: created directory `/var/log/cluster'
[root@node1 corosync]#

4.3 在node1和node2节点上检测

1:在node1和node2节点上面启动 corosync 的服务
[root@node1 corosync]# service corosync start
Starting Corosync Cluster Engine (corosync): [ OK ]
[root@node1 corosync]# ssh node2 'service corosync start'
Starting Corosync Cluster Engine (corosync): [ OK ]
[root@node1 corosync]#
2:在node1上验证corosync引擎是否正常启动了
[root@node1 ~]# grep -i -e "corosync cluster engine" -e "configuration file" /var/log/messages
Feb 7 22:51:41 node1 corosync[5149]: [MAIN ] Corosync Cluster Engine ('1.2.7'): started and ready to provide service.
Feb 7 22:51:41 node1 corosync[5149]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
3: 在node1上查看初始化成员节点通知是否发出
[root@node1 ~]# grep -i totem /var/log/messages
Feb 7 22:51:41 node1 corosync[5149]: [TOTEM ] Initializing transport (UDP/IP).
Feb 7 22:51:41 node1 corosync[5149]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Feb 7 22:51:42 node1 corosync[5149]: [TOTEM ] The network interface [192.168.101.81] is now up.
Feb 7 22:51:42 node1 corosync[5149]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
4: 在node1上检查过程中是否有错误产生（避免stonith的错误）
[root@node1 ~]# grep -i error: /var/log/messages |grep -v unpack_resources
出现如下1个错误：
Feb 7 22:51:43 node1 corosync[5149]: [pcmk ] ERROR: pcmk_wait_dispatch: Child process mgmtd exited (pid=5161, rc=100)此处
解决方法：
仔细看了/var/log/messages日志，或者使用crm_verify -L检查一下错误，其实没必要卸载重装。这个错误是由于缺少snoith设备引起的，并不会影响corosync的运行。可以忽略这个错误。
[root@node1 corosync]# crm_verify -L
crm_verify[5247]: 2012/02/07_23:09:37 ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
crm_verify[5247]: 2012/02/07_23:09:37 ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
crm_verify[5247]: 2012/02/07_23:09:37 ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Errors found during check: config not valid
-V may provide more details
[root@node1 corosync]#
5: 在node1上检查pacemaker时候已经启动了？（如下显示正常启动）
[root@node1 ~]# grep -i pcmk_startup /var/log/messages
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: CRM: Initialized
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] Logging: Initialized pcmk_startup
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: Maximum core file size is: 4294967295
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: Service: 9
Feb 7 22:51:42 node1 corosync[5149]: [pcmk ] info: pcmk_startup: Local hostname: node1.junjie.com
6: 在node2上验证corosync引擎是否正常启动了
[root@node2 ~]# grep -i -e "corosync cluster engine" -e "configuration file" /var/log/messages
Feb 7 22:53:01 node2 corosync[5246]: [MAIN ] Corosync Cluster Engine ('1.2.7'): started and ready to provide service.
Feb 7 22:53:01 node2 corosync[5246]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
7: 在node2上查看初始化成员节点通知是否发出
[root@node2 ~]# grep -i totem /var/log/messages
Feb 7 22:53:01 node2 corosync[5246]: [TOTEM ] Initializing transport (UDP/IP).
Feb 7 22:53:01 node2 corosync[5246]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Feb 7 22:53:02 node2 corosync[5246]: [TOTEM ] The network interface [192.168.101.82] is now up.
Feb 7 22:53:02 node2 corosync[5246]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb 7 22:53:03 node2 corosync[5246]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
8: 在node2上检查过程中是否有错误产生（避免stonith的错误，如下显示只有stonith错误，可忽略）
[root@node2 ~]# grep -i error: /var/log/messages |grep -v unpack_resources
Feb 7 22:53:03 node2 corosync[5246]: [pcmk ] ERROR: pcmk_wait_dispatch: Child process mgmtd exited (pid=5258, rc=100)
9: 在node2上检查pacemaker时候已经启动了？（如下显示正常启动）
[root@node2 ~]# grep -i pcmk_startup /var/log/messages
[root@node2 ~]# grep -i pcmk_startup /var/log/messages
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: CRM: Initialized
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] Logging: Initialized pcmk_startup
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: Maximum core file size is: 4294967295
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: Service: 9
Feb 7 22:53:02 node2 corosync[5246]: [pcmk ] info: pcmk_startup: Local hostname: node2.junjie.com
10: 在node1和node2上查看群集的状态
[root@node1 corosync]# crm status
============
Last updated: Tue Feb 7 23:18:39 2012
Stack: openais
Current DC: node1.junjie.com - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
0 Resources configured.
============

Online: [ node2.junjie.com node1.junjie.com ]

[root@node1 corosync]#

[root@node2 corosync]# crm status
============
Last updated: Tue Feb 7 23:20:06 2012
Stack: openais
Current DC: node1.junjie.com - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
0 Resources configured.
============

Online: [ node2.junjie.com node1.junjie.com ]

[root@node2 corosync]#
五．群集管理

5.1 配置群集的工作属性

corosync 默认启用了stonith，而当前集群并没有相应的stonith设备，因此此默认配置目前尚不可用，这可以通过如下命令先禁用stonith：

[root@node1 corosync]# cd

[root@node1 ~]# crm configure property stonith-enabled=false

[root@node1 ~]#

[root@node2 corosync]# cd

[root@node2 ~]# crm configure property stonith-enabled=false

[root@node2 ~]#

对于双节点的集群来说，我们要配置此选项来忽略quorum，即这时候票数不起作用，一个节点也能正常运行：

[root@node1 ~]#

[root@node1 ~]# crm configure property no-quorum-policy=ignore

[root@node1 ~]#

[root@node2 ~]#

[root@node2 ~]# crm configure property no-quorum-policy=ignore

[root@node2 ~]#

定义资源的粘性值，使资源不能再节点之间随意的切换，因为这样是非常浪费系统的资源的。

资源黏性值范围及其作用：

0 ：这是默认选项。资源放置在系统中的最适合位置。这意味着当负载能力“较好”或较差的节点变得可用时才转移资源。此选项的作用基本等同于自动故障回复，只是资源可能会转移到非之前活动的节点上；

大于0：资源更愿意留在当前位置，但是如果有更合适的节点可用时会移动。值越高表示资源越愿意留在当前位置；

小于0：资源更愿意移离当前位置。绝对值越高表示资源越愿意离开当前位置；

INFINITY：如果不是因节点不适合运行资源（节点关机、节点待机、达到migration-threshold 或配置更改）而强制资源转移，资源总是留在当前位置。此选项的作用几乎等同于完全禁用自动故障回复；

-INFINITY ：资源总是移离当前位置；

我们这里可以通过以下方式为资源指定默认黏性值：

[root@node1 ~]#

[root@node1 ~]# crm configure rsc_defaults resource-stickiness=100

[root@node1 ~]#

[root@node2 ~]#

[root@node2 ~]# crm configure rsc_defaults resource-stickiness=100

[root@node2 ~]#

5.2 定义群集服务及资源

5.2.1 改变drbd的状态

[root@node2 ~]# drbd-overview

0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----

[root@node2 ~]# drbdadm secondary mysql

[root@node2 ~]# drbd-overview

0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----

[root@node2 ~]#

[root@node1 ~]# drbd-overview

0:mysql Connected Secondary/Secondary UpToDate/UpToDate C r----

[root@node1 ~]# drbdadm primary mysql

[root@node1 ~]# drbd-overview

0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----

[root@node1 ~]#

5.2.2 配置drbd为群集资源

1、查看当前集群的配置信息，确保已经配置全局属性参数为两节点集群所适用

[root@node1 ~]# crm configure show

node node1.junjie.com

node node2.junjie.com

property $id="cib-bootstrap-options" \

dc-version="1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f" \

cluster-infrastructure="openais" \

expected-quorum-votes="2" \

stonith-enabled="false" \

no-quorum-policy="ignore"

rsc_defaults $id="rsc-options" \

resource-stickiness="100"

[root@node1 ~]#

2、将已经配置好的DRBD设备/dev/drbd0定义为集群服务；

[root@node1 ~]# service drbd stop

Stopping all DRBD resources: .

[root@node1 ~]# chkconfig drbd off

[root@node1 ~]# ssh node2 'service drbd stop'

Stopping all DRBD resources: .

[root@node1 ~]# ssh node2 'chkconfig drbd off'

[root@node1 ~]# drbd-overview

drbd not loaded

[root@node1 ~]# ssh node2 'drbd-overview'

drbd not loaded

[root@node1 ~]#

3、配置drbd为集群资源：

提供drbd的RA目前由OCF归类为linbit，其路径为/usr/lib/ocf/resource.d/linbit/drbd。我们可以使用如下命令来查看此RA及RA的meta信息：

[root@node1 ~]# crm ra classes

heartbeat

lsb

ocf / heartbeat linbit pacemaker

stonith

[root@node1 ~]# crm ra list ocf linbit

drbd

[root@node1 ~]#

4、查看drbd的资源代理的相关信息：

[root@node1 ~]# crm ra info ocf:linbit:drbd

This resource agent manages a DRBD resource

as a master/slave resource. DRBD is a shared-nothing replicated storage

device. (ocf:linbit:drbd)

Master/Slave OCF Resource Agent for DRBD

Parameters (* denotes required, [] the default):

drbd_resource* (string): drbd resource name

The name of the drbd resource from the drbd.conf file.

drbdconf (string, [/etc/drbd.conf]): Path to drbd.conf

Full path to the drbd.conf file.

Operations' defaults (advisory minimum):

start timeout=240

promote timeout=90

demote timeout=90

notify timeout=90

stop timeout=100

monitor_Slave interval=20 timeout=20 start-delay=1m

monitor_Master interval=10 timeout=20 start-delay=1m

[root@node1 ~]#

5、drbd需要同时运行在两个节点上，但只能有一个节点（primary/secondary模型）是Master，而另一个节点为Slave；因此，它是一种比较特殊的集群资源，其资源类型为多状态（Multi-state）clone类型，即主机节点有Master和Slave之分，且要求服务刚启动时两个节点都处于slave状态。

[root@node1 ~]# crm

crm(live)# configure

crm(live)configure# primitive mysqldrbd ocf:heartbeat:drbd params drbd_resource="mysql" op monitor role="Master" interval="30s" op monitor role="Slave" interval="31s" op start timeout="240s" op stop timeout="100s"

crm(live)configure# ms MS_mysqldrbd mysqldrbd meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify="true"

crm(live)configure# show mysqldrbd

primitive mysqldrbd ocf:heartbeat:drbd \

params drbd_resource="mysql" \

op monitor interval="30s" role="Master" \

op monitor interval="31s" role="Slave" \

op start interval="0" timeout="240s" \

op stop interval="0" timeout="100s"

crm(live)configure# show MS_mysqldrbd

ms MS_mysqldrbd mysqldrbd \

meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"

确定无误后，提交：

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# exit

bye

[root@node1 ~]#

6、查看当前集群运行状态：

[root@node1 ~]# crm status

============

Last updated: Tue Feb 7 23:51:09 2012

Stack: openais

Current DC: node1.junjie.com - partition with quorum

Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f

2 Nodes configured, 2 expected votes

1 Resources configured.

============

Online: [ node2.junjie.com node1.junjie.com ]

Master/Slave Set: MS_mysqldrbd [mysqldrbd]

Masters: [ node1.junjie.com ]

Slaves: [ node2.junjie.com ]

[root@node1 ~]#

由上面的信息可以看出此时的drbd服务的Primary节点为node1.junjie.com，Secondary节点为node2.junjie.com。当然，也可以在node1上使用如下命令验正当前主机是否已经成为mysql资源的Primary节点：

[root@node1 ~]# drbdadm role mysql

Primary/Secondary

[root@node1 ~]# drbd-overview

0:mysql Connected Primary/Secondary UpToDate/UpToDate C r----

[root@node1 ~]#

我们实现将drbd设置自动挂载至/mysqldata目录。此外，此自动挂载的集群资源需要运行于drbd服务的Master节点上，并且只能在drbd服务将某节点设置为Primary以后方可启动。

确保两个节点上的设备已经卸载：

[root@node1 ~]# umount /dev/drbd0

[root@node2 ~]# umount /dev/drbd0

以下还在node1上操作：

[root@node1 ~]# crm

crm(live)# configure

crm(live)configure# primitive MysqlFS ocf:heartbeat:Filesystem params device="/dev/drbd0" directory="/mnt/mysqldata" fstype="ext3" op start timeout=60s op stop timeout=60s

crm(live)configure#

crm(live)configure# show changed

primitive MysqlFS ocf:heartbeat:Filesystem \

params device="/dev/drbd0" directory="/mnt/mysqldata" fstype="ext3" \

op start interval="0" timeout="60s" \

op stop interval="0" timeout="60s"

crm(live)configure#

crm(live)configure# commit

crm(live)configure# exit

bye

[root@node1 ~]#

7、mysql资源的定义（node1上操作）

先为mysql集群创建一个IP地址资源，通过集群提供服务时使用，这个地址就是客户端访问mysql服务器使用的ip地址；

[root@node1 ~]# crm configure primitive myip ocf:heartbeat:IPaddr params ip=192.168.101.88

[root@node1 ~]# crm configure primitive mysqlserver lsb:mysqld

[root@node1 ~]# crm status

============

Last updated: Wed Feb 8 00:01:23 2012

Stack: openais

Current DC: node1.junjie.com - partition with quorum

Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f

2 Nodes configured, 2 expected votes

4 Resources configured.

============

Online: [ node2.junjie.com node1.junjie.com ]

Master/Slave Set: MS_mysqldrbd [mysqldrbd]

Masters: [ node1.junjie.com ]

Slaves: [ node2.junjie.com ]

MysqlFS (ocf::heartbeat:Filesystem): Started node1.junjie.com

myip (ocf::heartbeat:IPaddr): Started node2.junjie.com

mysqlserver (lsb:mysqld): Started node1.junjie.com

[root@node1 ~]#

8、配置资源的各种约束：

集群拥有所有必需资源，但它可能还无法进行正确处理。资源约束则用以指定在哪些群集节点上运行资源，以何种顺序装载资源，以及特定资源依赖于哪些其它资源。pacemaker共给我们提供了三种资源约束方法：

1）Resource Location（资源位置）：定义资源可以、不可以或尽可能在哪些节点上运行

2）Resource Collocation（资源排列）：排列约束用以定义集群资源可以或不可以在某个节点上同时运行

3）Resource Order（资源顺序）：顺序约束定义集群资源在节点上启动的顺序。

定义约束时，还需要指定分数。各种分数是集群工作方式的重要组成部分。其实，从迁移资源到决定在已降级集群中停止哪些资源的整个过程是通过以某种方式修改分数来实现的。分数按每个资源来计算，资源分数为负的任何节点都无法运行该资源。在计算出资源分数后，集群选择分数最高的节点。INFINITY（无穷大）目前定义为 1,000,000。加减无穷大遵循以下3个基本规则：

1）任何值 + 无穷大 = 无穷大

2）任何值 - 无穷大 = -无穷大

3）无穷大 - 无穷大 = -无穷大

定义资源约束时，也可以指定每个约束的分数。分数表示指派给此资源约束的值。分数较高的约束先应用，分数较低的约束后应用。通过使用不同的分数为既定资源创建更多位置约束，可以指定资源要故障转移至的目标节点的顺序。

我们要定义如下的约束：

[root@node1 ~]# crm

crm(live)# configure

crm(live)configure# colocation MysqlFS_with_mysqldrbd inf: MysqlFS MS_mysqldrbd:Master myip mysqlserver

crm(live)configure# order MysqlFS_after_mysqldrbd inf: MS_mysqldrbd:promote MysqlFS:start

crm(live)configure# order myip_after_MysqlFS mandatory: MysqlFS myip

crm(live)configure# order mysqlserver_after_myip mandatory: myip mysqlserver

crm(live)configure# show changed

colocation MysqlFS_with_mysqldrbd inf: MysqlFS MS_mysqldrbd:Master myip mysqlserver

order MysqlFS_after_mysqldrbd inf: MS_mysqldrbd:promote MysqlFS:start

order myip_after_MysqlFS inf: MysqlFS myip

order mysqlserver_after_myip inf: myip mysqlserver

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# exit

bye

[root@node1 ~]#

9.查看配置信息和状态，并测试：

[root@node1 ~]# crm configure show

node node1.junjie.com

node node2.junjie.com

primitive MysqlFS ocf:heartbeat:Filesystem \

params device="/dev/drbd0" directory="/mnt/mysqldata" fstype="ext3" \

op start interval="0" timeout="60s" \

op stop interval="0" timeout="60s"

primitive myip ocf:heartbeat:IPaddr \

params ip="192.168.101.88"

primitive mysqldrbd ocf:heartbeat:drbd \

params drbd_resource="mysql" \

op monitor interval="30s" role="Master" \

op monitor interval="31s" role="Slave" \

op start interval="0" timeout="240s" \

op stop interval="0" timeout="100s"

primitive mysqlserver lsb:mysqld

ms MS_mysqldrbd mysqldrbd \

meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"

colocation MysqlFS_with_mysqldrbd inf: MysqlFS MS_mysqldrbd:Master myip mysqlserver

order MysqlFS_after_mysqldrbd inf: MS_mysqldrbd:promote MysqlFS:start

order myip_after_MysqlFS inf: MysqlFS myip

order mysqlserver_after_myip inf: myip mysqlserver

property $id="cib-bootstrap-options" \

dc-version="1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f" \

cluster-infrastructure="openais" \

expected-quorum-votes="2" \

stonith-enabled="false" \

no-quorum-policy="ignore"

rsc_defaults $id="rsc-options" \

resource-stickiness="100"

[root@node1 ~]#

[root@node1 ~]# crm status

============

Last updated: Wed Feb 8 00:08:06 2012

Stack: openais

Current DC: node1.junjie.com - partition with quorum

Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f

2 Nodes configured, 2 expected votes

4 Resources configured.

============

Online: [ node2.junjie.com node1.junjie.com ]

Master/Slave Set: MS_mysqldrbd [mysqldrbd]

Masters: [ node1.junjie.com ]

Slaves: [ node2.junjie.com ]

MysqlFS (ocf::heartbeat:Filesystem): Started node1.junjie.com

myip (ocf::heartbeat:IPaddr): Started node1.junjie.com

mysqlserver (lsb:mysqld): Started node1.junjie.com

[root@node1 ~]#

可见，服务现在在node1上正常运行：

在node1上的操作，查看群集的运行状态：

[root@node1 ~]# service mysqld status

MySQL running (8720) [ OK ]

[root@node1 ~]# mount |grep drbd

/dev/drbd0 on /mnt/mysqldata type ext3 (rw)

[root@node1 ~]#

[root@node1 ~]# ll /mnt/mysqldata/

total 24

drwxr-xr-x 5 mysql mysql 4096 Feb 8 00:05 data

-rw-r--r-- 1 root root 4 Feb 7 21:28 f1

-rw-r--r-- 1 root root 0 Feb 7 21:28 f2

drwx------ 2 root root 16384 Feb 7 21:26 lost+found

[root@node1 ~]# ifconfig eth0:0

eth0:0 Link encap:Ethernet HWaddr 00:0C:29:AE:83:D1

inet addr:192.168.101.88 Bcast:192.168.101.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

Interrupt:67 Base address:0x2000

[root@node1 ~]#

在node2上的操作，查看群集的运行状态：

[root@node2 ~]# service mysqld status

MySQL is not running [FAILED]

[root@node2 ~]# mount |grep drbd

[root@node2 ~]# ll /mnt/mysqldata/

total 0

[root@node2 ~]# ifconfig eth0:0

eth0:0 Link encap:Ethernet HWaddr 00:0C:29:D1:D4:32

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

Interrupt:67 Base address:0x2000

[root@node2 ~]#

10.继续测试群集：

继续测试：

在node1上操作，让node1下线：

[root@node1 ~]# crm status

============

Last updated: Wed Feb 8 00:16:25 2012

Stack: openais

Current DC: node1.junjie.com - partition with quorum

Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f

2 Nodes configured, 2 expected votes

4 Resources configured.

============

Node node1.junjie.com: standby

Online: [ node2.junjie.com ]

Master/Slave Set: MS_mysqldrbd [mysqldrbd]

Masters: [ node2.junjie.com ]

Stopped: [ mysqldrbd:0 ]

MysqlFS (ocf::heartbeat:Filesystem): Started node2.junjie.com

myip (ocf::heartbeat:IPaddr): Started node2.junjie.com

mysqlserver (lsb:mysqld): Started node2.junjie.com

[root@node1 ~]#

在node2上的操作，查看群集的运行状态：

可见我们的资源已经都切换到了node2上：

[root@node2 ~]# service mysqld status

MySQL running (9093) [ OK ]

[root@node2 ~]# mount |grep drbd

/dev/drbd0 on /mnt/mysqldata type ext3 (rw)

[root@node2 ~]# ll /mnt/mysqldata/

total 24

drwxr-xr-x 5 mysql mysql 4096 Feb 8 00:16 data

-rw-r--r-- 1 root root 4 Feb 7 21:28 f1

-rw-r--r-- 1 root root 0 Feb 7 21:28 f2

drwx------ 2 root root 16384 Feb 7 21:26 lost+found

[root@node2 ~]# ifconfig eth0:0

eth0:0 Link encap:Ethernet HWaddr 00:0C:29:D1:D4:32

inet addr:192.168.101.88 Bcast:192.168.101.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

Interrupt:67 Base address:0x2000

[root@node2 ~]#

现在一切正常，我们可以验证mysql服务是否能被正常访问：

首先，在node2上面建立一个用户user1，密码：123456.

我们定义的是通过VIP:192.168.101.88来访问mysql服务，现在node2上建立一个可以让某个网段主机能访问的账户(这个内容会同步drbd设备同步到node1上)：

[root@node2 ~]# mysql

...

mysql> grant all on *.* to user1@'192.168.%.%' identified by '123456';

Query OK, 0 rows affected (0.01 sec)

mysql> flush privileges;

Query OK, 0 rows affected (0.00 sec)

mysql> exit

Bye

[root@node2 ~]#

客户端访问测试

192.168．101.100客户Ping测试192.168.101.88

192.168．101.100客户访问mysql数据库192.168.101.88（成功访问）

至此：使用corosync+drbd+pacemaker实现mysql服务器的高可用集群成功完成！.

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
mysql服务器的HA集群之corosync+drbd+pacemaker实现

本文属转载，请尊重他人的劳动成果，转载时请务必以超链接形式标明文章原始出处、作者信息和本声明。否则将追究法律责任。http://xjzhujunjie.blog.51cto.com/3582724/886317案例应用：红帽企业群集和存储管理之mysql服务器的HA集群之corosync+drbd+pacemaker实现案例应用企业需求：某公司需要通过HA实现服务器的高可用
复制链接

扫一扫

评论

被折叠的条评论为什么被折叠?

到【灌水乐园】发言

查看更多评论

添加红包

成就一亿技术人!

hope_wisdom

发出的红包

实付元

使用余额支付

点击重新获取

扫码支付

钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。