三、RHCS的corosync组件实现HA(高可用)。

1、pacemaker作为corosync的插件运行

搭建环境：

ms.dtedu.com：管理HA的站点（ansible）

node5.dtedu.com：高可用节点1

node6.dtedu.com：高可用节点2

资源：vip+web+fielsystem

前提条件：

1、时间同步

2、dns解析

3、ssh互信

4、关闭iptables

5、关闭selinux

注意：运行networkmanager组件，将不能把此节点放在集群中

1.1安装ansible软件,然后安装corosync、pecemaker，安装pacemaker不能安装heartbeat,安装不上注意仓库的选择问题。

[root@ms.dtedu.com~]$ansible all -a "yum install -y corosync pacemaker"

node5.dtedu.com | SUCCESS | rc=0 >>

已加载插件：fastestmirror, refresh-packagekit

设置安装进程

Loading mirror speeds from cached hostfile

 * base: mirror.bit.edu.cn

 * epel: mirrors.tuna.tsinghua.edu.cn

 * extras: mirrors.tuna.tsinghua.edu.cn

 * updates: mirrors.tuna.tsinghua.edu.cn

依赖关系解决


================================================================================

 软件包                      架构        版本                   仓库       大小

================================================================================

正在安装:

 pacemaker                   x86_64      1.1.15-5.el6           base      443 k

为依赖而安装:

 cifs-utils                  x86_64      4.8.1-20.el6           base       65 k

 clusterlib                  x86_64      3.0.12.1-84.el6        base      109 k

 cman                        x86_64      3.0.12.1-84.el6        base      454 k

 cyrus-sasl-md5              x86_64      2.1.23-15.el6_6.2      base       47 k

 fence-agents                x86_64      4.0.15-13.el6          base      193 k

 fence-virt                  x86_64      0.2.3-24.el6           base       39 k

 gnutls-utils                x86_64      2.12.23-21.el6         base      109 k

 ipmitool                    x86_64      1.8.15-2.el6           base      465 k

 libtasn1-devel              x86_64      2.3-6.el6_5            base       61 k

 libvirt-client              x86_64      0.10.2-62.el6          base      4.1 M

 modcluster                  x86_64      0.16.2-35.el6          base      210 k

 nc                          x86_64      1.84-24.el6            base       57 k

 net-snmp-utils              x86_64      1:5.5-60.el6           base      177 k

 numactl                     x86_64      2.0.9-2.el6            base       74 k

 oddjob                      x86_64      0.30-6.el6             base       60 k

 openais                     x86_64      1.1.1-7.el6            base      192 k

 openaislib                  x86_64      1.1.1-7.el6            base       82 k

 pacemaker-cli               x86_64      1.1.15-5.el6           base      291 k

 pacemaker-cluster-libs      x86_64      1.1.15-5.el6           base       85 k

 pacemaker-libs              x86_64      1.1.15-5.el6           base      483 k

 perl-Net-Telnet             noarch      3.03-11.el6            base       56 k

 pexpect                     noarch      2.3-6.el6              base      147 k

 pyOpenSSL                   x86_64      0.13.1-2.el6           base      263 k

 python-suds                 noarch      0.4.1-3.el6            base      218 k

 quota                       x86_64      1:3.17-23.el6          base      202 k

 resource-agents             x86_64      3.9.5-46.el6           base      389 k

 ricci                       x86_64      0.16.2-87.el6          base      633 k

 sg3_utils                   x86_64      1.28-12.el6            base      498 k

 tcp_wrappers                x86_64      7.6-58.el6             base       70 k

 yajl                        x86_64      1.0.7-3.el6            base       27 k

为依赖而更新:

 gnutls                      x86_64      2.12.23-21.el6         base      389 k

 gnutls-devel                x86_64      2.12.23-21.el6         base      1.2 M

 net-snmp-devel              x86_64      1:5.5-60.el6           base      307 k

 net-snmp-libs               x86_64      1:5.5-60.el6           base      1.5 M

 nspr                        x86_64      4.13.1-1.el6           base      114 k

 nss                         x86_64      3.27.1-13.el6          base      873 k

 nss-sysinit                 x86_64      3.27.1-13.el6          base       50 k

 nss-tools                   x86_64      3.27.1-13.el6          base      443 k

 nss-util 


软件包               架构            版本                  仓库           大小

================================================================================

正在安装:

 corosync             x86_64          1.4.7-5.el6           base          216 k

为依赖而安装:

 corosynclib          x86_64          1.4.7-5.el6           base          194 k


                    x86_64      3.27.1-3.el6           base       68 k


1.2安装crmsh，pssh软件包。crmsh依赖于pssh。

[root@ms.dtedu.com~]$ansible all -a "chdir=/etc/yum.repos.d wget http://download.opensuse.org/repositories/network:ha-clustering:Stable/RedHat_RHEL-6/network:ha-clustering:Stable.repo"

node6.dtedu.com | SUCCESS | rc=0 >>

--2017-04-10 06:31:50--  http://download.opensuse.org/repositories/network:ha-clustering:Stable/RedHat_RHEL-6/network:ha-clustering:Stable.repo

正在解析主机 download.opensuse.org... 195.135.221.134, 2001:67c:2178:8::13

正在连接 download.opensuse.org|195.135.221.134|:80... 已连接。

已发出 HTTP 请求，正在等待回应... 301 Moved Permanently

位置：http://download.opensuse.org/repositories/network:ha-clustering:/Stable/RedHat_RHEL-6/network:ha-clustering:Stable.repo [跟随至新的 URL]

--2017-04-10 06:31:51--  http://download.opensuse.org/repositories/network:ha-clustering:/Stable/RedHat_RHEL-6/network:ha-clustering:Stable.repo

再次使用存在的到 download.opensuse.org:80 的连接。

已发出 HTTP 请求，正在等待回应... 301 Moved Permanently

位置：http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/RedHat_RHEL-6/network:ha-clustering:Stable.repo [跟随至新的 URL]

--2017-04-10 06:31:51--  http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/RedHat_RHEL-6/network:ha-clustering:Stable.repo

再次使用存在的到 download.opensuse.org:80 的连接。

已发出 HTTP 请求，正在等待回应... 200 OK

长度：345 [text/plain]

正在保存至: “network:ha-clustering:Stable.repo”


     0K                                                       100% 28.4M=0s


[root@ms.dtedu.com~]$ansible all -a "yum -y install crmsh"


1.3配置文件讲解（/etc/corosync.conf.example）


sync]# cat corosync.conf |grep -v ^# |grep -v ^$

compatibility: whitetank//是否兼容whitetank版本，就是0.8版本，兼容的话新功能不能使用。

totem {//心跳线配置模块

version: 2

# secauth: Enable mutual node authentication. If you choose to

# enable this ("on"), then do remember to create a shared

# secret with "corosync-keygen".

secauth: off//是否启用安全认证模式

threads: 0//启动线程数量

# interface: define at least one interface to communicate

# over. If you define more than one interface stanza, you must

# also set rrp_mode.

interface {//定义心跳信息传递接口

                # Rings must be consecutively numbered, starting at 0.

ringnumber: 0//信息循环次数

# This is normally the *network* address of the

# interface to bind to. This ensures that you can use

# identical instances of this configuration file

# across all your cluster nodes, without having to

# modify this option.

bindnetaddr: 192.168.1.0//绑定的网络地址，用于心跳线的网卡ip地址。

# However, if you have multiple physical network

# interfaces configured for the same subnet, then the

# network address alone is not sufficient to identify

# the interface Corosync should bind to. In that case,

# configure the *host* address of the interface

# instead:

# bindnetaddr: 192.168.1.1

# When selecting a multicast address, consider RFC

# 2365 (which, among other things, specifies that

# 239.255.x.x addresses are left to the discretion of

# the network administrator). Do not reuse multicast

# addresses across multiple Corosync clusters sharing

# the same network.

mcastaddr: 224.5.5.5//组播地址

# Corosync uses the port you specify here for UDP

# messaging, and also the immediately preceding

# port. Thus if you set this to 5405, Corosync sends

# messages over UDP ports 5405 and 5404.

mcastport: 5405//组播端口

# Time-to-live for cluster communication packets. The

# number of hops (routers) that this ring will allow

# itself to pass. Note that multicast routing must be

# specifically enabled on most network routers.

ttl: 1

}

}

logging {//定义日志信息

# Log the source file and line where messages are being

# generated. When in doubt, leave off. Potentially useful for

# debugging.

fileline: off

# Log to standard error. When in doubt, set to no. Useful when

# running in the foreground (when invoking "corosync -f")

to_stderr: no

# Log to a log file. When set to "no", the "logfile" option

# must not be set.

to_logfile: yes

logfile: /var/log/cluster/corosync.log

# Log to the system log daemon. When in doubt, set to yes.

to_syslog: yes//是否将日志信息写入的/var/log/message中，建议no

# Log debug messages (very verbose). When in doubt, leave off.

debug: off

# Log messages with time stamps. When in doubt, set to on

# (unless you are only logging to syslog, where double

# timestamps can be annoying).

timestamp: on//是否打开时间戳，可以关闭

logger_subsys {

subsys: AMF

debug: off

}

}

Service {//以模块方式运行pecemaker

ver:0

name:pacemaker

}



1.4 制作corosync通信间的安全秘钥。将authkey、corosync.cnf复制到其他节点上。

[root@node5.dtedu.com /etc/corosync]# corosync-keygen 

Corosync Cluster Engine Authentication key generator.

Gathering 1024 bits for key from /dev/random.

Press keys on your keyboard to generate entropy.

Writing corosync key to /etc/corosync/authkey.


[root@node5.dtedu.com /etc/corosync]# scp authkey corosync.conf node6:/etc/corosync/

authkey                                       100%  128     0.1KB/s   00:00    

corosync.conf                                 100% 2663     2.6KB/s   00:00    

[root@node5.dtedu.com /etc/corosync]# 


1.5关闭节点上的NetworkManager服务

[root@ms.dtedu.com~]$ansible all -a "chkconfig NetworkManager off"

node5.dtedu.com | SUCCESS | rc=0 >>



node6.dtedu.com | SUCCESS | rc=0 >>



[root@ms.dtedu.com~]$ansible all -a "service NetworkManager stop"

node5.dtedu.com | SUCCESS | rc=0 >>

Stopping NetworkManager daemon: [FAILED]


node6.dtedu.com | SUCCESS | rc=0 >>

Stopping NetworkManager daemon: [  OK  ]


1.6启动corosync服务

[root@ms.dtedu.com~]$ansible all -a "service corosync start"

node6.dtedu.com | SUCCESS | rc=0 >>

Starting Corosync Cluster Engine (corosync): [  OK  ]


node5.dtedu.com | SUCCESS | rc=0 >>

Starting Corosync Cluster Engine (corosync): [  OK  ]



1.7检查服务启动情况。

检查corosync引擎是否正常启动

[root@node5.dtedu.com /etc/corosync]# grep -e "Corosync Cluster Engine" -e "configuration file" /var/log/cluster/corosync.log 

Apr 10 10:14:23 corosync [MAIN  ] Corosync Cluster Engine ('1.4.7'): started and ready to provide service.

Apr 10 10:14:23 corosync [MAIN  ] Successfully read main configuration file '/etc/corosync/corosync.conf'.


查看初始化成员节点通知是否正常发送

[root@node5.dtedu.com /etc/corosync]# grep TOTEM /var/log/cluster/corosync.log 

Apr 10 10:14:23 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).

Apr 10 10:14:23 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).

Apr 10 10:14:24 corosync [TOTEM ] The network interface [192.168.1.23] is now up.

Apr 10 10:14:24 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.

Apr 10 10:14:24 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.


检查启动过程中是否有错误产生，针对资源错误进行检查

[root@node5.dtedu.com /etc/corosync]# grep ERROR: /var/log/cluster/corosync.log  |grep -v unpack_resources


检查pacemaker是否正常启动

[root@node5.dtedu.com /etc/yum.repos.d]# grep pcmk_startup /var/log/cluster/corosync.log 

Apr 10 13:17:19 corosync [pcmk  ] info: pcmk_startup: CRM: Initialized

Apr 10 13:17:19 corosync [pcmk  ] Logging: Initialized pcmk_startup

Apr 10 13:17:19 corosync [pcmk  ] info: pcmk_startup: Maximum core file size is: 18446744073709551615

Apr 10 13:17:19 corosync [pcmk  ] info: pcmk_startup: Service: 9

Apr 10 13:17:19 corosync [pcmk  ] info: pcmk_startup: Local hostname: node5.dtedu.com



查看高可用节点间的状态

[root@node6.dtedu.com /etc/yum.repos.d]# service corosync start

Starting Corosync Cluster Engine (corosync):               [  OK  ]

[root@node6.dtedu.com /etc/yum.repos.d]# crm status

Stack: classic openais (with plugin)

Current DC: node5.dtedu.com (version 1.1.15-5.el6-e174ec8) - partition with quorum

Last updated: Mon Apr 10 13:47:46 2017 Last change: Mon Apr 10 13:47:38 2017 by hacluster via crmd on node5.dtedu.com

, 2 expected votes

2 nodes and 0 resources configured


Online: [ node5.dtedu.com node6.dtedu.com ]


No resources


用来检查corosync是否有语法错误

[root@node5.dtedu.com /etc/yum.repos.d]# crm_verify -LV

   error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined

   error: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option

   error: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity

Errors found during check: config not valid