Corosync+Pacemaker+NFS+Mysql高可用集群部署

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/Field_Yang/article/details/81591928

Corosync+Pacemaker+NFS+Mysql高可用集群部署(使用资源管理工具crmsh配置

框架:crmsh(Corosync+pacemaker)+nfs+mysql

集群节点1:192.168.88.132 cen7.field.com

集群节点2:192.168.88.133 node2.field.com

集群节点3:192.168.88.135 node3.field.com

vip: 192.168.88.188  资源代理:ocf:heartbeat:IPaddr

nfs服务器:node3.field.com  资源代理:ocf:heartbeat:Filesystem

mysql服务器:cen7.field.com node1.field.com  资源代理:lsb:mysqld

 

配置集群的前提

(1)、时间同步;

(2)、基于当前正在使用的主机名互相访问;

(3)、是否会用到仲裁设备;

参考 :《使用资源管理工具pcs配置Corosync+pacemaker》

 

一、配置NFS服务器:共享目录作为mysqldatadir

该步骤中:主要通过创建逻辑卷并设置开机自动挂载到nfs共享目录,用于存放mysql数据库文件,其所有者必须是mysql用户和mysql组。

1、创建逻辑卷

1)、fdisk创建物理卷分区

[root@node3 ~]# fdisk  /dev/sda

欢迎使用 fdisk (util-linux 2.23.2)。

更改将停留在内存中,直到您决定将更改写入磁盘。

使用写入命令前请三思。

命令(输入 m 获取帮助):p

磁盘 /dev/sda:32.2 GB, 32212254720 字节,62914560 个扇区

Units = 扇区 of 1 * 512 = 512 bytes

扇区大小(逻辑/物理):512 字节 / 512 字节

I/O 大小(最小/最佳):512 字节 / 512 字节

磁盘标签类型:dos

磁盘标识符:0x00057c95

   设备 Boot      Start         End      Blocks   Id  System

/dev/sda1   *        2048     1075199      536576   83  Linux

/dev/sda2         1075200     3174399     1049600   82  Linux swap / Solaris

/dev/sda3         3174400    36728831    16777216   83  Linux

命令(输入 m 获取帮助):n

Partition type:

   p   primary (3 primary, 0 extended, 1 free)

   e   extended

Select (default e): p

已选择分区 4

起始 扇区 (36728832-62914559,默认为 36728832):

将使用默认值 36728832

Last 扇区, +扇区 or +size{K,M,G} (36728832-62914559,默认为 62914559):+2G

分区 4 已设置为 Linux 类型,大小设为 2 GiB

命令(输入 m 获取帮助):t

分区号 (1-4,默认 4):4

Hex 代码(输入 L 列出所有代码)8e  -->说明:用作物理卷的分区id8e

已将分区“Linux”的类型更改为“Linux LVM”

命令(输入 m 获取帮助):w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: 设备或资源忙.

The kernel still uses the old table. The new table will be used at

the next reboot or after you run partprobe(8) or kpartx(8)

正在同步磁盘。

2)、使用partprobe让内核识别新建的分区

[root@node3 ~]# partprobe /dev/sda

3)、查看新建立的分区

[root@node3 ~]# fdisk -l

磁盘 /dev/sda:32.2 GB, 32212254720 字节,62914560 个扇区

Units = 扇区 of 1 * 512 = 512 bytes

扇区大小(逻辑/物理):512 字节 / 512 字节

I/O 大小(最小/最佳):512 字节 / 512 字节

磁盘标签类型:dos

磁盘标识符:0x00057c95

   设备 Boot      Start         End      Blocks   Id  System

/dev/sda1   *        2048     1075199      536576   83  Linux

/dev/sda2         1075200     3174399     1049600   82  Linux swap / Solaris

/dev/sda3         3174400    36728831    16777216   83  Linux

/dev/sda4        36728832    40923135     2097152   8e  Linux LVM  -->标记为LVM

4)、创建物理卷

[root@node3 ~]# pvcreate /dev/sda4

  Physical volume "/dev/sda4" successfully created.

5)、创建卷组 

[root@node3 ~]# vgcreate myvg /dev/sda4

  Volume group "myvg" successfully created

6)、创建大小为500M的逻辑卷 

[root@node3 ~]# lvcreate -L 500M -n lv_mydata myvg

  Logical volume "lv_mydata" created.

7)、格式化逻辑卷

[root@node3 ~]# mke2fs -j /dev/myvg/lv_mydata

mke2fs 1.42.9 (28-Dec-2013)

文件系统标签=

OS type: Linux

块大小=1024 (log=0)

分块大小=1024 (log=0)

Stride=0 blocks, Stripe width=0 blocks

128016 inodes, 512000 blocks

25600 blocks (5.00%) reserved for the super user

第一个数据块=1

Maximum filesystem blocks=67633152

63 block groups

8192 blocks per group, 8192 fragments per group

2032 inodes per group

Superblock backups stored on blocks:

8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: 完成                           

正在写入inode表: 完成                           

Creating journal (8192 blocks): 完成

Writing superblocks and filesystem accounting information: 完成

2、挂载共享存储

1)、创建挂载的目录/mydata作为nfs共享存储介质

[root@node3 ~]# mkdir /mydata

2)、设置开机自动挂载

[root@node3 ~]# vim /etc/fstab

/dev/myvg/lv_mydata     /mydata    ext3    defaults    0 0

3)、挂载测试挂载成功与否

[root@node3 ~]# mount -a

[root@node3 ~]# ls /mydata/

lost+found

3、启动nfs服务并配置共享存储

1)、启动nfs服务

[root@node3 ~]# systemctl start nfs.service

[root@node3 ~]# systemctl status nfs.service

● nfs-server.service - NFS server and services

   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)

   Active: active (exited) since 六 2018-08-04 03:43:47 CST; 6s ago

  Process: 1442 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)

  Process: 1438 ExecStartPre=/bin/sh -c /bin/kill -HUP `cat /run/gssproxy.pid` (code=exited, status=0/SUCCESS)

  Process: 1436 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)

 Main PID: 1442 (code=exited, status=0/SUCCESS)

   CGroup: /system.slice/nfs-server.service

8月 04 03:43:47 node3.field.com systemd[1]: Starting NFS server and services...

8月 04 03:43:47 node3.field.com systemd[1]: Started NFS server and services.

2)、添加共享存储文件,用于存放mysql数据的目录

[root@node3 ~]# vim /etc/exports

#/www/hadocs 192.168.88.0/24(rw)

/mydata 192.168.88.0/24(rw,no_root_squash)

#该网段的主机可读写,不需root用户

3)、编辑测试页

[root@node3 ~]#  echo "<h1>Test page on NFS Service</h1>" >/mydata/index.html

4)、重启nfs服务并修改共享目录属组属主为mysql,注意:此步骤为必须操作

[root@node3 ~]# systemctl restart nfs.service

[root@node3 ~]# chown -R mysql.mysql /mydata

chown: 无效的用户: "mysql.mysql"

[root@node3 ~]# groupadd -g 306  -r mysql

[root@node3 ~]# useradd -g mysql -u 306 -s /sbin/nologin mysql

[root@node3 ~]# id mysql

uid=306(mysql) gid=306(mysql) 组=306(mysql)

[root@node3 ~]# chown -R mysql.mysql /mydata

 

二、编译、安装、配置、测试Mariadb

该步骤主要分两大步:cen7节点安装和node2节点安

(一)、cen7节点上的编译、安装、配置、测试Mariadb

1、安装前配置

1)、创建mysql用户、组和挂载目录

[root@cen7 ~]# groupadd -g 306 -r mysql

[root@cen7 ~]# useradd -u 306 -g mysql -r mysql

[root@cen7 ~]# mkdir /mydata

2)、挂载NFS文件系统并确认

[root@cen7 ~]# mount -t nfs 192.168.88.135:/mydata /mydata

[root@cen7 ~]# mount |grep nfs

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)

192.168.88.135:/mydata on /mydata type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.88.132,local_lock=none,addr=192.168.88.135)

[root@cen7 ~]# ls /mydata

index.html  lost+found

3)、以mysql用户身份创建data目录,作为初始化时的数据存放目录(datadir

注意:mysql初始化只需在集群节点中的一个进行一次,后续集群会自动挂载配置,另一台机mariadb不需初始化,所以不用以mysql用户创建data目录

[root@cen7 ~]# su - mysql

su: 警告:无法更改到 /home/mysql 目录: 没有那个文件或目录

-bash-4.2$ cd /mydata

-bash-4.2$ mkdir data

-bash-4.2$ exit

登出

2、源码编译安装Mariadb

说明:此处采用10.0以上版本可能会因为c++版本低编译无法通过,笔者采用mariadb-5.5.61.tar.gz安装包

[root@cen7 ~]# ls

anaconda-ks.cfg                     mariadb-5.5.61.tar.gz    python-parallax-1.0.0a1-7.1.noarch.rpm

crmsh-3.0.0-6.2.noarch.rpm          ~None                      python-pssh-2.3.1-7.3.noarch.rpm

crmsh-scripts-3.0.0-6.2.noarch.rpm  pssh-2.3.1-7.3.noarch.rpm

1)、编译安装mariadb

[root@cen7 ~]# tar xf mariadb-5.5.61.tar.gz -C /usr/local

[root@cen7 ~]# cd /usr/local/

[root@cen7 local]# ln -sv mariadb-5.5.61 mysql

"mysql" -> "mariadb-5.5.61"

[root@cen7 local]# cd mysql

注意:修改文件属主属组为mysql

[root@cen7 mysql]# chown -R mysql:mysql .

[root@cen7 mysql]# cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/mydata/data/ -DSYSCONFDIR=/etc  -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_READLINE=1 -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_LIBWRAP=0 -DMYSQL_UNIX_ADDR=/tmp/mysql.sock -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci

[root@cen7 mysql]# make && make install

[root@cen7 mysql]# chown -R mysql:mysql  .

2)、初始化mysql数据库:只需在该节点进行初始化,后续会自动挂载。

[root@cen7 mysql]# scripts/mysql_install_db --user=mysql --datadir=/mydata/data

Installing MariaDB/MySQL system tables in '/mydata/data' ...

180803  6:02:53 [Note] ./bin/mysqld (mysqld 5.5.61-MariaDB) starting as process 60647 ...

OK

Filling help tables...

180803  6:02:55 [Note] ./bin/mysqld (mysqld 5.5.61-MariaDB) starting as process 60656 ...

OK

To start mysqld at boot time you have to copy

support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !

To do so, start the server, then issue the following commands:

'./bin/mysqladmin' -u root password 'new-password'

'./bin/mysqladmin' -u root -h cen7.field.com password 'new-password'

Alternatively you can run:

'./bin/mysql_secure_installation'

which will also give you the option of removing the test

databases and anonymous user created by default.  This is

strongly recommended for production servers.

See the MariaDB Knowledgebase at http://mariadb.com/kb or the

MySQL manual for more instructions.

You can start the MariaDB daemon with:

cd '.' ; ./bin/mysqld_safe --datadir='/mydata/data'

You can test the MariaDB daemon with mysql-test-run.pl

cd './mysql-test' ; perl mysql-test-run.pl

Please report any problems at http://mariadb.org/jira

The latest information about MariaDB is available at http://mariadb.org/.

You can find additional information about the MySQL part at:

http://dev.mysql.com

Consider joining MariaDB's strong and vibrant community:

https://mariadb.org/get-involved/

[root@cen7 mysql]# chown -R root  .

3)、复制my-large.cnf为mysql主配置文件并修改

[root@cen7 mysql]#  cp support-files/my-large.cnf  /etc/my.cnf

cp:是否覆盖"/etc/my.cnf"? y

[root@cen7 mysql]# vim /etc/my.cnf

#thread_concurrency=CPU*2

thread_concurrency = 6

#指定mysql数据文件的存放位置:

datadir = /mydata/data

4)、配置使用service管理mysqld服务并关闭开机自启动,用CRM控制mysql启停

[root@cen7 mysql]# cp support-files/mysql.server  /etc/rc.d/init.d/mysqld

[root@cen7 mysql]# chkconfig --add mysqld

[root@cen7 mysql]# chkconfig mysqld off

[root@cen7 mysql]# service mysqld start

env: /etc/init.d/mysqld: 权限不够

[root@cen7 mysql]# chmod +x /etc/rc.d/init.d/mysqld

3、启动mysql服务并确认状态

[root@cen7 mysql]# service mysqld start

Starting MariaDB.180803 06:08:41 mysqld_safe Logging to '/mydata/data/cen7.field.com.err'.

180803 06:08:41 mysqld_safe Starting mysqld daemon with databases from /mydata/data

. SUCCESS!

[root@cen7 mysql]# ps -ef | grep 3306

mysql    61180 60874  1 06:08 pts/0    00:00:00 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/mydata/data --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=/mydata/data/cen7.field.com.err --pid-file=/mydata/data/cen7.field.com.pid --socket=/tmp/mysql.sock --port=3306

root     61233  3454  2 06:09 pts/0    00:00:00 grep --color=auto 3306

[root@cen7 mysql]# service mysqld stop

Shutting down MariaDB..... SUCCESS!

4、针对用户操作习惯的更改

使用源码包编译安装mariadb使用方法与用户习惯有所出入,通过创建链接,配置环境变量可以使mysql符合用户操作习惯,此步可以不进行。

1)、输出mysql的man手册至man命令的查找路径

[root@cen7 mysql]# vim /etc/man.config

MANPATH  /usr/local/mysql/man

2)、将mysql的头文件链接至系统头文件路径/usr/include

[root@cen7 mysql]# ln -sv /usr/local/mysql/include  /usr/include/mysql

"/usr/include/mysql" -> "/usr/local/mysql/include"

3)、输出mysql的库文件给系统库查找路径

[root@cen7 mysql]# echo '/usr/local/mysql/lib' > /etc/ld.so.conf.d/mysql.conf

4)、让系统重新载入系统库

[root@cen7 mysql]# ldconfig

5)、编辑/etc/profile添加mysql环境,让系统所有用户可以直接使用mysql的相关命令:

[root@cen7 mysql]# vim /etc/profile

MYSQL_HOME=/usr/local/mysql

PATH=$PATH:$MYSQL_HOME/bin

6)、使配置生效

[root@cen7 mysql]# . /etc/profile

5、验证mysql是否安装成功:启动mysql,连接mysql客户端。

[root@cen7 mysql]# service mysqld start

Starting MariaDB.180803 06:12:02 mysqld_safe Logging to '/mydata/data/cen7.field.com.err'.

180803 06:12:02 mysqld_safe Starting mysqld daemon with databases from /mydata/data

.. SUCCESS!

[root@cen7 mysql]# mysql

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 2

Server version: 5.5.61-MariaDB Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> quit

Bye

6、复制配置文件到node2节点:mysql配置文件各节点相同

[root@cen7 corosync]# scp /etc/my.cnf node2:/etc/my.cnf

[root@cen7 corosync]# service mysqld stop

Shutting down MariaDB.... SUCCESS!

7、卸载共享目录:最后必须卸载共享目录

[root@cen7 corosync]# umount /mydata

(二)、node2.field.com上的配置Mariadb

node2节点上mariadb安装配置同cen7节点,只是不需要再次初始化数据库,此处不再针对操作做出说明。

以下是操作步骤及代码:

[root@node2 ~]# tar xf mariadb-5.5.61.tar.gz -C /usr/local/

[root@node2 ~]# groupadd -g 306 -r mysql

[root@node2 ~]# useradd -u 306 -g mysql -r mysql

[root@node2 ~]# mkdir /mydata

[root@node2 ~]# cd /usr/local/

[root@node2 local]# ln -sv mariadb-5.5.61 mysql

"mysql" -> "mariadb-5.5.61"

[root@node2 local]# cd mysql

[root@node2 mysql]# chown -R mysql:mysql  .

[root@node2 mysql]# yum install cmake openssl-devel zlib-devel ncurses-devel bision gcc gcc-c++ -y

[root@node2 mysql]# cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/mydata/data/ -DSYSCONFDIR=/etc  -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_READLINE=1 -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_LIBWRAP=0 -DMYSQL_UNIX_ADDR=/tmp/mysql.sock -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci

[root@node2 mysql]# make && make install

[root@node2 mysql]# chown -R root:mysql  .

[root@node2 mysql]#  cp support-files/mysql.server  /etc/rc.d/init.d/mysqld

[root@node2 mysql]# chkconfig --add mysqld

[root@node2 mysql]# chkconfig mysqld off

[root@node2 mysql]# vim /etc/man.config

[root@node2 mysql]# ln -sv /usr/local/mysql/include  /usr/include/mysql

"/usr/include/mysql" -> "/usr/local/mysql/include"

[root@node2 mysql]# echo '/usr/local/mysql/lib' > /etc/ld.so.conf.d/mysql.conf

[root@node2 mysql]# ldconfig

[root@node2 mysql]# vim /etc/profile

MYSQL_HOME=/usr/local/mysql

PATH=$PATH:$MYSQL_HOME/bin

[root@node2 mysql]# vim /etc/profile

[root@node2 mysql]# . /etc/profile

[root@node2 mysql]# systemctl start mysqld.service

Failed to start mysqld.service: Unit not found.

[root@node2 mysql]# systemctl start mysql.service

Failed to start mysql.service: Unit not found.

[root@node2 mysql]# chmod +x /etc/rc.d/init.d/mysqld

[root@node2 mysql]# service mysqld start

Starting MariaDB.180803 06:49:01 mysqld_safe Logging to '/mydata/data/node2.field.com.err'.

180803 06:49:01 mysqld_safe Starting mysqld daemon with databases from /mydata/data

/usr/local/mysql/bin/mysqld_safe_helper: Can't create/write to file '/mydata/data/node2.field.com.err' (Errcode: 2)

 ERROR!

#此报错是因为未挂载mysql数据存放目录

挂载/mydata目录,验证Mariadb是否安装成功

[root@node2 corosync]# mount -t nfs 192.168.88.135:/mydata /mydata

[root@node2 corosync]# service mysqld start

Starting MariaDB.180803 07:04:16 mysqld_safe Logging to '/mydata/data/node2.field.com.err'.

180803 07:04:16 mysqld_safe Starting mysqld daemon with databases from /mydata/data

 SUCCESS!

[root@node2 corosync]# mysql

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 2

Server version: 5.5.61-MariaDB Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

MariaDB [(none)]> quit

Bye

#可以看到:挂载/mydata目录后,Mariadb客户端接入成功。

注意:测试成功后关闭mariadb并卸载共享目录

[root@node2 corosync]# service mysqld stop

[root@node2 corosync]# umount /mydata

 

三、使用crmsh配置Corosync+Pacemaker

1、将集群节点加入/etc/ansible/hosts中,便于管理

#或使用systemctl start|stop|restart xxxx

[root@cen7 corosync]# vim /etc/ansible/hosts

[hamysql]

192.168.88.132

192.168.88.133

2、启动corosyncpacemaker,删除此前配置

[root@cen7 mysql]# ansible hamysql -m service -a 'name=corosync state=started'

[root@cen7 mysql]# ansible hamysql -m service -a 'name=pacemaker state=started'

[root@cen7 mysql]# crm status

Stack: corosync

Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum

Last updated: Sat Aug  4 04:33:13 2018

Last change: Sat Aug  4 01:03:41 2018 by root via cibadmin on cen7.field.com

2 nodes configured

3 resources configured

Online: [ cen7.field.com node2.field.com ]

Full list of resources:

 Resource Group: mycluster

     mysqlip  (ocf::heartbeat:IPaddr):  Started node2.field.com

     mynfs     (ocf::heartbeat:Filesystem):   Stopped

     mysqld   (lsb:mysqld):    Stopped

使用crm交互模式:删除此前定义的资源

[root@cen7 mysql]# crm

crm(live)# resource

crm(live)resource# help

#删除资源之前需要先停止资源,各命令用法可以使用“help [command]”查看

crm(live)resource# help stop

crm(live)resource# show

 Resource Group: mycluster

     mysqlip  (ocf::heartbeat:IPaddr):  Started

     mynfs     (ocf::heartbeat:Filesystem):   Stopped

     mysqld   (lsb:mysqld):    Stopped

crm(live)resource# stop mysqlip

crm(live)resource# show

 Resource Group: mycluster

     mysqlip  (ocf::heartbeat:IPaddr):  Stopped (disabled)

     mynfs     (ocf::heartbeat:Filesystem):   Stopped

     mysqld   (lsb:mysqld):    Stopped

crm(live)resource# cd

crm(live)# configure

crm(live)configure# help delete

crm(live)configure# delete mycluster

crm(live)configure# delete mysqlip

crm(live)configure# delete mynfs

crm(live)configure# delete mysqld

crm(live)configure# verify

ERROR: Warnings found during check: config may not be valid

#注意:如果有无意义的配置会报如上error,使用show查看配置,如此处的cluster-name

#直接使用edit编辑删除无效配置:wq保存退出即可。

crm(live)configure# show

node 1: cen7.field.com \

         attributes standby=off

node 2: node2.field.com \

         attributes standby=off

property cib-bootstrap-options: \

         have-watchdog=false \

         dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \

         cluster-infrastructure=corosync \

         cluster-name=mycluster \

         stonith-enabled=false \

         default-resource-stickiness=0 \

         no-quorum-policy=ignore

crm(live)configure# edit

crm(live)configure# verify

crm(live)configure# verify

#可以看到,删除无效配置后verify不会报ERROR

crm(live)configure# show

node 1: cen7.field.com \

         attributes standby=off

node 2: node2.field.com \

         attributes standby=off

property cib-bootstrap-options: \

         have-watchdog=false \

         dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \

         cluster-infrastructure=corosync \

         stonith-enabled=false \

         no-quorum-policy=ignore

crm(live)configure# commit

crm(live)configure# quit

bye

3、编辑配置文件corosync.conf

[root@cen7 corosync]# cp corosync.conf corosync.conf.bak080222

[root@cen7 corosync]# cat corosync.conf.bak080215 >corosync.conf

[root@cen7 corosync]# vim corosync.conf

[root@cen7 corosync]# grep -v '^[[:space:]]*#' corosync.conf

totem {

         version: 2

         crypto_cipher: aes128

         crypto_hash: sha1

        secauth: on

         interface {

                   ringnumber: 0

                   bindnetaddr: 192.168.88.0

                   mcastaddr: 239.188.1.188

                   mcastport: 5405

                   ttl: 1

         }

}

nodelist {        

         node {

                   ring0_addr: 192.168.88.132

                   nodeid: 1

                   }

         node {

                   ring0_addr: 192.168.88.133

                   nodeid: 2

              }

}

logging {

         fileline: off

         to_stderr: no

         to_logfile: yes

         logfile: /var/log/cluster/corosync.log

         to_syslog: no

         debug: off

         timestamp: on

         logger_subsys {

                   subsys: QUORUM

                   debug: off

         }

}

quorum {

         provider: corosync_votequorum

}

4、生成节点间通信时用到的认证密钥文件,并复制到node2节点

[root@cen7 corosync]# corosync-keygen

Corosync Cluster Engine Authentication key generator.

Gathering 1024 bits for key from /dev/random.

Press keys on your keyboard to generate entropy.

Press keys on your keyboard to generate entropy (bits = 920).

Press keys on your keyboard to generate entropy (bits = 1000).

Writing corosync key to /etc/corosync/authkey.

[root@cen7 corosync]# ls

authkey  corosync.conf  corosync.conf.bak080215  corosync.conf.bak080222  corosync.conf.example  corosync.conf.example.udpu  corosync.xml.example  uidgid.d

[root@cen7 corosync]#  scp -p authkey corosync.conf node2:/etc/corosync/

authkey                                                                                                                                                      100%  128    44.1KB/s   00:00   

corosync.conf   

5、启动corosyncpacemaker,并确认状态是否正常                                                                                        

[root@cen7 corosync]# ansible hamysql -m service -a 'name=corosync state=started enabled=yes'

[root@cen7 corosync]# ansible hamysql -m service -a 'name=pacemaker state=started enabled=yes'

1)、查看是否能够正常发出初始化成员节点通知

[root@cen7 corosync]# grep  TOTEM  /var/log/messages|tail

Aug  3 06:25:24 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:348) was formed. Members

Aug  3 06:25:27 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:352) was formed. Members

Aug  3 06:25:29 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:356) was formed. Members joined: 3

Aug  3 06:26:16 cen7 corosync[62060]: [TOTEM ] A processor failed, forming new configuration.

Aug  3 06:26:16 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:368) was formed. Members joined: 3 2 left: 3 2

Aug  3 06:26:16 cen7 corosync[62060]: [TOTEM ] Failed to receive the leave message. failed: 3 2

Aug  3 06:28:21 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:372) was formed. Members

Aug  3 06:30:14 cen7 corosync[62060]: [TOTEM ] A processor failed, forming new configuration.

Aug  3 06:30:14 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:376) was formed. Members

Aug  3 06:34:54 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:380) was formed. Members left: 2

2)、查看corosync引擎状态

[root@cen7 corosync]# grep -e "Corosync Cluster Engine" -e "configuration file" /var/log/messages|tail -n 3

Aug  4 04:16:16 cen7 systemd: Starting Corosync Cluster Engine...

Aug  4 04:16:23 cen7 corosync: Starting Corosync Cluster Engine (corosync): [  确定  ]

Aug  4 04:16:23 cen7 systemd: Started Corosync Cluster Engine.

3)、确认启动过程无ERROR产生

[root@cen7 corosync]# grep ERROR: /var/log/messages | grep -v unpack_resources

6、查看集群节点状态

1)、查看cen7节点状态:正常

[root@cen7 corosync]# systemctl status corosync.service

● corosync.service - Corosync Cluster Engine

   Loaded: loaded (/usr/lib/systemd/system/corosync.service; enabled; vendor preset: disabled)

   Active: active (running) since 六 2018-08-04 04:16:23 CST; 41min ago

     Docs: man:corosync

           man:corosync.conf

           man:corosync_overview

  Process: 765 ExecStart=/usr/share/corosync/corosync start (code=exited, status=0/SUCCESS)

 Main PID: 795 (corosync)

   CGroup: /system.slice/corosync.service

           └─795 corosync

8月 04 04:16:16 cen7.field.com systemd[1]: Starting Corosync Cluster Engine...

8月 04 04:16:23 cen7.field.com corosync[765]: Starting Corosync Cluster Engine (corosync): [  确定  ]

8月 04 04:16:23 cen7.field.com systemd[1]: Started Corosync Cluster Engine.

[root@cen7 corosync]# corosync-cfgtool -s

Printing ring status.

Local node ID 1

RING ID 0

         id      = 192.168.88.132    -->说明:此处如果为127.0.0.1则说明启动错误

#编辑/etc/hosts文件,注销127.0.0.1行,只保留ip和域名后重启Corosync即可

         status       = ring 0 active with no faults

2)、关闭stonith

[root@cen7 corosync]# crm configure property stonith-enabled=false

使用crm_verify检查配置有没问题     

[root@cen7 corosync]# crm_verify -L -V

3)、查看node2节点状态:正常

[root@node2 ~]# service corosync status

Redirecting to /bin/systemctl status corosync.service

● corosync.service - Corosync Cluster Engine

   Loaded: loaded (/usr/lib/systemd/system/corosync.service; enabled; vendor preset: disabled)

   Active: active (running) since 六 2018-08-04 04:16:23 CST; 34min ago

     Docs: man:corosync

           man:corosync.conf

           man:corosync_overview

  Process: 839 ExecStart=/usr/share/corosync/corosync start (code=exited, status=0/SUCCESS)

 Main PID: 858 (corosync)

   CGroup: /system.slice/corosync.service

           └─858 corosync

8月 04 04:16:17 node2.field.com systemd[1]: Starting Corosync Cluster Engine...

8月 04 04:16:23 node2.field.com corosync[839]: Starting Corosync Cluster Engine (corosync): [  确定  ]

8月 04 04:16:23 node2.field.com systemd[1]: Started Corosync Cluster Engine.

[root@node2 ~]# corosync-cfgtool -s

Printing ring status.

Local node ID 2

RING ID 0

         id      = 192.168.88.133

         status       = ring 0 active with no faults

4)、查看集群状态信息

[root@cen7 mysql]# crm status

Stack: corosync

Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum

Last updated: Sat Aug  4 04:45:27 2018

Last change: Sat Aug  4 04:45:18 2018 by root via cibadmin on cen7.field.com

2 nodes configured

0 resources configured

Online: [ cen7.field.com node2.field.com ]

No resources

#可以看到集群节点的启动状态:已经Online

 

四、使用crmsh定义资源,配置Corosync+Pacemaker+nfs+Mysql高可用集群

说明:笔者使用crm交互式模式定义

1、定义vip资源ocf:heartbeat:IPaddr

部分资源定义命令说明:

primitive定义资源:

monitor:定义资源监控选项:interval(检测间隔时间)timeout(监控超时时间)

[root@cen7 corosync]# crm

crm(live)# configure

#以下命令表示定义vip资源mysqlip:虚拟IP地址“192.168.88.188”,检测间隔时间为30s,超时时长为20s

crm(live)configure# primitive mysqlip ocf:heartbeat:IPaddr params ip="192.168.88.188" op monitor interval=30s timeout=20s

crm(live)configure# verify

#verify检查语法

crm(live)configure# commit

#commit提交配置

crm(live)configure# show

node 1: cen7.field.com \

         attributes standby=off

node 2: node2.field.com \

         attributes standby=off

primitive mysqlip IPaddr \

         params ip=192.168.88.188 \

         op monitor interval=30s timeout=20s

property cib-bootstrap-options: \

         have-watchdog=false \

         dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \

         cluster-infrastructure=corosync \

         stonith-enabled=false \

         no-quorum-policy=ignore

crm(live)configure# cd

crm(live)# status

Stack: corosync

Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum

Last updated: Sat Aug  4 05:07:38 2018

Last change: Sat Aug  4 05:04:36 2018 by root via cibadmin on cen7.field.com

2 nodes configured

1 resource configured

Online: [ cen7.field.com node2.field.com ]

Full list of resources:

 mysqlip  (ocf::heartbeat:IPaddr):  Started cen7.field.com

#可以看到:资源mysqlip已经启动在cen7

crm(live)# quit

bye

[root@cen7 corosync]# ip addr list | grep ens

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    inet 192.168.88.132/24 brd 192.168.88.255 scope global noprefixroute ens32

    inet 192.168.88.188/24 brd 192.168.88.255 scope global secondary ens32

#可以看到:vip已经启动在cen7    

2、定义nfs文件系统资源:文件系统资源属于ocf

部分命令及参数说明:

param资源管理参数:文件系统资源中,必须指明的参数有device(设备)、directory(挂载目录)、fstype(文件系统类型)

start启动资源选项:interval(间隔时间)、timeout(超时时间)

stop停止资源选项:interval(间隔时间)、timeout(超时时间)

monitor资源监控选项:interval(间隔时间)timeout(监控超时时间)

[root@cen7 corosync]# crm

crm(live)# configure

#以下命令表示创建nfs文件系统资源:

#"192.168.88.135:/mydata"nfs共享目录挂载到/mydata目录下,启动超时时间60s,关闭超时时间60s,监控检测间隔时间为20s,超时时长为40s

crm(live)configure# primitive mynfs ocf:heartbeat:Filesystem params device="192.168.88.135:/mydata" directory="/mydata" fstype="nfs" op start timeout=60s op stop timeout=60s op monitor interval=20s timeout=40s

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# cd

crm(live)# status

Stack: corosync

Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum

Last updated: Sat Aug  4 05:10:30 2018

Last change: Sat Aug  4 05:10:26 2018 by root via cibadmin on cen7.field.com

2 nodes configured

2 resources configured

Online: [ cen7.field.com node2.field.com ]

Full list of resources:

 mysqlip  (ocf::heartbeat:IPaddr):  Started cen7.field.com

 mynfs     (ocf::heartbeat:Filesystem):   Started node2.field.com

可以发现:资源mynfs已经定义并成功启动在node2节点上

#未配置约束时资源会顺序启动在不同节点

crm(live)# quit

bye

3、定义mysqld资源:编译安装的mysql属于lsb资源代理

#以下命令表示定义mysqld资源,检测间隔时间为30s,超时时长为30s

crm(live)configure# primitive mysql ocf:heartbeat:mysql op monitor interval=30s timeout=30s

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# cd

crm(live)# status

Stack: corosync

Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum

Last updated: Sat Aug  4 05:25:07 2018

Last change: Sat Aug  4 05:24:57 2018 by root via cibadmin on cen7.field.com

2 nodes configured

3 resources configured

Online: [ cen7.field.com node2.field.com ]

Full list of resources:

 mysqlip  (ocf::heartbeat:IPaddr):  Started cen7.field.com

 mynfs     (ocf::heartbeat:Filesystem):   Started node2.field.com

 mysqlsrv         (ocf::heartbeat:mysql):   Stopped

Failed Actions:

* mysqlsrv_start_0 on node2.field.com 'not installed' (5): call=37, status=complete, exitreason='Setup problem: couldn't find command: /usr/bin/safe_mysqld',

    last-rc-change='Sat Aug  4 05:24:59 2018', queued=0ms, exec=63ms

* mysqlsrv_start_0 on cen7.field.com 'not installed' (5): call=37, status=complete, exitreason='Setup problem: couldn't find command: /usr/bin/safe_mysqld',

    last-rc-change='Sat Aug  4 05:24:58 2018', queued=0ms, exec=64ms

crm(live)# configure

crm(live)configure# edit

#因为是编译安装的mysql,不能用ocf设置,删除mysql资源定义并:wq保存退出

改用lsb下的mysqld定义mysql资源

crm(live)configure# primitive mysqld lsb:mysqld

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# cd

crm(live)# status

Stack: corosync

Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum

Last updated: Sat Aug  4 05:26:50 2018

Last change: Sat Aug  4 05:26:44 2018 by root via cibadmin on cen7.field.com

2 nodes configured

3 resources configured

Online: [ cen7.field.com node2.field.com ]

Full list of resources:

 mysqlip  (ocf::heartbeat:IPaddr):  Started cen7.field.com

 mynfs     (ocf::heartbeat:Filesystem):   Started node2.field.com

 mysqld   (lsb:mysqld):    Starting node2.field.com

Failed Actions:

* mysqld_start_0 on cen7.field.com 'unknown error' (1): call=44, status=complete, exitreason='',

    last-rc-change='Sat Aug  4 05:26:45 2018', queued=0ms, exec=1376ms

#此错误是因为没找到datadir中的初始化文件,重启Corosync可以解决

4、定义约束:资源必须启动在同一个节点

约束说明:

colocation  #排列约束,指定哪些资源捆绑一起,在同一节点上运行

order       #指定排列约束中的资源启动顺序,该顺序和colocation顺序相反

location    #位置约束,指定资源首选在哪些节点上运行

定义约束的顺序:排列约束-->顺序约束-->位置约束

“inf:”表示黏性值无穷大,定义inf:的两个资源始终在同一节点运行。

“-inf:”表示黏性值负无穷,定义-inf:的两个资源始终不在同一节点运行。

#以下配置表示mysqlip(mynfs mysqld)间的黏性值无穷大,即三个资源始终在同一节点上运行。

crm(live)configure# colocation mysqlip_with_mynfs_and_mysqld inf: mysqlip ( mynfs mysqld)

crm(live)configure# verify

crm(live)configure# commit

#mandatory默认值,表示强制约束,强制约束mynfsmysqld的状态必须相同且均为start

crm(live)configure# order mysqld-after-mynfs mandatory: mynfs mysqld:start

crm(live)configure# verify

crm(live)configure# commit

#强制约束mysqld资源必须在mysqlip资源启动后再启动

crm(live)configure# order mysqld-after-mysqlip  mandatory: mysqlip mysqld

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# show

node 1: cen7.field.com \

         attributes standby=off

node 2: node2.field.com \

         attributes standby=off

primitive mynfs Filesystem \

         params device="192.168.88.135:/mydata" directory="/mydata" fstype=nfs \

         op start timeout=60s interval=0 \

         op stop timeout=60s interval=0 \

         op monitor interval=20s timeout=40s

primitive mysqld lsb:mysqld

primitive mysqlip IPaddr \

         params ip=192.168.88.188 \

         op monitor interval=30s timeout=20s

order mysqld-after-mynfs Mandatory: mynfs mysqld:start

order mysqld-after-mysqlip Mandatory: mysqlip mysqld

colocation mysqlip_with_mynfs_and_mysqld inf: mysqlip ( mynfs mysqld )

property cib-bootstrap-options: \

         have-watchdog=false \

         dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \

         cluster-infrastructure=corosync \

         stonith-enabled=false \

         no-quorum-policy=ignore

crm(live)configure# quit

bye

 

五、测试高可用集群配置成功与否,并检测高可用性

1、重启corosyncpacemaker

[root@cen7 corosync]# ansible hamysql -m service -a 'name=corosync state=restarted'

[root@cen7 corosync]# ansible hamysql -m service -a 'name=pacemaker state=restarted'

2、查看集群状态信息

[root@cen7 corosync]# crm status

Stack: corosync

Current DC: node2.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum

Last updated: Sat Aug  4 05:34:30 2018

Last change: Sat Aug  4 05:30:59 2018 by root via cibadmin on cen7.field.com

2 nodes configured

3 resources configured

Online: [ cen7.field.com node2.field.com ]

Full list of resources:

 mysqlip  (ocf::heartbeat:IPaddr):  Started node2.field.com

 mynfs     (ocf::heartbeat:Filesystem):   Started node2.field.com

 mysqld   (lsb:mysqld):    Started node2.field.com

3、在node2节点测试

1)、连接mysql客户端:成功

[root@node2 ~]# mysql

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 2

Server version: 5.5.61-MariaDB Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> quit

Bye

2)、VIP流转:成功

[root@node2 ~]# ip addr list | grep ens

2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    inet 192.168.88.133/24 brd 192.168.88.255 scope global noprefixroute dynamic ens33

    inet 192.168.88.188/24 brd 192.168.88.255 scope global secondary ens33

4、集群高可用性测试

1)、置node2.field.com状态为备用状态,检查能否自动切换

[root@cen7 corosync]# crm

crm(live)# node

crm(live)node# standby node2.field.com

crm(live)node# cd

crm(live)# status

Stack: corosync

Current DC: node2.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum

Last updated: Sat Aug  4 05:38:52 2018

Last change: Sat Aug  4 05:37:51 2018 by root via crm_attribute on cen7.field.com

2 nodes configured

3 resources configured

Node node2.field.com: standby

Online: [ cen7.field.com ]

Full list of resources:

 mysqlip  (ocf::heartbeat:IPaddr):  Started cen7.field.com

 mynfs     (ocf::heartbeat:Filesystem):   Started cen7.field.com

 mysqld   (lsb:mysqld):    Started cen7.field.com

crm(live)# quit

bye

2)、连接mysql测试:成功

[root@cen7 corosync]# mysql

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 2

Server version: 5.5.61-MariaDB Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mysql              |

| performance_schema |

| test               |

+--------------------+

4 rows in set (0.21 sec)

MariaDB [(none)]> quit

Bye

3)、查看VIP是否流转成功:成功

[root@cen7 corosync]# ip addr list | grep ens

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    inet 192.168.88.132/24 brd 192.168.88.255 scope global noprefixroute ens32

    inet 192.168.88.188/24 brd 192.168.88.255 scope global secondary ens32

 

附:crmsh整体配置 

[root@cen7 corosync]# crm

crm(live)# node

crm(live)node# online node2.field.com

crm(live)node# cd

crm(live)# configure

crm(live)configure# group mycluster  mysqlip  mynfs mysqld

INFO: modified order:mysqld-after-mysqlip from mysqlip to mycluster

INFO: modified colocation:mysqlip_with_mynfs_and_mysqld from mysqlip to mycluster

INFO: modified order:mysqld-after-mynfs from mynfs to mycluster

INFO: modified colocation:mysqlip_with_mynfs_and_mysqld from mynfs to mycluster

INFO: modified order:mysqld-after-mynfs from mysqld to mycluster

INFO: modified order:mysqld-after-mysqlip from mysqld to mycluster

INFO: modified colocation:mysqlip_with_mynfs_and_mysqld from mysqld to mycluster

crm(live)configure# verify

crm(live)configure# commit

crm(live)configure# show

node 1: cen7.field.com \

         attributes standby=off

node 2: node2.field.com \

         attributes standby=off

primitive mynfs Filesystem \

         params device="192.168.88.135:/mydata" directory="/mydata" fstype=nfs \

         op start timeout=60s interval=0 \

         op stop timeout=60s interval=0 \

         op monitor interval=20s timeout=40s

primitive mysqld lsb:mysqld

primitive mysqlip IPaddr \

         params ip=192.168.88.188 \

         op monitor interval=30s timeout=20s

group mycluster mysqlip mynfs mysqld

property cib-bootstrap-options: \

         have-watchdog=false \

         dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \

         cluster-infrastructure=corosync \

         stonith-enabled=false \

         no-quorum-policy=ignore

crm(live)configure# cd

crm(live)# status

Stack: corosync

Current DC: node2.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum

Last updated: Sat Aug  4 05:42:51 2018

Last change: Sat Aug  4 05:42:45 2018 by root via cibadmin on cen7.field.com

2 nodes configured

3 resources configured

Online: [ cen7.field.com node2.field.com ]

Full list of resources:

 Resource Group: mycluster

     mysqlip  (ocf::heartbeat:IPaddr):  Started cen7.field.com

     mynfs     (ocf::heartbeat:Filesystem):   Started cen7.field.com

     mysqld   (lsb:mysqld):    Started cen7.field.com

crm(live)# quit

bye

[root@cen7 corosync]#

 

 

 

 

 

阅读更多
想对作者说点什么?

博主推荐

换一批

没有更多推荐了,返回首页