基于Keepalived和NFS搭建简单高可用环境

This is a simple solution to achieve high-availability which needs 3 servers: master, backup and nfs-server.

1. Current there is a centos virtual machine of VirtualBox called centos4 as nfs-server, we can copy it to centos5 (master-server) and  centos6 (backup-server), this step can be done in VirtualBox UI, then we need to login as root and change hostname and ip address of copies

1) change hostname
$ hostname
centos4
$ hostnamectl set-hostname centos5
2) change ipaddress
$ ifconfig enp0s3
inet 192.168.55.14  netmask 255.255.192.0
$ vi /etc/sysconfig/network-scripts/ifcfg-enp0s3
Change IPADDR=192.168.55.14 to IPADDR=192.168.55.15
3) change /etc/hosts
$ vi /etc/hosts
Add 192.168.55.15 centos5
4) Check if firewall closed, otherwise close it
$ systemctl status firewalld.service
● firewalld.service
   Loaded: masked (/dev/null; bad)
   Active: inactive (dead)
$ getenforce
Disabled
5) reboot
$ reboot now

Do simliar thing on centos6, It is good idea to set colored PS1 to distinguish different server as following:

export PS1="\[\e[35m\]\w# \[\e[m\]"        # 30~37 represent different color

2. Setup NFS-server on centos4, login as root

1) install nfs-utils
$ yum install nfs-utils
2) create share directory
$ mkdir -p /var/nfs_share
$ chmod -R 755 /var/nfs_share
$ chown nfsnobody:nfsnobody /var/nfs_share
3) configure clients in /etc/exports as following

$ cat /etc/exports
/var/nfs_share_dir    centos5(rw,sync,no_root_squash)
/var/nfs_share_dir    centos6(rw,sync,no_root_squash)

4) enable and start nfs services
$ systemctl enable rpcbind
$ systemctl enable nfs-server
$ systemctl enable nfs-lock
$ systemctl enable nfs-idmap
$ systemctl start rpcbind
$ systemctl start nfs-server
$ systemctl start nfs-lock
$ systemctl start nfs-idmap

3. Setup NFS-client on centos5 and centos6

1) install nfs-utils
$ yum install nfs-utils
2) create a directory as mount point
$ mkdir -p /mnt/nfs_share
3) mount the nfs
$ mount -t nfs centos4:/var/nfs_share /mnt/nfs_share
4) veriy if mounted
$ df -h
centos4:/var/nfs_share       46G   11G   35G  25% /mnt/nfs_share
5) mount permanently, add following line to /etc/fstab
centos4:/var/nfs_share /mnt/nfs_share nfs defaults 0 0

Do simliar thing on centos6

4. Setup keepalived on centos5 (as master)

1) install keepalived
$ yum install keepalived
$ rpm -ql keepalived
/etc/keepalived
/etc/keepalived/keepalived.conf
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/lib/systemd/system/keepalived.service
...
2) configure keepalived.conf
$ ip a            # firstly get nic and subnet
inet 192.168.55.15/18 brd 192.168.63.255 scope global noprefixroute enp0s3
$ modify /etc/keepalived/keepalived.conf as following

! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id centos5
   vrrp_skip_check_adv_addr
   # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP            # BACKUP is for 'nonpreempt' option
    interface enp0s3
    virtual_router_id 51
    priority 100            # this server has higher priority than backup
    nopreempt               # doesnot preempt master role from lower priority server                     
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.55.10/18 dev enp0s3 label enp0s3:0        # virtual ip
    }
    notify_master "/etc/keepalived/notify.sh master /mnt/nfs_share_dir/ka.log"
    notify_backup "/etc/keepalived/notify.sh backup /mnt/nfs_share_dir/ka.log"
    notify_fault "/etc/keepalived/notify.sh fault /mnt/nfs_share_dir/ka.log"
}

3) enable and start keepalived
$ systemctl enable --now keepalived.service
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
$ systemctl status keepalived.service
Running
4) verify virtual ip
$ hostname -I                 # virtual ip is 192.168.55.10
192.168.55.15 192.168.55.10
$ ip a
inet 192.168.55.15/18 brd 192.168.63.255 scope global noprefixroute enp0s3
inet 192.168.55.10/18 scope global secondary enp0s3:0

5. Setup keepalived on centos6 (as backup)

1) install keepalived as 4.1
2) configure centos5 as backup
$ ip a    # firstly get nic and subnet
inet 192.168.55.16/18 brd 192.168.63.255 scope global noprefixroute enp0s3
$ modify /etc/keepalived/keepalived.conf as following

! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id centos6
   vrrp_skip_check_adv_addr
   # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s3
    virtual_router_id 51    # same to Master
    priority 50             # lower priority machine
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.55.10/18 dev enp0s3 label enp0s3:0
    }
    notify_master "/etc/keepalived/notify.sh master /mnt/nfs_share_dir/ka.log"    
    notify_backup "/etc/keepalived/notify.sh backup /mnt/nfs_share_dir/ka.log"    
    notify_fault "/etc/keepalived/notify.sh fault /mnt/nfs_share_dir/ka.log"    
}

3) enable and start keepalived
$ systemctl enable --now keepalived.service
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
$ systemctl status keepalived.service
Running
4) verify virtual ip
$ hostname -I                # there is no virtual ip
192.168.55.16

6. Test from centos04

1) packages to VRRP multicast addresses (224.0.0.18)
$ tcpdump -i enp0s3 -nn host 224.0.0.18
16:25:54.329920 IP 192.168.55.15 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20    # before switch, notice vird and prio
16:25:54.940243 IP 192.168.55.15 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 0, authtype simple, intvl 1s, length 20        # stop keepalived serivce on centos5
16:25:55.747317 IP 192.168.55.16 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20       # after switch
2) nodify shell script
shell script notify.sh has been put on /etc/keepalived/ of centos5 and centos6, and defined in keepalived.conf:

#!/bin/bash

notify() {
	body="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
	echo $body >> $2 
}

if [ $# -lt 2 ]
then
	echo "Usage: $(basename $0) {master|backup|fault} logfile"
	exit 1
fi

case $1 in
master)
	notify master $2
	;;
backup)
	notify backup $2
	;;	
fault)
	notify fault $2
	;;
*)
	echo "Usage: $(basename $0) {master|backup|fault} logfile"
	exit 1
	;;
esac

notify.sh will be invorked when state transition happening, then it will write a log to ka.log on nfs:
2023-03-17 13:50:57: vrrp transition, centos5 changed to be master
2023-03-17 13:51:16: vrrp transition, centos6 changed to be backup
2023-03-17 15:18:52: vrrp transition, centos6 changed to be master
2023-03-17 15:26:44: vrrp transition, centos5 changed to be master
2023-03-17 15:27:00: vrrp transition, centos6 changed to be backup

reference -
https://dev.to/prajwalmithun/setup-nfs-server-client-in-linux-and-unix-27id
man keepalived.conf