https://www.ibm.com/developerworks/community/blogs/mhhaque/entry/configure_two_node_highly_available_cluster_using_kvm_fencing_on_rhel7?lang=en
This article will walk through the steps required to build a highly-available Apache cluster (httpd) on RHEL 7. In the Red Hat Enterprise Linux 7, the cluster stack has moved to Pacemaker/Corosync, with a new command line tool to manage the cluster (pcs
, replacing commands such as ccs
and clusvcadm
in earlier releases).
Click the below link for the Red Hat High Availability Add-On Underlying Technology Changes:
Our Cluster configuration scenario:
The cluster will be a two node cluster comprising nodes rhel7dns01 and rhel7dns02, and iSCSI shared storage will be presented from node munshi. There will be a 2GB LUN presented for shared storage.
In our scenario, munshi machine is our KVM Host & rhel7dns01 & rhel7dns02 VM Guest Host.
To verify some basic configuration on both node:
[root@rhel7dns02 nodes]# cat /etc/hosts
127.0.0.1 localhost loca
192.168.122.100 rhel
192.168.122.200 rhel
192.168.122.1 munshi.example.com munshi # KVM HOST AND iSCSI TARGET
Some Crosscheck before Starting the Cluster Configuration:
[root@rhel7dns02 nodes]# ifconfig eth2
eth2: flag
inet 192.168.122.200 netmask 255.255.255.0 broadcast 192.168.122.255
inet6 fe80
ether 52:54:00:09:dc:6a txqueuelen 1000 (Ethernet)
RX packets 26758 bytes 10359461 (9.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 16562 bytes 1892743 (1.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@rhel7dns01 ~]# cat /etc/hosts
127.0.0.1 localhost loca
192.168.122.100 rhel
192.168.122.200 rhel
192.168.122.1 munshi.example.com munshi
[root@rhel7dns01 ~]# ifconfig eth0
eth0: flag
inet 192.168.122.100 netmask 255.255.255.0 broadcast 192.168.122.255
inet6 fe80
ether 52:54:00:23:53:78 txqueuelen 1000 (Ethernet)
RX packets 27976 bytes 11457960 (10.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20403 bytes 13574743 (12.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
To confirm the firewall is running under FirewallD control on both node:
[root@rhel7dns01 nodes]# firewall-cmd --state
running
[root@rhel7dns02 nodes]# firewall-cmd --state
running
[root@rhel7dns01 ~]# firewall-cmd --permanent --ad
[root@rhel7dns01 ~]# firewall-cmd --reload
[root@rhel7dns02 nodes]# firewall-cmd --permanent --ad
[root@rhel7dns02 nodes]# firewall-cmd --reload
To install the appropriate packages for cluster setup:
[root@rhel7dns01 ~]# yum -y install pacemaker corosync fence-agents-all fence-agents-virsh fence-virt pcs dlm lvm2-cluster gfs2-utils iscs
To verify the installed packages on both node:
[root@rhel7dns02 ~]# rpm -qa|egrep "pac
pace
fenc
gfs2
coro
pace
pace
fenc
coro
dlm-
lvm2
dlm-
fenc
pace
pcs-
How to configure iSCSI Target on RHEL7 or RHEL6:
Please go to the below link for those configuration.
To configure iSCSI initiator on both node as a shared storage:
[root@rhel7dns01 ~]# cat /etc
Init
[root@rhel7dns01 ~]# systemctl start iscsi.service
[root@rhel7dns01 ~]# iscsiadm -m discovery -t st -p munshi.example.com
192.
[root@rhel7dns01 ~]# iscsiadm -m node -T iqn.
Logging in to [iface: default, target: iqn.
Login to [iface: default, target: iqn.
To Verify the Shared Disk on Both Node:
[root@rhel7dns01 ~]# fdisk -l
::::
Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Pre Activity for the Cluster Configuration:
To change or set a password for the hacluster user on both node:
[root@rhel7dns01 ~]# grep hacluster /etc/passwd
hacluster:189:189:cluster user
[root@rhel7dns01 ~]# echo password|passwd --stdin hacluster
Changing password for user hacluster.
passwd: all authentication tokens updated successfully.
[root@rhel7dns02 ~]# grep hacluster /etc/passwd
hacluster:189:189:cluster user
[root@rhel7dns02 ~]# echo password|passwd --stdin hacluster
Changing password for user hacluster.
passwd: all authentication tokens updated successfully.
To start the pcsd.service and enable this as a start-up service:
[root@rhel7dns01 ~]# systemctl enable pcsd.service
ln -s '/us
[root@rhel7dns01 ~]# systemctl start pcsd.service
[root@rhel7dns01 ~]# systemctl is-active pcsd.service
active
[root@rhel7dns02 ~]# systemctl enable pcsd.service
ln -s '/us
[root@rhel7dns02 ~]# systemctl start pcsd.service
[root@rhel7dns02 ~]# systemctl is-active pcsd.service
active
To authenticate pcs to pcsd on nodes specified:
Syntax: pcs cluster auth [node] [...] [-u username] [-p password] [--local] [--force]
[root@rhel7dns01 ~]# pcs cluster auth rhel
Username: hacluster
Password:
rhel
rhel
To check, some file will populated under the /var/lib/pcsd directory
[root@rhel7dns01 ~]# ls -la /var/lib/pcsd/
total 24
drwx------. 2 root root 94 Apr 9 06:01 .
drwxr-xr-x. 55 root root 4096 Apr 9 05:43 ..
-rwx------. 1 root root 60 Apr 9 05:54 pcsd.cookiesecret
-rwx------. 1 root root 1224 Apr 9 05:54 pcsd.crt
-rwx------. 1 root root 1679 Apr 9 05:54 pcsd.key
-rw-r--r--. 1 root root 362 Apr 9 06:02 pcs_users.conf
-rw-------. 1 root root 132 Apr 9 06:02 tokens
[root@rhel7dns01 ~]# date
Thu Apr 9 06:03:53 EDT 2015
To Start the Cluster Configuration:
To create and configure two node cluster:
Syntax: pcs cluster setup [--start] [--local] [--enable] --name <cluster name> <nod
<udpu|udp>] [--rrpmode active|passive] [--addr0 <addr/net> [[[--mcast0 <address>] [--mcastport0 <port>] [--ttl0 <ttl>]] | [--broad‐
cast0]] [--addr1 <addr/net> [[[--mcast1 <address>] [--mcastport1 <port>] [--ttl1 <ttl>]] | [--broadcast1]]]] [--w
[--a
out>] [--consensus <timeout>] [--miss_count_const <count>] [--fail_recv_const <failures>]
[root@rhel7dns01 ~]# pcs cluster setup --start --name webcluster01 rhel
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
rhel
rhel
rhel
rhel
To enable the cluster service to start automatically at boot:
Syntax: pcs cluster setup start [--all] [node] [...]
[root@rhel7dns01 ~]# pcs cluster enable --all
rhel
rhel
[root@rhel7dns01 ~]# pcs cluster status
Cluster Status:
Last updated: Thu Apr 9 06:21:58 2015
Last change: Thu Apr 9 06:18:53 2015 via crmd on rhel
Stack: corosync
Current DC: rhel
Version: 1.1.
2 Nodes configured
0 Resources configured
PCSD Status:
rhel
rhel
Fencing Devices Configuration:
To configure the Fencing device for this cluster:
We can get details configuration of a particular fance agent using below coammand:
1. fence_virsh:
[root@rhel7dns02 /]# pcs stonith describe fence_virsh
Stonith options for: fence_scsi
ipport: TCP/UDP port to use for connection with device
ipaddr (required): IP Address or Hostname
::::
pcmk_host_check: How to determin which machines are controlled by the device.
Here we are going to use fence_virsh agent. And going to verify the communication between fence agent & KVM host:
[root@rhel7dns01 ~]# fence_virsh -a 192.168.122.1 -l root -p Mhhaque0987 -n rhel7 -o status
Status: ON
To Configure fence_virsh:
Syntax: pcs stonith create <stonith id> <stonith device type> [stonith device options]
[root@rhel7dns01 ~]# pcs stonith create fence_dns01_xvm fence_virsh ipad
[root@rhel7dns01 ~]# pcs stonith create fence_dns02_xvm fence_virsh ipad
2. fence_xvm:
We need to do some aditional package installation & configuration require for fence_xvm fenceing. Below link will help us to do that.
After that, we need to verify & configure fence_xvm fencing for our cluster. Run below command on KVM host:
[root@munshi ~]# fence_xvm -o list
rhel7 eb40
rhel7-2 6502
::::
Run below command on VM Guest machine:
[root@rhel7dns01 ~]# fence_xvm -o list
Id Name State
----
5 rhel7 running
6 rhel7-2 running
::::
rhel6.4 shut off
To verify the communication between fence agent & KVM host:
[root@rhel7dns01 ~]# fence_xvm -a 225.0.0.12 -k /etc
Status: ON
[root@rhel7dns01 ~]# fence_xvm -a 225.0.0.12 -k /etc
To configure fence_xvm:
[root@rhel7dns01 ~]# pcs stonith create fence_pcmk1_xvm fence_xvm port="rhel7" pcmk
[root@rhel7dns01 ~]# pcs stonith create fence_pcmk2_xvm fence_xvm port="rhel7-2" pcmk
Cluster Resources & Resurce Group:
1. FileSystem:
We are going to create a file-system using that shared disk::
[root@rhel7dns02 /]# fdisk /dev
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x7a795bc4.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
::::
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
On the other node, run partprobe
so that the new partition is visible without the need to reboot:
[root@rhel7dns01 ~]# partprobe
To create filesystem:
[root@rhel7dns02 /]# ls -l /dev/disk/by-id/
::::
lrwxrwxrwx. 1 root root 9 Apr 9 07:35 wwn-
lrwxrwxrwx. 1 root root 10 Apr 9 07:35 wwn-
lrwxrwxrwx. 1 root root 9 Apr 10 05:15 /dev
[root@rhel7dns02 /]# mkfs.ext4 /dev
mke2fs 1.42.9 (28-Dec-2013)
::::
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
Temporarily mount the new filesystem on one node, and separately populate the /var/ww directory for testing, remembering to unmount once done:
Mount and create the sub directory as per the apache service required:
[root@rhel7dns02 /]# ls /var/www/
cgi-bin html error
[root@rhel7dns02 /]# mount /dev
[root@rhel7dns02 /]# mkdir /var
[root@rhel7dns02 /]# ls /var/www/
cgi-bin error html lost+found
Now, we are going to create a cluster file system resource wwwfs_rso into our existing cluster in a new resource group named httpdgrp which will be used to group the resources together as one unit:
[root@rhel7dns02 /]# pcs resource create wwwfs_rso Filesystem devi
[root@rhel7dns02 /]# pcs resource show
Resource Group: httpdgrp
wwwfs_rso (ocf
[root@rhel7dns01 ~]# pcs resource show wwwfs_rso
Resource: wwwfs_rso (class=ocf provider=heartbeat type=Filesystem)
Attributes: devi
Operations: start interval=0s timeout=60 (www
stop interval=0s timeout=60 (www
monitor interval=20 timeout=40 (www
Now, we are going to configure a cluster virtual IP resource vip_rso into our existing cluster in a existing resource group named httpdgrp.
2. Virtual IP or Service IP Address:
Add an IPaddr2
address resource – will be the floating virtual IP address that will failover between the nodes.
[root@rhel7dns02 /]# pcs resource create vip_rso IPaddr2 ip=192.168.122.111 cidr_netmask=24 --group httpdgrp
[root@rhel7dns01 ~]# pcs resource show vip_rso
Resource: vip_rso (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.122.111 cidr_netmask=24
Operations: start interval=0s timeout=20s (vip
stop interval=0s timeout=20s (vip
monitor interval=10s timeout=20s (vip
3. Adding Application (Web Server)
The Apache cluster health check uses the Apache server-status
handler, so add the following to your httpd.conf
[root@rhel7dns01 ~]# tail /etc
# Supplemental configuration
#
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Location>
Cluster will monitor Apache Service Up & Running using the Service Status.
[root@rhel7dns02 /]# pcs resource create httpd_rso apache conf
[root@rhel7dns02 /]# pcs resource show
Resource Group: httpdgrp
wwwfs_rso (ocf
vip_rso (ocf
httpd_rso (ocf
Open the firewall on both nodes to allow HTTP access from the client:
[root@rhel7dns01 ~]# firewall-cmd --add-service=http --permanent
[root@rhel7dns01 ~]# firewall-cmd --reload
HA Cluster Checking & Fail-over Testing:
[root@rhel7dns02 /]# pcs status
Cluster name: webcluster01
Last updated: Fri Apr 10 09:02:26 2015
Last change: Fri Apr 10 09:01:26 2015 via cibadmin on rhel
Stack: corosync
Current DC: rhel
Version: 1.1.12-a14efad
2 Nodes configured
5 Resources configured
Online: [ rhel
Full list of resources:
Resource Group: httpdgrp
wwwfs_rso (ocf
vip_rso (ocf
httpd_rso (ocf
fence_dns01_xvm (sto
fence_dns02_xvm (sto
PCSD Status:
rhel
rhel
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
To move resource group to other node:
[root@rhel7dns01 ~]# pcs resource move httpdgrp rhel
[root@rhel7dns01 ~]# pcs status
Cluster name: webcluster01
Last updated: Fri Apr 10 09:04:52 2015
Last change: Fri Apr 10 09:04:51 2015 via cibadmin on rhel
Stack: corosync
Current DC: rhel
Version: 1.1.12-a14efad
2 Nodes configured
5 Resources configured
Online: [ rhel
Full list of resources:
Resource Group: httpdgrp
wwwfs_rso (ocf
vip_rso (ocf
httpd_rso (ocf
fence_dns01_xvm (sto
fence_dns02_xvm (sto
PCSD Status:
rhel
rhel
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@rhel7dns01 ~]# pcs status
Cluster name: webcluster01
Last updated: Fri Apr 10 09:04:55 2015
Last change: Fri Apr 10 09:04:51 2015 via cibadmin on rhel
Stack: corosync
Current DC: rhel
Version: 1.1.12-a14efad
2 Nodes configured
5 Resources configured
Online: [ rhel
Full list of resources:
Resource Group: httpdgrp
wwwfs_rso (ocf
vip_rso (ocf
httpd_rso (ocf
fence_dns01_xvm (sto
fence_dns02_xvm (sto
PCSD Status:
rhel
rhel
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
Hope this document will help you to configure two node cluster for your application on KVM or PowerKVM. Please feel free to contact with us for any further concern.
Questioning is the best way to learn. So always, You all are welcome to leave your comments.