ClusterXL and State Synchronization
The health of ClusterXL can be examined using a number of different commands:
cphaprob –a if
cphaprob state
cphaprob list
cpstat ha –f all | more
fw ctl pstat
Use the ‘cphaprob –a if’ command on the cluster members to check which interfaces have been configu
red for state synchronization and verify the sync mode is consistent on the cluster members:
Example output:
[Expert@Zulu]# cphaprob –a if
eth1c0 non sync(non secured)
eth2c0 non sync(non secured)
eth3c0 non sync(non secured)
eth4c0 sync(secured), multicast
Virtual cluster interfaces: 3
eth1c0 192.168.1.1
eth2c0 192.168.2.1
eth3c0 10.1.1.1
[Expert@Zulu]#
[Expert@Shaka]# cphaprob –a if
eth1c0 non sync(non secured)
eth2c0 non sync(non secured)
eth3c0 non sync(non secured)
eth4c0 sync(secured), broadcast
Virtual cluster interfaces: 3
eth1c0 192.168.1.1
eth2c0 192.168.2.1
eth3c0 10.1.1.1
[Expert@Shaka]#
In the above example, interface eth4c0 hasbeen configured on both cluster members for state sync but
the sync mode is inconsistent, one is using multicast and the other broadcast mode. Ensure the c
luster members use the same mode. (The default mode is multicast.)
The following document explains how to change between broadcast and multicast mode:
sk20576: How to set ClusterXL Control Protocol (CCP) in broadcast mode in ClusterXL
Use the ‘cphaprob state’ command to check if state sync is up and running. The local and remote
state synchronization IP addresses should be displayed and their state should be shown as ‘Active’ on
the HA Master and ‘Standby’ on the HA Backup. In a load-sharing
cluster the state should be shown as
‘Active’ on both the local and remote firewalls:
Example output - HA:
[Expert@Zulu]# cphaprob state
Cluster Mode: New High Availability (Active Up)
Number Unique Address Assigned Load State
1 (local)
1.1.1.1
100%
Active
2 1.1.1.2 0% Standby
[Expert@Zulu]#
In a HA cluster configuration (above), one member should be Active and the other Standby.
Example output – Load-Sharing:
[Expert@Dingaan]# cphaprob state
Cluster Mode: New High Availability (Active Up)
Number Unique Address Assigned Load State
1 (local)
1.1.1.3
50%
Active
2 1.1.1.4 50% Active
[Expert@Dingaan]#
In a load-sharing cluster configuration (above), both members should be shown as Active.
Example output – HA or Load-Sharing:
[Expert@Zulu]# cphaprob state
Cluster Mode: New High Availability (Active Up)
Number Unique Address Assigned Load State
1 (local) 1.1.1.1 100% Active
[Expert@Zulu]#
Remote cluster partner is missing!
If the remote partner is not shown it will be usually be due to one of the following:
There is no network connectivity between the members of the cluster on the state sync
network
The partner does not have state synchronization enabled
One partner is using broadcast mode and the other is using multicast mode
One of the monitored processes has an issue, such as no policy loaded
The partner firewall is down.
Example output - HA or Load-Sharing:
[Expert@Zulu]# cphaprob state
Cluster Mode: New High Availability (Active Up)
Number Unique Address Assigned Load State
1 (local)
1.1.1.1
100%
Active
2 1.1.1.2 0% Ready
[Expert@Zulu]#
Partner is in the ‘Ready’ state. If one of the partners is in the ‘Ready’ state it indicates that
there is an issue with state synchronization.
The ‘Ready’ state is normally caused by another member of the cluster running
a higher version of code or HFA, for example, as would happen during
an upgrade. This state is also seen when CoreXL has been configured to use a
different number of cores on the individual cluster members. For further information see:
sk42096: Cluster member with CoreXL is in 'Ready' state
The ‘Ready’ state can also occur if a cluster member receives state synchronization traffic from a
different cluster that is using the same mac magic number and the other cluster is running a
higher version of code. For further information see:
sk36913: Connecting several clusters on the same network
Example output - HA or Load-Sharing:
[Expert@Zulu]# cphaprob state
Cluster Mode: New High Availability (Active Up)
Number Unique Address Assigned Load State
1 (local)
1.1.1.1
100%
Active
2 1.1.1.2 0% Down
[Expert@Zulu]#
A remote cluster member is in the ‘Down’ state indicates
that there is either a problem on the remote member or the state synchronization network between the c
luster members is broken.
To investigate why a member shows itself to be locally ‘Down’ use the ‘cpstat ha –f all | more’ comma
nd on the firewall that shows ‘Down’. This command displays the Problem Notification Table and
the state of health of the monitored processes:
Example output (truncated):
[Expert@Zulu]# cpstat ha –f all | more
Problem Notification table
-------------------------------------------------
|Name |Status |Priority|Verified|Descr|
-------------------------------------------------
|Synchronization|OK | 0| 3383| |
|Filter |OK | 0| 3383| |
|cphad |OK | 0| 0| |
|fwd |OK | 0| 0| |
-------------------------------------------------
All monitored processes have the ‘OK’ status.
Example output (truncated):
[Expert@Shaka]# cpstat ha –f all | more
Problem Notification table
-------------------------------------------------
|Name |Status |Priority|Verified|Descr|
-------------------------------------------------
|Synchronization|problem| 0| 3383| |
|Filter |problem| 0| 3383| |
|cphad |OK | 0| 0| |
|fwd |OK | 0| 0| |
-------------------------------------------------
State synchronization is in a problem state because the policy is unloaded on this cluster member.
Installing the policy will fix this issue.
Alternatively, the ‘cphaprob list’ command displays
the same information plus some additional details:
Example output:
[Expert@Zulu]# cphaprob list
Registered Devices:
Device Name: Synchronization
Registration number: 0
Timeout: none
Current state: OK
Time since last report: 12139.6 sec
Device Name: Filter
Registration number: 1
Timeout: none
Current state: OK
Time since last report: 12124.5 sec
Device Name: cphad
Registration number: 2
Timeout: 5 sec
Current state: OK
Time since last report: 0.6 sec
Device Name: fwd
Registration number: 3
Timeout: 5 sec
Current state: OK
Time since last report: 0.6 sec
All monitored processes are shown as ‘OK’.
Assuming that state synchronization on the cluster is healthy, use the following command to check if t
he state tables are synchronized:
fw tab –t connections –s
Simultaneously execute the command on both cluster members; compare the values
of #VALS. The values
on both firewalls should be similar if the state synchronization mechanism is working unless
a lot of delayed notification is in use.
Example output:
[Expert@Zulu]# fw tab –t connections -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 3222 38026 9820
[Expert@Zulu]#
[Expert@Shaka]# fw tab –t connections -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 3187 38026 9808
[Expert@Shaka]#
The #PEAK may be different depending on the uptime and when the last peak number of connection
s occurred.
The #VALS on a HA pair should always be similar.
Examine the output of the sync section of ‘fw ctl pstat’.
Example output:
Sync: Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 13880231, retransmitted : 5, retrans reqs : 524, acks : 70
Sync packets received:
total : 692409645, were queued : 720, dropped by net : 517
retrans reqs : 5, received 43019 acks retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
Callback statistics: handled 42940 cb, average delay : 1, max delay : 4
If the dropped by net counter has incremented then some sync packets have been lost and the
problem needs to be investigated to find the cause.
For further information please refer to:
sk34476: Explanation of Sync section in the output of fw ctl pstat command
转载于:https://blog.51cto.com/jeffsoung/1681234