check if private interconnect CRS can trans net heartbeat-1445075.1

最新推荐文章于 2021-04-18 19:23:25 发布

cuilian2013

最新推荐文章于 2021-04-18 19:23:25 发布

阅读量117

点赞数

Node reboot or eviction: How to check if your private interconnect CRS can transmit network heartbeats (文档 ID 1445075.1)

In this Document

Goal

Fix

Applies to:

Oracle Server - Enterprise Edition - Version 10.1.0.2 to 11.2.0.3 [Release 10.1 to 11.2]
Information in this document applies to any platform.

Goal

Frequently, in the case of node reboots, the log of the CSS daemon processes (ocssd.log) indicates that the network heartbeat from one or more remote nodes was not received (for example, the message "CRS-1610:Network communication with node xxxxxx (3) missing for 90% of timeout interval. Removal of this node from cluster in 2.656 seconds" appears in the ocssd.log), and that the node subsequently was rebooted (to avoid a split brain or because it was evicted by another node).

The script. in here performs the network connectivity check using ssh. This check complements ping or traceroute since ssh uses TCP protocol while ping uses ICMP and traceroute in Linux/Unix uses UDP (traceroute on Windows use ICMP).

The network communication involves both the actual physical connection and the OS layer such as IP, UDP, and TCP.

CRS (10g and 11.1) uses TCP to communicate, so using ssh to test the connection as well as TCP and IP layer is a better test than ping or traceroute.

Because CRS on 11.2 uses UDP to communicate, using ssh to test TCP is not the optimal test, but this test will complement the traceroute test.

The script. tests the private interconnect once every 5 seconds, so this script. will put an insignificant load on the server.

Fix

1) Create a file in a location of your choice and copy and paste the lines in the following note box:

#!/bin/ksh

export TODAY=`date "+%Y%m%d"`
while [ $TODAY -lt ] # format needs to be YearMonthDate
do
export TODAY=`date "+%Y%m%d"`
export LOGFILE=/interconnect_test_${TODAY}.log
ssh "hostname; date" >> $LOGFILE 2>&1
ssh "hostname; date" >> $LOGFILE 2>&1

echo "" >> $LOGFILE
echo "" >> $LOGFILE

sleep 5
done

2) Replace with real private interconnect IP address or private interconnect host name. The script. will execute the commands, "hostname" and "date", and output to a log file.

3) If there are more than two nodes in the cluster, add more lines to issue
sh "hostname; date" >> $LOGFILE 2>&1
Make sure that this script. issues ssh to every node including the local node.

4) Replace with a real directory name where the output of this script. will go.
The script. will likely grow less than one MB every day, so you do not need large amount of space.
You can also regularly delete old log files.

5) Replace with the date and year that you want the script. to stop running. The format has to be YearMonthDate like 20121231 for December 31, 2012.

6) Save the file and issue "chmod +x " to make the script. executable.

7) Make sure that the ssh works without asking for any password over the private interconnect.
It is best to first test the ssh connection over the private interconnect from all nodes to every other node including itself (local node).

8) Issue "nohup &" to run the script. in background.
Run this script. from every node in the cluster.

===============================================

How to interpret the output in the log file:

When there is a problem with the private interconnect or when the node is down, the date shown in log file will not be once every 5 seconds but longer.

If the difference is more than 10 seconds between succeeding dates when the script. was running, then the network/server is having serious delay in transmitting network heartbeats. If the difference is greater than 30 seconds, the node will reboot, so you will likely not see the difference that is greater than 30 seconds.

Find out the approximate time that the node is rebooted and check when the script. show last output before the node is rebooted. If the time difference is more than 15 seconds, then the network problem is the cause of the missing network heartbeats. Investigate the reason that ssh (a regular OS command) hang.

==============================================

The following script. is an example from the three node cluster:

#!/bin/ksh

export TODAY=`date "+%Y%m%d"`
while [ $TODAY -lt 20121231 ] # format needs to be YearMonthDate
do
export TODAY=`date "+%Y%m%d"`
export LOGFILE=/tmp/interconnect_test_${TODAY}.log
ssh drrac1-priv "hostname; date" >> $LOGFILE 2>&1
ssh drrac2-priv "hostname; date" >> $LOGFILE 2>&1
ssh drrac3-priv "hostname; date" >> $LOGFILE 2>&1

echo "" >> $LOGFILE
echo "" >> $LOGFILE

sleep 5
done

Database - RAC/Scalability Community
To discuss this topic further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Database - RAC/Scalability Community