一大早接到电话,RAC数据库第二节点报00512,LISTENER无法启动,赶到单位检查系统
[oracle@cqlthxdbqz02 ~]$ lsnrctl start
Error listening on: (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
TNS-12542: TNS:address already in use
TNS-12560: TNS:protocol adapter error
TNS-00512: Address already in use
Linux Error: 98: Address already in use
1、检查CRS状态
crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
ora.FRA.dg
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
ora.LISTENER.lsnr
ONLINE OFFLINE cqlthxdbqz01
ONLINE OFFLINE cqlthxdbqz02
ora.OCR.dg
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
ora.asm
ONLINE ONLINE cqlthxdbqz01 Started
ONLINE ONLINE cqlthxdbqz02 Started
ora.gsd
OFFLINE OFFLINE cqlthxdbqz01
OFFLINE OFFLINE cqlthxdbqz02
ora.net1.network
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
ora.ons
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE cqlthxdbqz02
ora.cqlthxdbqz01.vip
1 ONLINE OFFLINE
ora.cqlthxdbqz02.vip
1 ONLINE OFFLINE
ora.cqlthxqz.db
1 OFFLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE cqlthxdbqz02 Open
ora.cvu
1 ONLINE ONLINE cqlthxdbqz02
ora.oc4j
1 ONLINE ONLINE cqlthxdbqz02
ora.scan1.vip
1 ONLINE ONLINE cqlthxdbqz02
经检查发现VIP服务有问题
cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#eth0-Public IP
162.12.1.1 cqlthxdbqz01
162.12.1.3 cqlthxdbqz02
#eth1 PRIVATE IP
192.168.1.1 cqlthxdbqz01-prvi
192.168.1.3 cqlthxdbqz02-prvi
#VIP
162.12.1.5 cqlthxdbqz01-vip
162.12.1.7 cqlthxdbqz02-vip
#SCAN
162.12.1.9 cqlthxdbqzscan
检查两节点VIP状态
[oracle@cqlthxdbqz02 ~]$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:86:27:29
inet addr:162.12.1.3 Bcast:162.12.7.255 Mask:255.255.248.0
inet6 addr: fe80::250:56ff:fe86:2729/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:123920 errors:0 dropped:0 overruns:0 frame.:0
TX packets:66614 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15172312 (14.4 MiB) TX bytes:9523726 (9.0 MiB)
eth0:1 Link encap:Ethernet HWaddr 00:50:56:86:27:29
inet addr:162.12.1.9 Bcast:162.12.7.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1 Link encap:Ethernet HWaddr 00:50:56:86:27:2A
inet addr:192.168.1.3 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe86:272a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11154490 errors:0 dropped:0 overruns:0 frame.:0
TX packets:11361273 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6036085762 (5.6 GiB) TX bytes:6710910488 (6.2 GiB)
eth1:1 Link encap:Ethernet HWaddr 00:50:56:86:27:2A
inet addr:169.254.8.126 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2039486 errors:0 dropped:0 overruns:0 frame.:0
TX packets:2039486 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:540441821 (515.4 MiB) TX bytes:540441821 (515.4 MiB)
[root@cqlthxdbqz01 ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:86:27:27
inet addr:162.12.1.1 Bcast:162.12.7.255 Mask:255.255.248.0
inet6 addr: fe80::250:56ff:fe86:2727/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1225955 errors:0 dropped:0 overruns:0 frame.:0
TX packets:1095256 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:150377995 (143.4 MiB) TX bytes:249780494 (238.2 MiB)
eth0:1 Link encap:Ethernet HWaddr 00:50:56:86:27:27
inet addr:162.12.1.7 Bcast:162.12.7.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1 Link encap:Ethernet HWaddr 00:50:56:86:27:28
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe86:2728/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11348596 errors:0 dropped:0 overruns:0 frame.:0
TX packets:10936251 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6606810678 (6.1 GiB) TX bytes:5927368221 (5.5 GiB)
eth1:1 Link encap:Ethernet HWaddr 00:50:56:86:27:28
inet addr:169.254.58.53 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:982793 errors:0 dropped:0 overruns:0 frame.:0
TX packets:982793 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:311888597 (297.4 MiB) TX bytes:311888597 (297.4 MiB)
经检查,第二节点没有VIP任何信息,第一节点只有一个第二节点的VIP,造成了CRS VIP服务出现了问题。
尝试重启集群,VIP服务仍然无法启动。只好2节点关闭数据库,停集群,重启操作系统:
su -
crsctl stop crs -f
reboot
重启2节点主机后,观察CRS状态
oracle@cqlthxdbqz01 ~]$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:86:27:27
inet addr:162.12.1.1 Bcast:162.12.7.255 Mask:255.255.248.0
inet6 addr: fe80::250:56ff:fe86:2727/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3778 errors:0 dropped:0 overruns:0 frame.:0
TX packets:2553 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:665005 (649.4 KiB) TX bytes:439290 (428.9 KiB)
eth0:2 Link encap:Ethernet HWaddr 00:50:56:86:27:27
inet addr:162.12.1.5 Bcast:162.12.7.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth0:3 Link encap:Ethernet HWaddr 00:50:56:86:27:27
inet addr:162.12.1.9 Bcast:162.12.7.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1 Link encap:Ethernet HWaddr 00:50:56:86:27:28
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe86:2728/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13495 errors:0 dropped:0 overruns:0 frame.:0
TX packets:10163 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:8927631 (8.5 MiB) TX bytes:6228089 (5.9 MiB)
eth1:1 Link encap:Ethernet HWaddr 00:50:56:86:27:28
inet addr:169.254.58.53 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:5160 errors:0 dropped:0 overruns:0 frame.:0
TX packets:5160 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4209655 (4.0 MiB) TX bytes:4209655 (4.0 MiB)
[grid@cqlthxdbqz02 ~]$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:86:27:29
inet addr:162.12.1.3 Bcast:162.12.7.255 Mask:255.255.248.0
inet6 addr: fe80::250:56ff:fe86:2729/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3115 errors:0 dropped:0 overruns:0 frame.:0
TX packets:2671 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:635163 (620.2 KiB) TX bytes:660337 (644.8 KiB)
eth0:1 Link encap:Ethernet HWaddr 00:50:56:86:27:29
inet addr:162.12.1.7 Bcast:162.12.7.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1 Link encap:Ethernet HWaddr 00:50:56:86:27:2A
inet addr:192.168.1.3 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe86:272a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:86026 errors:0 dropped:0 overruns:0 frame.:0
TX packets:36905 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:83856297 (79.9 MiB) TX bytes:20688032 (19.7 MiB)
eth1:1 Link encap:Ethernet HWaddr 00:50:56:86:27:2A
inet addr:169.254.8.126 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:17094 errors:0 dropped:0 overruns:0 frame.:0
TX packets:17094 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:8568002 (8.1 MiB) TX bytes:8568002 (8.1 MiB)
此时2节点VIP已经恢复正常,每个节点都能读取到自己的VIP
--检查CRS,资源正常
[grid@cqlthxdbqz02 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
ora.FRA.dg
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
ora.LISTENER.lsnr
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
ora.OCR.dg
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
ora.asm
ONLINE ONLINE cqlthxdbqz01 Started
ONLINE ONLINE cqlthxdbqz02 Started
ora.gsd
OFFLINE OFFLINE cqlthxdbqz01
OFFLINE OFFLINE cqlthxdbqz02
ora.net1.network
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
ora.ons
ONLINE ONLINE cqlthxdbqz01
ONLINE ONLINE cqlthxdbqz02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE cqlthxdbqz01
ora.cqlthxdbqz01.vip
1 ONLINE ONLINE cqlthxdbqz01
ora.cqlthxdbqz02.vip
1 ONLINE ONLINE cqlthxdbqz02
ora.cqlthxqz.db
1 ONLINE ONLINE cqlthxdbqz01 Open
2 ONLINE ONLINE cqlthxdbqz02 Open
ora.cvu
1 ONLINE ONLINE cqlthxdbqz01
ora.oc4j
1 ONLINE ONLINE cqlthxdbqz01
ora.scan1.vip
1 ONLINE ONLINE cqlthxdbqz01
至此2节点均恢复正常。
询问相关人员,得知昨晚由于添加存储,该RAC为OEL 6+VM,因此存储挂上虚拟机后,需重启主机方能识别存储,操作人员为了不影响业务操作,重启主机做了如下操作:
关闭第一节点实例--reboot第一节点操作系统--1节点系统启动后立即关闭第二点实例--reboot第二节点操作系统
原因看来是由于在操作reboot第一节点时未能关闭集群软件而直接关闭操作系统,导致第一节点VIP漂移到第二节点,待第二节点reboot时,操作人员没有检查集群状态,此时原VIP还未漂移回第一节点,就关闭第二节点,造成VIP在2节点发生紊乱。由于VIP在2节点信息完全不对,因此listener无法启动,第二节点总是报地址已被使用。
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/23371754/viewspace-756924/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/23371754/viewspace-756924/