Oracle RAC异常进程占用端口导致VIP监听注册出现Not All Endpoints Registered

Oracle RAC VIP监听注册情况如下:

[grid@rac11gn1 ~]$ crsctl status res -t
ora.LISTENER.lsnr
               ONLINE  INTERMEDIATE rac11gn1                    Not All Endpoints Registered
               ONLINE  online       rac11gn2                    

排查流程:
1.尝试手动注册
2.检查/etc/hosts文件
3.排查local_listener参数,看参数设置是否正确
4.排查监听配置文件是否正确
5.重启集群

以上流程确认没有问题.故跟踪VIP监听器日志:

VIP监听器日志:

Error listening on: (ADDRESS=(PROTOCOL=TCP)(RATE_LIMIT=YES)(HOST=192.168.238.190)(PORT=1521)(IP=FIRST))
Error listening on: (ADDRESS=(PROTOCOL=TCP)(RATE_LIMIT=YES)(HOST=192.168.238.192)(PORT=1521))
06-JAN-2021 14:57:17 * service_register * LsnrAgt * 12542
TNS-12542: TNS:address already in use
 TNS-12560: TNS:protocol adapter error
  TNS-00512: Address already in use
   Linux Error: 98: Address already in use
    TNS-12542: TNS:address already in use
     TNS-12560: TNS:protocol adapter error
      TNS-00512: Address already in use
       Linux Error: 98: Address already in use
06-JAN-2021 14:57:18 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=rac11gn1)(USER=grid))(COMMAND=status)
(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=186647552)) * status * 0
Wed Jan 06 14:58:17 2021
06-JAN-2021 14:58:17 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=rac11gn1)(USER=grid))(COMMAND=status)
(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=186647552)) * status * 0
06-JAN-2021 14:58:17 * service_died * LsnrAgt * 12537
06-JAN-2021 14:58:17 * version * 0

这边VIP地址是不可能被占用的,进而排查端口占用情况:
查询数据库1521端口使用情况:

[root@rac11gn1 ~]# netstat -anp | grep 1521
tcp     0     0 192.168.238.190:13787   192.168.238.194:1521      ESTABLISHED 2548/ora_pmon_ora11 
tcp     0     0 192.168.238.192:58528   192.168.238.192:1521      ESTABLISHED 1625/asm_pmon_+ASM1 
tcp     0     0 192.168.238.192:58518   192.168.238.192:1521      ESTABLISHED 2548/ora_pmon_ora11 
tcp     0     0 :::1521                     :::*                        LISTEN      24027/tnslsnr       
tcp     0     0 ::ffff:192.168.238.192:1521 ::ffff:192.168.238.19:57782 ESTABLISHED 24027/tnslsnr       
tcp     0     0 ::ffff:192.168.238.192:1521 ::ffff:192.168.238.19:58518 ESTABLISHED 24027/tnslsnr       
tcp     0     0 ::ffff:192.168.238.192:1521 ::ffff:192.168.238.19:58528 ESTABLISHED 24027/tnslsnr       
tcp     0     0 ::ffff:192.168.238.192:1521 ::ffff:192.168.238.19:57776 ESTABLISHED 24027/tnslsnr    

异常进程占用1521,kill该进程:

[root@rac11gn1 ~]# kill -9 24027

[root@rac11gn1 ~]# netstat -anp | grep 1521
tcp     0    0 192.168.238.190:13787    192.168.238.194:1521  ESTABLISHED 2548/ora_pmon_ora11 
tcp     0    0 ::ffff:192.168.238.192:1521 ::ffff:192.168.238.19:58518 TIME_WAIT   -                   
tcp     0    0 ::ffff:192.168.238.192:1521 ::ffff:192.168.238.19:58528 TIME_WAIT   -                   

再次监控日志:可以正常注册.

06-JAN-2021 14:58:18 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=rac11gn1)(USER=grid))(COMMAND=status)
(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=186647552)) * status * 0
Wed Jan 06 14:59:17 2021
06-JAN-2021 14:59:17 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=rac11gn1)(USER=grid))(COMMAND=status)
(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=186647552)) * status * 0
06-JAN-2021 14:59:17 * service_died * LsnrAgt * 12537
06-JAN-2021 14:59:17 * version * 0
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.238.190)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.238.192)(PORT=1521)))
06-JAN-2021 14:59:17 * service_register * LsnrAgt * 0
06-JAN-2021 14:59:18 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=rac11gn1)(USER=grid))(COMMAND=status)
(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=186647552)) * status * 0
Dynamic address is already listened on 
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.238.192)(PORT=1521)))
06-JAN-2021 14:59:18 * service_register * ora11g1 * 0
06-JAN-2021 14:59:21 * service_update * ora11g1 * 0
06-JAN-2021 14:59:24 * service_update * ora11g1 * 0