Oracle 12C RAC的单机Standby returning error ORA-16191

寒风凌厉大雪节气的周末冻得出不了门，坐在家中喝热茶，突然接到朋友从公司打来的电话：为Oracle 12C RAC主库安装好的单机standby DB日志无法和主库同步报错ORA-16191

DB版本：Oracle 12.1.0.2 RAC+ single DB (PS:这版本其实不稳定啦)

检查alert.log：

Error 1017 received logging on to the standby

------------------------------------------------------------

Check that the primary and standby are using a password file

and remote_login_passwordfile is set to SHARED or EXCLUSIVE, 

and that the SYS password is same in the password files.

      returning error ORA-16191

------------------------------------------------------------

FAL[client, USER]: Error 16191 connecting to CMMDB for fetching gap sequence

Sun Dec 09 11:42:34 2018

Errors in file /u01/app/oracle/diag/rdbms/cmmdbs/CMMDB/trace/CMMDB_pr00_18146.trc:

ORA-16191: Primary log shipping client not logged on standby

Sun Dec 09 11:42:44 2018

FAL[client]: Failed to request gap sequence

 GAP - thread 2 sequence 788218-788218

 DBID 3690862883 branch 928463524

FAL[client]: All defined FAL servers have been attempted.

分析：

此问题通常和remote_login_passwordfile参数，密码文件，sys账号是否有lock等相关

诊断：

1. 检查remote_login_passwordfile主备库中均设置为EXCLUSIVE，说明与此无关

2. 检查sys账号没有被lock，说明与此无关

3. 检查密码文件，朋友说：standby中是使用orapwd命令建立。这里问题就来了 ，12C联机文档及 文档 ID 1984091.1 中明确指出密码文件需求从主库RAC中pwcopy到备库。注，12.1以后版本中密码文件是所有节点共享的存放在ASM中

https://docs.oracle.com/database/121/SBYDB/log_transport.htm#SBYDB4751

If the SSL authentication requirements are not met, then each database must use a remote login password file. In an Oracle Data Guard configuration, all physical and snapshot standby databases must use a copy of the password file from the primary database, and that copy mu