RAC打Patch遭遇PRKC-1002

同事在两节点的RAC上打oneoff patch，在其中一个节点上运行opatch apply，按理应该会自动检测并且分发到另外一个节点，进行patch，但是在分发到第二个节点的时候遇到了PRKC-1002的报错，此时，如果在另外一个节点运行opatch lsinventory也会报错，检测不到原来打过哪些patch了。

我先开启opatch的debug，来查看为什么在节点2上无法运行lsinventory了:

au11qapa70tels2>export OPATCH_DEBUG=true
au11qapa70tels2>$ORACLE_HOME/OPatch/opatch lsinventory
 
...
 
parserXMLFile : start
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /ora/product/10.2.0/db_1/inventory/oneoffs/6920066/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /ora/product/10.2.0/db_1/inventory/oneoffs/6920066/etc/config/inventory.xml
parserXMLFile : start
parserXMLFile : start
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /ora/product/10.2.0/db_1/inventory/oneoffs/8350262/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /ora/product/10.2.0/db_1/inventory/oneoffs/8350262/etc/config/inventory.xml
PatchObject::PatchObject now looks for the actions file /ora/product/10.2.0/db_1/inventory/oneoffs/8350262/etc/config/actions
PatchObject::PatchObject now looks for the inventory file /ora/product/10.2.0/db_1/inventory/oneoffs/8350262/etc/config/inventory
Loading the information from inventory/filemap.
 
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
 
Locker::release()
OUISessionManager::unRegister()
Un-Registering the caller : OPatch
LsInventory::getInstance() returns
LsInventorySession failed: Unable to create patchObject
Cleaning up the directory : "/ora/product/10.2.0/db_1/.patch_storage/patch_unzip"...
 
OPatch failed with error code 73
 
au11qapa70tels2>

我们看到，是因为缺少了一些文件，如 /ora/product/10.2.0/db_1/inventory/oneoffs/6920066/etc/config/actions.xml等等。

au11qapa70tels2>ls -l /ora/product/10.2.0/db_1/inventory/oneoffs/8350262/etc/config/actions.xml
/ora/product/10.2.0/db_1/inventory/oneoffs/8350262/etc/config/actions.xml: No such file or directory
au11qapa70tels2>ls -l /ora/product/10.2.0/db_1/inventory/oneoffs/8350262/etc/config/inventory.xml
/ora/product/10.2.0/db_1/inventory/oneoffs/8350262/etc/config/inventory.xml: No such file or directory
au11qapa70tels2>ls -l /ora/product/10.2.0/db_1/inventory/oneoffs/8350262/
/ora/product/10.2.0/db_1/inventory/oneoffs/8350262/: No such file or directory

可以看到，整个/ora/product/10.2.0/db_1/inventory/oneoffs/8350262/路径都不存在。

那么，为什么在部署这个patch的时候，这个路径会在node2上没有呢？我们来看看部署patch时完整的log：

1. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>cat opatch2011-11-17_15-42-59PM.log
2. SEVERE:OPatch invoked as follows: 'lsinventory '
3. INFO:
4. Oracle Home       : /ora/product/10.2.0/db_1
5. Central Inventory : /ora/admin/oraInventory
6.    from           : /var/opt/oracle/oraInst.loc
7. OPatch version    : 10.2.0.4.3
8. OUI version       : 10.2.0.4.0
9. OUI location      : /ora/product/10.2.0/db_1/oui
10. Log file location : /ora/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2011-11-17_15-42-59PM.log
11.  
12. INFO:Starting LsInventorySession at Thu Nov 17 15:43:02 EST 2011
13. INFO:Lsinventory Output file location : /ora/product/10.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2011-11-17_15-42-59PM.txt
14.  
15. INFO:--------------------------------------------------------------------------------
16. INFO:Installed Top-level Products (4):
17.  
18. INFO:Oracle Database 10g                                                  10.2.0.1.0
19. INFO:Oracle Database 10g Release 2 Patch Set 2                            10.2.0.3.0
20. INFO:Oracle Database 10g Release 2 Patch Set 3                            10.2.0.4.0
21. INFO:Oracle Database Vault                                                10.2.0.4.0
22. INFO:There are 4 products installed in this Oracle Home.
23.  
24. INFO:
25. INFO:Interim patches (4) :
26.  
27. INFO:Patch  6672979      : applied on Tue Nov 30 20:14:18 EST 2010
28.    Created on 11 Aug 2008, 07:06:04 hrs PST8PDT
29.    Bugs fixed:
30.      6672979
31.  
32. INFO:Patch  5945060      : applied on Tue Nov 30 19:19:13 EST 2010
33.    Created on 22 Mar 2009, 22:52:01 hrs US/Pacific
34.    Bugs fixed:
35.      5945060
36.  
37. INFO:Patch  6920066      : applied on Tue Nov 30 18:22:45 EST 2010
38.    Created on 26 Jun 2008, 07:29:30 hrs US/Pacific
39.    Bugs fixed:
40.      6920066
41.  
42. INFO:Patch  8208643      : applied on Tue Nov 30 18:20:16 EST 2010
43.    Created on 23 Mar 2009, 01:12:32 hrs US/Pacific
44.    Bugs fixed:
45.      7505535, 7688543, 7319691, 8208643, 7489862
46.  
47. INFO:
48. INFO:
49. INFO:Rac system comprising of multiple nodes
50. INFO:  Local node = au11qapa50tels2
51.   Remote node = au11qapa70tels2
52.  
53. INFO:--------------------------------------------------------------------------------
54. INFO:Finishing LsInventorySession at Thu Nov 17 15:43:09 EST 2011
55. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>
56. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>
57. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>
58. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>
59. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>
60. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>
61. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>
62. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>
63. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>cat opatch2011-11-17_16-05-54PM.log
64. SEVERE:OPatch invoked as follows: 'apply '
65. INFO:
66. Oracle Home       : /ora/product/10.2.0/db_1
67. Central Inventory : /ora/admin/oraInventory
68.    from           : /var/opt/oracle/oraInst.loc
69. OPatch version    : 10.2.0.4.3
70. OUI version       : 10.2.0.4.0
71. OUI location      : /ora/product/10.2.0/db_1/oui
72. Log file location : /ora/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2011-11-17_16-05-54PM.log
73.  
74. INFO:Starting ApplySession at Thu Nov 17 16:05:57 EST 2011
75. INFO:Starting Apply Session at Thu Nov 17 16:05:57 EST 2011
76. INFO:ApplySession applying interim patch '8350262' to OH '/ora/product/10.2.0/db_1'
77. INFO:Starting to apply patch to local system at Thu Nov 17 16:05:58 EST 2011
78. INFO:Start the Apply initScript at Thu Nov 17 16:05:58 EST 2011
79. INFO:Finish the Apply initScript at Thu Nov 17 16:05:58 EST 2011
80. INFO:Patch 8350262 has no conflicts/superset wiht any other patch processed till now
81. INFO:Checking conflicts for patch: 8350262
82. INFO:Checking conflicts/supersets for patch: 8350262 with patch:8350262
83. INFO:Checking conflicts/supersets for patch: 8350262 with patch:6672979
84. INFO:Checking conflicts/supersets for patch: 8350262 with patch:5945060
85. INFO:Checking conflicts/supersets for patch: 8350262 with patch:6920066
86. INFO:Checking conflicts/supersets for patch: 8350262 with patch:8208643
87. INFO:
88. Running prerequisite checks...
89. INFO:Space Needed : 249742
90. INFO:Prereq checkPatchApplicableOnCurrentPlatform Passed on patch :8350262
91. INFO:
92. OPatch detected the node list and the local node from the inventory.  OPatch will patch the local system then propagate the patch to the remote nodes.
93.  
94. INFO:Start saving patch at Thu Nov 17 16:06:08 EST 2011
95. INFO:Finish saving patch at Thu Nov 17 16:06:08 EST 2011
96. INFO:Start backing up system for restore at Thu Nov 17 16:06:08 EST 2011
97. INFO:Backing up files and inventory (not for auto-rollback) for the Oracle Home
98. INFO:Backing up files affected by the patch '8350262' for restore. This might take a while...
99. INFO:Finish backing up system for restore at Thu Nov 17 16:06:09 EST 2011
100. INFO:Current working directory is : /ora/product/10.2.0/db_1/OPatch
101. INFO:This is a platform specific OPatch that has ocm.zip
102. INFO:The ocm location is /ora/product/10.2.0/db_1/OPatch/ocm/ocm.zip
103. INFO:
104. The Oracle Configuration Manager is installed previously in the Oracle Home.
105. INFO:
106. The Oracle Configuration Manager is already configured. OPatch does nothing.
107.  
108. INFO:Start backing up system for rollback at Thu Nov 17 16:06:10 EST 2011
109. INFO:Backing up files affected by the patch '8350262' for rollback. This might take a while...
110. INFO:Finish backing up system for rollback at Thu Nov 17 16:06:28 EST 2011
111. INFO:Start modifying the system at Thu Nov 17 16:06:28 EST 2011
112. INFO:
113. Patching component oracle.sysman.agent.core, 10.2.0.4.0a...
114. INFO:Updating jar file "/ora/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with "/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
115. INFO:Updating jar file "/ora/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with "/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
116. INFO:Updating jar file "/ora/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with "/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/RootCert.class"
117. INFO:Updating jar file "/ora/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with "/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/SecConstants.class"
118. INFO:Updating jar file "/ora/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
119. INFO:Updating jar file "/ora/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
120. INFO:Updating jar file "/ora/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/RootCert.class"
121. INFO:Updating jar file "/ora/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/SecConstants.class"
122. INFO:Finish modifying the system at Thu Nov 17 16:08:49 EST 2011
123. INFO:ApplySession adding interim patch '8350262' to inventory
124. INFO:Start saving patch to inventory at Thu Nov 17 16:08:49 EST 2011
125. INFO:Registered Encoding value is : ISO8859-1 for the file /ora/product/10.2.0/db_1/inventory/oneoffs/8350262/etc/config/actions.xml
126. INFO:Finish saving patch to inventory at Thu Nov 17 16:09:04 EST 2011
127. INFO:
128. Verifying the update...
129. INFO:Inventory check OK: Patch ID 8350262 is registered in Oracle Home inventory with proper meta-data.
130. INFO:   verifying 8 jar files.
131. INFO:Verify top-level entry oracle/sysman/eml/sec/fsc/FSWalletUtil.class using file /ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/scratch/work/FSWalletUtil.class and /ora/admin/patch/8350262/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class
132. INFO:Verify top-level entry oracle/sysman/eml/sec/rep/RepWalletUtil.class using file /ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/scratch/work/RepWalletUtil.class and /ora/admin/patch/8350262/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class
133. INFO:Verify top-level entry oracle/sysman/eml/sec/util/RootCert.class using file /ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/scratch/work/RootCert.class and /ora/admin/patch/8350262/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/RootCert.class
134. INFO:Verify top-level entry oracle/sysman/eml/sec/util/SecConstants.class using file /ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/scratch/work/SecConstants.class and /ora/admin/patch/8350262/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/SecConstants.class
135. INFO:Verify top-level entry oracle/sysman/eml/sec/fsc/FSWalletUtil.class using file /ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/scratch/work/FSWalletUtil.class and /ora/admin/patch/8350262/files//sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class
136. INFO:Verify top-level entry oracle/sysman/eml/sec/rep/RepWalletUtil.class using file /ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/scratch/work/RepWalletUtil.class and /ora/admin/patch/8350262/files//sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class
137. INFO:Verify top-level entry oracle/sysman/eml/sec/util/RootCert.class using file /ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/scratch/work/RootCert.class and /ora/admin/patch/8350262/files//sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/RootCert.class
138. INFO:Verify top-level entry oracle/sysman/eml/sec/util/SecConstants.class using file /ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/scratch/work/SecConstants.class and /ora/admin/patch/8350262/files//sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/SecConstants.class
139. INFO:Files check OK: Files from Patch ID 8350262 are present in Oracle Home.
140. INFO:Finish applying patch to local system at Thu Nov 17 16:09:18 EST 2011
141. INFO:Starting to apply patch to remote nodes 'au11qapa70tels2'  at Thu Nov 17 16:09:18 EST 2011
142. INFO:
143. Patching in rolling mode.
144.  
145. INFO:Updating nodes 'au11qapa70tels2'
146. INFO:   Apply-related files are:
147.      FP = "/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_files.txt"
148.      DP = "/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_dirs.txt"
149.      MP = "/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/make_cmds.txt"
150.      RC = "/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/remote_cmds.txt"
151.  
152. INFO:Instantiating the file "/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_files.txt" with actual path.
153. INFO:Propagating files to remote nodes...
154. INFO:Instantiating the file "/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_dirs.txt" with actual path.
155. INFO:Propagating directories to remote nodes...
156. WARNING:OUI-67188:
157. OPatch failed to copy directories to remote nodes 'au11qapa70tels2' .  Detail: Error occurred on nodes:
158. All-node error message = Error while copying files inside directory '/ora/product/10.2.0/db_1' based on listed file '/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_dirs.txt.instantiated' with excludeListFile 'null' to nodes 'au11qapa70tels2'. [PRKC-1002 : All the submitted commands did not execute successfully]
159. ----------------------------------------------------------------------------------
160. au11qapa70tels2:
161.     ###################################################################
162.     # This system is for the use of authorized users only.            #
163.     # Individuals using this computer system without authority, or in #
164.     # excess of their authority, are subject to having all of their   #
165.     # activities on this system monitored and recorded by system      #
166.     # personnel.                                                      #
167.     #                                                                 #
168.     # In the course of monitoring individuals improperly using this   #
169.     # system, or in the course of system maintenance, the activities  #
170.     # of authorized users may also be monitored.                      #
171.     #                                                                 #
172.     # Anyone using this system expressly consents to such monitoring  #
173.     # and is advised that if such monitoring reveals possible         #
174.     # evidence of criminal activity, system personnel may provide the #
175.     # evidence of such monitoring to law enforcement officials.       #
176.     ###################################################################
177.  
178.     WARNING: Access to this computer system is limited to authorised users only.
179.     Unauthorised users may be subject to prosecution under the Crimes
180.     Act or State legislation.
181.  
182.     Please note, ALL CUSTOMER DETAILS are confidential and must not be
183.     disclosed.
184. ----------------------------------------------------------------------------------
185.  
186. INFO:
187. OPatch failed to copy directories to remote nodes 'au11qapa70tels2' .
188. Do you want to proceed? [y|n]
189. INFO:Start to wait for user-input at Thu Nov 17 16:09:22 EST 2011
190. INFO:Finish waiting for user-input at Thu Nov 17 16:10:04 EST 2011
191. INFO:User Responded with: Y
192. INFO:Finish applying patch to remote nodes at Thu Nov 17 16:10:04 EST 2011
193. INFO:--------------------------------------------------------------------------------
194. INFO:The following warnings have occurred during OPatch execution:
195. INFO:1) OUI-67188:
196. OPatch failed to copy directories to remote nodes 'au11qapa70tels2' .  Detail: Error occurred on nodes:
197. All-node error message = Error while copying files inside directory '/ora/product/10.2.0/db_1' based on listed file '/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_dirs.txt.instantiated' with excludeListFile 'null' to nodes 'au11qapa70tels2'. [PRKC-1002 : All the submitted commands did not execute successfully]
198. ----------------------------------------------------------------------------------
199. au11qapa70tels2:
200.     ###################################################################
201.     # This system is for the use of authorized users only.            #
202.     # Individuals using this computer system without authority, or in #
203.     # excess of their authority, are subject to having all of their   #
204.     # activities on this system monitored and recorded by system      #
205.     # personnel.                                                      #
206.     #                                                                 #
207.     # In the course of monitoring individuals improperly using this   #
208.     # system, or in the course of system maintenance, the activities  #
209.     # of authorized users may also be monitored.                      #
210.     #                                                                 #
211.     # Anyone using this system expressly consents to such monitoring  #
212.     # and is advised that if such monitoring reveals possible         #
213.     # evidence of criminal activity, system personnel may provide the #
214.     # evidence of such monitoring to law enforcement officials.       #
215.     ###################################################################
216.  
217.     WARNING: Access to this computer system is limited to authorised users only.
218.     Unauthorised users may be subject to prosecution under the Crimes
219.     Act or State legislation.
220.  
221.     Please note, ALL CUSTOMER DETAILS are confidential and must not be
222.     disclosed.
223. ----------------------------------------------------------------------------------
224.  
225. INFO:--------------------------------------------------------------------------------
226. SEVERE:OUI-67204:OPatch Session completed with warnings.
227. INFO:Finishing ApplySession at Thu Nov 17 16:10:04 EST 2011
228. INFO:Total time spent waiting for user-input is 42 seconds.  Finish at Thu Nov 17 16:10:04 EST 2011
229. au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/cfgtoollogs/opatch>

请注意，在140行之前的log是说明在node1上的patch过程，到140行为止，说明在node1上patch是成功的。在140行之后是在node2上的patch的log。

到153行为止，在node2上分发文件的过程是没有问题的，而且在node2上检查
在/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_files.txt.instantiated中所示的3个文件都是存在的：

/ora/product/10.2.0/db_1/sysman/jlib/emCORE.jar
/ora/product/10.2.0/db_1/sysman/jlib/emd_java.jar
/ora/product/10.2.0/db_1/inventory/ContentsXML/comps.xml

在158行，开始报错了，以下的文件夹在node2上无法建立：

/ora/product/10.2.0/db_1/inventory/oneoffs/8350262
/ora/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44

该2个文件夹无法建立，所以我们在node2上运行lsinventory也会报错找不到上述文件夹中的文件。

因此，从log中我们看到，那些patch的jar 文件，只是拷贝到了node2上，连编译都还没编译（因为编译需要将执行语句从node1分发到node2上的/oneoffs/8350262目录下）。在node上，该patch是部署失败的。

那么是什么原因造成了PRKC-1002？我们在metalink上找到了一些线索：

PRKC-1002 Error When Copying Files to Remote Nodes [ID 312437.1]
=========================================
Cause
This is because the date on the local node is higher than the date on the remote node(s).
Solution
To fix this, set the times of the nodes to be the same.  You can do this with "ntpupdate < nodename > " to sync the dates.

难道是时间不一致？分别在2台机器上运行date看时间，似乎没有差异。我们再试试在同一台机器上运行date：

au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/.patch_storage/6672979_Aug_11_2008_07_06_04/rac>date;ssh au11qapa70tels2 date
Thursday, 17 November 2011  9:29:47 PM EST
###################################################################
# This system is for the use of authorized users only.            #
# Individuals using this computer system without authority, or in #
# excess of their authority, are subject to having all of their   #
# activities on this system monitored and recorded by system      #
# personnel.                                                      #
#                                                                 #
# In the course of monitoring individuals improperly using this   #
# system, or in the course of system maintenance, the activities  #
# of authorized users may also be monitored.                      #
#                                                                 #
# Anyone using this system expressly consents to such monitoring  #
# and is advised that if such monitoring reveals possible         #
# evidence of criminal activity, system personnel may provide the #
# evidence of such monitoring to law enforcement officials.       #
###################################################################
 
WARNING: Access to this computer system is limited to authorised users only.
Unauthorised users may be subject to prosecution under the Crimes
Act or State legislation.
 
Please note, ALL CUSTOMER DETAILS are confidential and must not be
disclosed.
Thursday, 17 November 2011  9:29:48 PM EST
au11qapa50tels2:TSPRD1:/ora/product/10.2.0/db_1/.patch_storage/6672979_Aug_11_2008_07_06_04/rac>

看出什么问题来了吗？是不是多了很多内容？平时运行date的时候，只有2行，而现在，你却看到了node2上的ssh banner？！

在metalink上还找到了两篇相关的文章：

Mass Deployement of 11g Agent in Cluster Environment Fails with "PRKC-1002" Due to SSH Banner [ID 1298360.1]：
=============================
Cause
SSH banner was enabled on RAC setup, due to this agent installation fails while copying files to one of the cluster node.
"$ssh node2" from node1 was returning banner right after the connection.
 
Solution
Disable the banner on both the RAC nodes and re-try the agent installation.
Once the SSH banner is disabled agent install should be successful.
and the banner can be enabled after the installation is complete on both the nodes.

How To Configure SSH for a RAC Installation [ID 300548.1]
=============
Disable banner (/etc/banner) on all cluster nodes when you
run clusterverify (cluvfy, runcluvfy)
install software
patch the system

因此，问题就出在这边了，由于设置了node的ssh banner，我估计在分发或者创建文件的时候，会通过ssh获取节点间的时间，如果有ssh banner，获取不到第一行显示的时间，因此出现上述报错。如果你搜索metelink，还会发现ssh banner会造成很多类似的问题，如gc agent无法push，cluvfy也会失败。

找到问题后，解决问题就比较简单了。而且还有不同的解决方式。

方法一：把node1上的相关目录拷贝到node2，进行回滚。回滚后，让SA取消ssh banner，重新部署patch。
方法二，也是推荐的方法，回滚之后，在node2上用opatch apply -local的方式进行patch。

总体来说，如果节点不是很多，不嫌麻烦的话，还是用-local的方式一个节点一个节点的手工部署过去来的安全。方式为：

opatch apply -local -oh < RDBMS_HOME >

如果部署多个patch：

opatch napply < patch_dir > -local -oh < RDBMS_HOME > -id < id1 > , < id2 > , < id3 > ....

原文地址：http://www.oracleblog.org/working-case/deploy-rac-patch-suffering-prkc-1002/