CRS-6706: Oracle Clusterware Release patch level ('3329448500') does not match Software patch level

oracle19.3.0.0 rac+centos7.6

补丁为19.5.0.0

启动crs报错如下：

[root@rac2 ~]# /oracle/app/19c/grid/bin/crsctl start crs
CRS-6706: Oracle Clusterware Release patch level ('3329448500') does not match Software patch level ('2845761567'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

处理过程：

检查两个节点安装的补丁版本，发现没有任何差别！

[root@rac2 ~]# /oracle/app/19c/grid/bin/crsctl query crs releasepatch
Oracle Clusterware release patch level is [3329448500] and the complete list of patches [29401763 29517242 29517247 29585399 30122149 30122167 30125133 ] have been applied on the local node. The release patch string is [19.5.0.0.0].
[root@rac2 ~]# /oracle/app/19c/grid/bin/kfod op=patches
---------------
List of Patches
===============
29401763
29517242
29517247
29585399
30122149
30122167
30125133
[root@rac2 ~]# /oracle/app/19c/grid/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
3329448500
[root@rac2 ~]# su - grid
Last login: Fri Nov 15 12:35:46 CST 2019 on pts/0
[grid@rac2 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2019, Oracle Corporation.  All rights reserved.

Oracle Home       : /oracle/app/19c/grid
Central Inventory : /oracle/app/oraInventory
   from           : /oracle/app/19c/grid/oraInst.loc
OPatch version    : 12.2.0.1.17
OUI version       : 12.2.0.7.0
Log file location : /oracle/app/19c/grid/cfgtoollogs/opatch/opatch2019-11-15_12-41-31PM_1.log

Lsinventory Output file location : /oracle/app/19c/grid/cfgtoollogs/opatch/lsinv/lsinventory2019-11-15_12-41-31PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac2
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1): 

Oracle Grid Infrastructure 19c                                       19.0.0.0.0
There are 1 products installed in this Oracle Home.

Interim patches (4) :

Patch  30125133     : applied on Fri Nov 15 12:21:58 CST 2019
Unique Patch ID:  23151502
Patch description:  "Database Release Update : 19.5.0.0.191015 (30125133)"
   Created on 9 Oct 2019, 00:10:29 hrs PST8PDT
   Bugs fixed:
     14735102, 17428816, 19080742, 19697993, 20313356, 21374587, 21965541
     23296836, 23606241, 24687075, 25756945, 25806201, 25883179, 25986062
     25997810, 26476244, 26611353, 26739322, 26872233, 27126938, 27244999
     27359766, 27369515, 27406105, 27411022, 27423500, 27453490, 27458357
     27489107, 27666312, 27710072, 27846298, 27880025, 27934711, 27935464
     27941110, 27957203, 27967484, 28064977, 28072567, 28125947, 28129791
     28181021, 28210681, 28271693, 28279456, 28294563, 28313275, 28319114
     28326928, 28350595, 28371123, 28379065, 28381939, 28386259, 28390273
     28395302, 28410431, 28431445, 28435333, 28463226, 28475242, 28484299
     28489419, 28498976, 28502773, 28513333, 28521330, 28534475, 28542455
     28547068, 28558645, 28561704, 28567417, 28567819, 28569897, 28572533
     28572544, 28572667, 28572834, 28587723, 28593682, 28594086, 28597221
     28601957, 28605066, 28606598, 28612239, 28625862, 28627033, 28636532
     28643718, 28644549, 28645570, 28646200, 28646939, 28649388, 28655209
     28663782, 28672457, 28673945, 28692103, 28692275, 28694872, 28696373
     28705231, 28710385, 28710734, 28714461, 28715727, 28718469, 28719348
     28720418, 28721497, 28722229, 28730079, 28740708, 28742555, 28749853
     28752923, 28755011, 28758722, 28760206, 28765983, 28767240, 28769456
     28772390, 28774416, 28776811, 28777214, 28781754, 28785531, 28789531
     28791852, 28795551, 28802734, 28804517, 28810381, 28811560, 28815123
     28815355, 28817449, 28819640, 28820669, 28824482, 28827682, 28831971
     28833912, 28835937, 28836716, 28849776, 28854004, 28855922, 28862532
     28863432, 28863487, 28867992, 28873575, 28875089, 28876253, 28876639
     28878865, 28884931, 28888327, 28889730, 28892794, 28897512, 28899663
     28901126, 28905457, 28907196, 28912691, 28915561, 28917080, 28918429
     28919145, 28922227, 28922532, 28922608, 28925634, 28925880, 28933158
     28936114, 28937717, 28938698, 28940179, 28940281, 28941901, 28942455
     28945421, 28945994, 28950868, 28951533, 28952168, 28954762, 28955606
     28956908, 28957292, 28957723, 28958088, 28960863, 28962775, 28965084
     28965231, 28966444, 28974083, 28977322, 28983095, 28983486, 28985478
     28986207, 28986231, 28986326, 28986481, 28988482, 28989306, 28993295
     28993353, 28994307, 28996376, 29000000, 29001305, 29001888, 29002488
     29002784, 29002927, 29003738, 29006318, 29006621, 29007321, 29007353
     29007775, 29008035, 29008669, 29009513, 29010126, 29011936, 29012609
     29013475, 29014076, 29015118, 29017265, 29018655, 29019121, 29021063
     29021352, 29024054, 29024552, 29024732, 29026582, 29026606, 29027456
     29027694, 29027940, 29031575, 29031600, 29032234, 29032276, 29032457
     29032607, 29033052, 29033145, 29033200, 29033280, 29034587, 29037290
     29038528, 29039089, 29039510, 29040739, 29043554, 29043651, 29043725
     29044763, 29044954, 29047850, 29048178, 29048289, 29048498, 29048605
     29050357, 29050560, 29050765, 29050886, 29051702, 29051953, 29052726
     29053783, 29056024, 29056270, 29056560, 29056767, 29059011, 29060216
     29061959, 29062692, 29062848, 29062860, 29062868, 29110526, 29110783
     29110790, 29110797, 29110802, 29110805, 29111598, 29113282, 29113305
     29117526, 29117642, 29119077, 29120223, 29122224, 29122254, 29123297
     29123432, 29123482, 29124368, 29125036, 29125374, 29125380, 29126345
     29127957, 29128693, 29128935, 29129450, 29129497, 29129712, 29130219
     29131539, 29132869, 29132938, 29134447, 29135383, 29135649, 29136111
     29138641, 29139956, 29141316, 29141341, 29141685, 29142609, 29142667
     29143516, 29144995, 29145214, 29145730, 29149829, 29150338, 29151520
     29152357, 29155099, 29157051, 29157389, 29158680, 29158899, 29159909
     29159936, 29160174, 29162095, 29163156, 29163415, 29163437, 29163524
     29163567, 29167111, 29167342, 29167374, 29167940, 29168137, 29168219
     29168433, 29169073, 29169215, 29170232, 29171683, 29171942, 29172618
     29172826, 29173140, 29173373, 29173817, 29174004, 29176318, 29177466
     29177543, 29177886, 29178385, 29180313, 29180455, 29180559, 29180893
     29181153, 29181231, 29181620, 29181743, 29181923, 29182019, 29183912
     29184297, 29184666, 29185193, 29186456, 29189302, 29189307, 29189889
     29190235, 29190474, 29190740, 29191541, 29192419, 29192468, 29192685
     29193207, 29194205, 29194367, 29194493, 29194827, 29194981, 29195279
     29195337, 29195758, 29196725, 29198092, 29199635, 29199733, 29200316
     29200700, 29201494, 29201539, 29202104, 29202850, 29203122, 29203166
     29203425, 29203443, 29203604, 29205281, 29205323, 29205419, 29205463
     29205767, 29205918, 29206109, 29206605, 29207073, 29208260, 29208732
     29211457, 29211724, 29212012, 29212433, 29212611, 29213320, 29213351
     29213613, 29213775, 29213850, 29213879, 29214561, 29214960, 29216746
     29216984, 29217294, 29217472, 29217828, 29217848, 29218570, 29219205
     29219273, 29220079, 29221248, 29221891, 29222031, 29222784, 29223833
     29223859, 29223967, 29224065, 29224605, 29225076, 29225168, 29225758
     29227602, 29228869, 29229164, 29229754, 29229844, 29229955, 29230252
     29230565, 29231133, 29232117, 29232154, 29232449, 29233415, 29234123
     29236573, 29237538, 29237744, 29240307, 29241345, 29241651, 29242017
     29242884, 29243958, 29245137, 29245160, 29246163, 29247415, 29247712
     29247906, 29248495, 29248552, 29248835, 29248858, 29249991, 29250059
     29250317, 29251259, 29253184, 29253871, 29254031, 29254930, 29255178
     29255273, 29255431, 29255435, 29256426, 29259119, 29259320, 29260452
     29261547, 29261906, 29262512, 29262887, 29265448, 29266248, 29266899
     29267292, 29268412, 29269171, 29269228, 29269825, 29270585, 29273539
     29273570, 29273735, 29273812, 29273847, 29274428, 29274564, 29274627
     29275461, 29276272, 29277317, 29278218, 29279658, 29279751, 29279854
     29281527, 29281691, 29281796, 29282233, 29282898, 29285503, 29285788
     29285956, 29286037, 29287130, 29287705, 29292837, 29293072, 29293574
     29297863, 29297915, 29298220, 29299049, 29299082, 29299844, 29301463
     29301566, 29302963, 29303918, 29304781, 29306226, 29306713, 29311528
     29311588, 29312310, 29312672, 29312734, 29312753, 29313347, 29313417
     29313525, 29314539, 29317756, 29318410, 29319441, 29321489, 29323946
     29324568, 29324735, 29325087, 29325105, 29325257, 29325765, 29325993
     29327044, 29329042, 29329087, 29329807, 29330361, 29331066, 29331209
     29331380, 29331493, 29332292, 29332395, 29332771, 29333500, 29336843
     29337310, 29337742, 29338315, 29338453, 29338780, 29338913, 29339101
     29339155, 29341209, 29343086, 29345937, 29346057, 29346211, 29346943
     29347620, 29348176, 29350052, 29350762, 29351386, 29351662, 29351716
     29351735, 29351749, 29352298, 29352724, 29352867, 29352947, 29353271
     29353432, 29353821, 29353960, 29355654, 29356547, 29356704, 29356711
     29356752, 29358509, 29358828, 29360252, 29360285, 29360672, 29360911
     29360950, 29361319, 29361472, 29361801, 29363151, 29364171, 29364177
     29366940, 29367019, 29367561, 29368253, 29368310, 29372541, 29373418
     29373588, 29374179, 29375941, 29376346, 29377986, 29378029, 29378834
     29378913, 29379978, 29382784, 29382815, 29384781, 29384854, 29384864
     29385429, 29385652, 29386502, 29386635, 29386660, 29387073, 29387274
     29388020, 29388072, 29388094, 29388524, 29388830, 29389889, 29390011
     29390435, 29390785, 29391030, 29391237, 29391849, 29393291, 29394014
     29394140, 29394749, 29395657, 29397954, 29397996, 29398488, 29398863
     29399100, 29399121, 29399336, 29399938, 29402131, 29404483, 29405012
     29405462, 29405651, 29405996, 29407804, 29408853, 29409149, 29409455
     29410311, 29410834, 29411037, 29411469, 29412066, 29412269, 29417173
     29417719, 29417884, 29420834, 29421059, 29423016, 29423156, 29423826
     29424999, 29426241, 29429017, 29429264, 29429466, 29429566, 29430524
     29430866, 29431192, 29431485, 29434301, 29435474, 29435652, 29436454
     29436514, 29436727, 29437594, 29437712, 29438277, 29438736, 29439522
     29441196, 29443187, 29443250, 29444072, 29444282, 29444602, 29446669
     29449477, 29449845, 29450193, 29450421, 29450812, 29450936, 29451386
     29452576, 29452936, 29452953, 29454978, 29455424, 29457312, 29457370
     29457807, 29460252, 29461420, 29461791, 29462594, 29462767, 29462957
     29463047, 29463528, 29464616, 29464779, 29465177, 29467622, 29469565
     29470291, 29471860, 29476473, 29481584, 29483532, 29483672, 29483685
     29483712, 29483723, 29485099, 29486181, 29488894, 29489546, 29490256
     29492127, 29492939, 29493122, 29494245, 29495057, 29495684, 29497311
     29497696, 29498198, 29500257, 29500826, 29502561, 29503543, 29503631
     29503827, 29504492, 29504682, 29505668, 29507270, 29507616, 29510278
     29511611, 29514479, 29515134, 29515240, 29515476, 29515766, 29515834
     29516300, 29516766, 29517168, 29517883, 29521187, 29521748, 29521862
     29522358, 29522561, 29522662, 29523511, 29524599, 29525467, 29525886
     29527595, 29527610, 29529147, 29530440, 29530515, 29530812, 29531654
     29532532, 29541742, 29541769, 29542449, 29542643, 29543034, 29543956
     29546817, 29547867, 29548427, 29548687, 29548722, 29549071, 29549104
     29549154, 29549730, 29552773, 29553141, 29557144, 29557261, 29557336
     29558238, 29558975, 29559187, 29559446, 29559908, 29559981, 29565611
     29580394, 29580983, 29581771, 29584261, 29584693, 29586143, 29587765
     29597536, 29597754, 29598039, 29598046, 29598233, 29599008, 29599300
     29601461, 29603460, 29604002, 29604257, 29607136, 29608000, 29611020
     29611991, 29616244, 29616414, 29618074, 29618190, 29620042, 29622936
     29625065, 29625804, 29625876, 29626154, 29626732, 29628200, 29629430
     29629650, 29629681, 29631749, 29632095, 29632265, 29633697, 29633753
     29633936, 29634643, 29635427, 29635717, 29635990, 29637362, 29637526
     29638285, 29641736, 29645349, 29651183, 29651520, 29653132, 29653246
     29655668, 29656819, 29657422, 29657960, 29661028, 29661065, 29661722
     29663368, 29664087, 29664161, 29667994, 29668005, 29669413, 29670782
     29671363, 29672507, 29676089, 29677051, 29677733, 29677927, 29679856
     29683039, 29687214, 29687459, 29687718, 29687763, 29689145, 29692694
     29695425, 29695841, 29696310, 29700125, 29700460, 29703932, 29707099
     29707896, 29708915, 29715220, 29720133, 29721418, 29725781, 29741976
     29742223, 29747493, 29747648, 29749471, 29750673, 29751094, 29753244
     29754951, 29755821, 29756274, 29756444, 29757264, 29757651, 29758661
     29761911, 29766207, 29766503, 29766679, 29771032, 29773842, 29775393
     29779196, 29782211, 29782823, 29782866, 29785239, 29787766, 29791880
     29795712, 29795957, 29802695, 29809837, 29815713, 29836096, 29838773
     29845530, 29848849, 29850930, 29851733, 29858376, 29869086, 29869404
     29869887, 29875459, 29875565, 29881478, 29881839, 29887111, 29888621
     29893132, 29900203, 29902327, 29903299, 29903454, 29921318, 29926466
     29943670, 29997326, 30006159, 30015070, 30019864, 30024618, 30033547
     30034456, 30053036, 30075037, 30076197, 30092859, 30095591, 30114534
     30142907, 30155837, 30174401

Patch  30122167     : applied on Thu Nov 14 20:04:38 CST 2019
Unique Patch ID:  23047395
Patch description:  "ACFS RELEASE UPDATE 19.5.0.0.0 (30122167)"
   Created on 3 Sep 2019, 00:43:16 hrs PST8PDT
   Bugs fixed:
     27494830, 27917085, 28064731, 28293236, 28375150, 28553487, 28611527
     28687713, 28701011, 28740425, 28818513, 28844788, 28855761, 28860451
     28951588, 28960047, 29001307, 29030106, 29031452, 29039918, 29115917
     29127489, 29167352, 29173957, 29198743, 29229120, 29234059, 29250565
     29264772, 29302070, 29313039, 29338628, 29363565, 29379932, 29391373
     29428155, 29437701, 29482354, 29484738, 29520544, 29527221, 29560075
     29590813, 29705711, 29760083, 29779338, 29851205, 29862693, 29872187
     29893148, 29929003, 29963428, 30032562, 30057972, 30179531

Patch  30122149     : applied on Thu Nov 14 16:06:16 CST 2019
Unique Patch ID:  23051443
Patch description:  "OCW RELEASE UPDATE 19.5.0.0.0 (30122149)"
   Created on 19 Sep 2019, 19:36:02 hrs PST8PDT
   Bugs fixed:
     25736599, 26675491, 27148384, 27222128, 27572040, 27604329, 27760043
     27877830, 28302580, 28319989, 28406406, 28441307, 28470673, 28554809
     28621543, 28628631, 28642469, 28697242, 28699321, 28704073, 28710663
     28755846, 28772816, 28785321, 28800508, 28805158, 28808652, 28815557
     28847541, 28847572, 28847655, 28852065, 28865162, 28870111, 28870496
     28871040, 28874416, 28874529, 28877252, 28881191, 28881848, 28887933
     28888083, 28892095, 28911140, 28912188, 28917537, 28925250, 28925460
     28935956, 28940472, 28942694, 28951332, 28958624, 28963036, 28968779
     28973989, 28980448, 28984430, 28987100, 28995287, 29003207, 29003617
     29016294, 29018680, 29024876, 29026154, 29027933, 29030056, 29047127
     29051269, 29052850, 29053083, 29058476, 29111631, 29112455, 29117337
     29123444, 29125708, 29125786, 29129476, 29131772, 29132456, 29139727
     29146157, 29147849, 29149170, 29152603, 29152752, 29154631, 29154636
     29154829, 29159216, 29159661, 29160462, 29161923, 29169540, 29169739
     29170717, 29173618, 29181568, 29182920, 29183298, 29186091, 29191827
     29201143, 29201695, 29202442, 29209545, 29209972, 29210577, 29210610
     29210624, 29210683, 29213641, 29219627, 29224294, 29225861, 29229839
     29231008, 29235934, 29242906, 29243749, 29244495, 29244766, 29244968
     29248723, 29249583, 29251564, 29255616, 29257308, 29260224, 29261171
     29261695, 29271019, 29273360, 29274911, 29275362, 29276220, 29280752
     29282090, 29282666, 29285453, 29285621, 29288842, 29289634, 29290235
     29292232, 29293806, 29293928, 29294753, 29299830, 29307090, 29307109
     29311336, 29329675, 29330791, 29338139, 29339299, 29347106, 29351881
     29357821, 29360467, 29360775, 29367971, 29368725, 29376616, 29377086
     29379299, 29379381, 29380201, 29380253, 29380496, 29380527, 29381000
     29382296, 29385861, 29387212, 29390900, 29391301, 29392779, 29393649
     29395961, 29397888, 29398418, 29402110, 29407192, 29411931, 29413360
     29413448, 29433434, 29438073, 29445982, 29450417, 29452365, 29457319
     29460775, 29465047, 29468589, 29470298, 29471399, 29473188, 29485756
     29495628, 29499868, 29504266, 29505022, 29505730, 29507560, 29510452
     29518947, 29524627, 29528119, 29529127, 29529530, 29529973, 29535802
     29537427, 29540165, 29542152, 29545235, 29555663, 29583036, 29585881
     29592188, 29599431, 29599530, 29600906, 29601549, 29623009, 29632615
     29637215, 29641885, 29645874, 29655862, 29657866, 29659031, 29663838
     29671851, 29672162, 29672852, 29684609, 29686671, 29694497, 29721974
     29760252, 29770006, 29786752, 29797626, 29805654, 29817598, 29825944
     29841704, 29851066, 29862652, 29862828, 29866585, 29921525, 29958570
     30006937, 30032515, 30043518, 30165554, 30226726, 30300812, 3

Patch  29401763     : applied on Thu Apr 18 15:35:16 CST 2019
Unique Patch ID:  22759421
Patch description:  "TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)"
   Created on 11 Apr 2019, 22:26:25 hrs PST8PDT
   Bugs fixed:
     29286300

--------------------------------------------------------------------------------

OPatch succeeded.
[grid@rac2 ~]$ exit
logout

执行如下操作后，可启动crs
[root@rac2 ~]# /oracle/app/19c/grid/bin/clscfg -localpatch
clscfg: EXISTING configuration version 0 detected.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
[root@rac2 ~]# /oracle/app/19c/grid/crs/install/rootcrs.sh -lock
Using configuration parameter file: /oracle/app/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /oracle/app/grid/crsdata/rac2/crsconfig/crslock_rac2_2019-11-15_12-45-15AM.log
2019/11/15 12:45:18 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'

确认oracle权限正确：

[root@rac2 ~]# su - grid
Last login: Fri Nov 15 12:45:16 CST 2019 on pts/0
[grid@rac2 ~]$ ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 grid oinstall 441882008 Nov 15 12:22 /oracle/app/19c/grid/bin/oracle
[grid@rac2 ~]$ exit
logout

再次启动crs

[root@rac2 ~]# /oracle/app/19c/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac2 ~]# 

=============================

官方参考：

After patching manually on the 12.2 Grid Infrastructure home, the rootcrs.sh -postpatch fails with:

2017-11-19 16:29:27: Oracle CRS stack has been shut down
2017-11-19 16:29:27: The stack was already down before stopping it
2017-11-19 16:29:27: Starting CRS without resources...
2017-11-19 16:29:27: OHASD needs to be up for disabling CRS resource
2017-11-19 16:29:27: Executing cmd: /u01/app/12.2.0.1/grid/bin/crsctl start crs -noautostart
2017-11-19 16:29:27: Command output:
> CRS-6706: Oracle Clusterware Release patch level ('748994161') does not match Software patch level ('0'). Oracle Clusterware cannot be started.
> CRS-4000: Command Start failed, or completed with errors.

 

CHANGES

 In earlier Grid Infrastructure releases, the following options were available for manual patching:

A.  In 12.1.0.x, these two commands are used with opatchauto (opatchauto will run these commands) or with manual patching with opatch or opatchauto to unlock and lock the home for patching. The -prepatch requires that the CRS be
running on both nodes. The -postpatch requires that the -prepatch was run successfully.

rootcrs.sh -prepatch
rootcrs.sh -postpatch

 

B.  These two commands are from previous releases of GI <12.1, although they could still be used in 12.1. The -unlock command does not require CRS be running. The -patch command does not require that unlock
was run successfully. So these two commands could work around issues with patching. This is no longer the same case in 12.2 as the -patch option no longer exists.

rootcrs.sh -unlock
rootcrs.sh -patch

In 12.2, users must use rootcrs.sh -prepatch and rootcrs.sh -postpatch for manual patching. 

CAUSE

This issue was caused by rootcrs.sh -prepatch not run successfully before patching.  The user ran rootcrs.sh -unlock because rootcrs.sh -prepatch failed, and then applied the patch manually.  
 

SOLUTION

 Please use the following steps to complete the patching:

1.  Run the following command as the root user to complete the patching set up behind the scenes:

#GI_HOME/bin:>  ./clscfg -localpatch

 

2.  Run the following command as the root user to lock the GI home:

#GI_HOME/crs/install:>  ./rootcrs.sh -lock

 

3.  Run the following command as the root user to start the GI:

#GI_HOME/bin:>  ./crsctl start crs

=======================

This note lists causes/solutions for the following issue:

Oracle 12c Grid Infrastructure fails to start after patching:

CRS-6706: Oracle Clusterware Release patch level ('xxxxxxxx') does not match Software patch level ('xxxxxxxxx'). Oracle Clusterware cannot be started

This can be encountered - for example - in cases where the post patch application instruction to run 'rootcrs.pl -postpatch' is not followed.

 

SCOPE

 

DETAILS

 

Case1: Discrepancy found in the list of applied patches on different nodes:

Node 1:

<GI_HOME>/bin/kfod op=patches 
--------------- 
List of Patches 
=============== 
15994107 
16495244                                   ====>> this one doesn't exist on the other node
17027533 
17303297

<GI_HOME>/bin/kfod op=patchlvl 
------------------- 
Current Patch level 
=================== 
1581137133

Node 2:

<GI_HOME>/bin/kfod op=patches
--------------- 
List of Patches 
=============== 
15994107 
17027533 
17077442                                 ====>> this one doesn't exist on the other node
17303297

<GI_HOME>/bin/kfod op=patchlvl 
------------------- 
Current Patch level 
=================== 
3400658801

 

crsd.trc
======
2015-06-23 15:06:00.260076 :  OCRMAS:1797244672: th_hub_verify_master_pubdata: Shutdown CacheLocal. Patch Levels don't match. Local Patch Level [3467666221] != Cache Writer Patch Level [4137922036]
2015-06-23 15:06:00.260101 :  OCRAPI:1797244672: procr_ctx_set_invalid: ctx is in state [6].
2015-06-23 15:06:00.260103 :  OCRAPI:1797244672: procr_ctx_set_invalid: ctx set to invalid
2015-06-23 15:06:00.260193 :  OCRAPI:1797244672: procr_ctx_set_invalid: Aborting...

 

In 12c, GI home must have identical patches for the clusterware to start unless during rolling patching.

After applied the same patches on all nodes, GI started fine.

 

Case2: All nodes have same patches but different patch level:

Node1

$ /data01/12.1.0/grid/bin/kfod op=patches
---------------
List of Patches
===============
17303297
18365185
19121550

$ /data01/12.1.0/grid/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
2061097728

Node2

$ /data01/12.1.0/grid/bin/kfod op=patches
---------------
List of Patches
===============
17303297
18365185
19121550

$ /data01/12.1.0/grid/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
2061097728                                   ====>> same patch level

 

However opatch lsinventory shows the different patch level:

Patch level status of Cluster nodes :

Patching Level Nodes
-------------- -----
1216323972 node2                          ====>> different patch level
2061097728 node1

 

For 12.1 version :

Execute"<GI_HOME>/crs/install/rootcrs.sh -patch" as <root_user> on the problematic node and the patch level should be corrected.

For 12.2

Execute"<GI_HOME>/crs/install/rootcrs.pl -prepatch"  "<GI_HOME>/crs/install/rootcrs.pl -postpatch"and as <root_user> on the problematic node and the patch level should be corrected.

For Oracle Grid Infrastructure for a standalone server

Log in as root

# <GI_HOME>/crs/install/roothas.sh -unlock
# <GI_HOME>/crs/install/roothas.sh -patch