环境描述：
OS: AIX6.1+TL07
两节点RAC：qc550705、qc550707
升级前GI和DB版本：12.1.0.1
升级目标：将GI和DB版本升级到12.1.0.2，采用rolling upgrade的升级方式，因为我们没有配置DG，所以这里的滚动升级是指GI软件和DB软件升级期间尽量减少对生产库的影响，但是在DB instance升级期间是数据库时需要完全停止的
///
// 第一部分：GI升级
///
1、创建软件安装目录, RAC两个节点上都要创建su - grid
cd /oracle/app/grid/product
mkdir -p 12.1.0.2/grid_1
2、确保/oracle目录有至少20G的空闲空间
3、运行./runInstaller，见下图
耗费了很长时间去找上面这个报错的原因，发现在/oracle/app/oraInventory目录下有个名为locks的目录， root权限的：
root@qc550705:/oracle/app/oraInventory>ls -rlt
total 48
drwxrwx--- 2 grid oinstall 256 Jan 23 2014 oui
drwxrwx--- 2 grid oinstall 256 Jan 23 2014 ContentsXML
-rwxrwx--- 1 grid oinstall 1647 Jan 23 2014 orainstRoot.sh
-rw-rw---- 1 grid oinstall 59 Jan 23 2014 oraInst.loc
drwxrwx--- 7 grid oinstall 256 Jan 24 2014 backup
drwxrwx--- 2 root system 256 Jul 10 13:03 locks <---- 正是这个文件
drwxrwx--- 2 grid oinstall 16384 Jul 29 08:40 logslocks目录里有个writer.lock的文件，看内容是opatchauto产生的错误：
root@qc550705:/oracle/app/oraInventory>cd locks
root@qc550705:/oracle/app/oraInventory/locks>ls -rlt
total 8
-rw-rw---- 1 root system 2147 Jul 10 13:03 writer.lock
root@qc550705:/oracle/app/oraInventory/locks>cat *
2015-07-10_01-03-24-PM
java.lang.Throwable
at oracle.sysman.oii.oiit.OiitLockHeartbeat.writeStackTrace(OiitLockHeartbeat.java:186)
at oracle.sysman.oii.oiit.OiitLockHeartbeat.<init>(OiitLockHeartbeat.java:171)
at oracle.sysman.oii.oiit.OiitTargetLocker.getWriterLock(OiitTargetLocker.java:280)
at oracle.sysman.oii.oiit.OiitTargetLocker.getWriterLock(OiitTargetLocker.java:185)
at oracle.sysman.oii.oiit.OiitTargetLocker.getWriterLock(OiitTargetLocker.java:172)
at oracle.sysman.oii.oiic.OiicStandardInventorySession.acquireLocks(OiicStandardInventorySession.java:429)
at oracle.sysman.oii.oiic.OiicStandardInventorySession.initSession(OiicStandardInventorySession.java:311)
at oracle.sysman.oii.oiic.OiicStandardInventorySession.initSession(OiicStandardInventorySession.java:240)
at oracle.sysman.oii.oiic.OiicStandardInventorySession.initSession(OiicStandardInventorySession.java:189)
at oracle.opatch.OUIInventorySession.initSession(OUIInventorySession.java:71)
at oracle.opatch.OUISessionManager.setupSession(OUISessionManager.java:149)
at oracle.opatch.OUISessionManager.lockCentralInventory(OUISessionManager.java:232)
at oracle.opatch.OUISessionManager.instantiate(OUISessionManager.java:87)
at oracle.opatch.OUISessionManager.getOUIInventorySession(OUISessionManager.java:511)
at oracle.opatch.InventorySessionManager.getInventorySession(InventorySessionManager.java:38)
at oracle.opatch.Locker.lock(Locker.java:337)
at oracle.opatch.OracleHomeInventory.lock(OracleHomeInventory.java:2435)
at oracle.opatch.OracleHomeInventory.createInventoryObj(OracleHomeInventory.java:2557)
at oracle.opatch.OracleHomeInventory.getInstance(OracleHomeInventory.java:1024)
at oracle.opatch.GenericInventory.createInstance(GenericInventory.java:41)
at oracle.opatchauto.gi.Crs.getInstance(Crs.java:149)
at oracle.opatchauto.gi.GIPatchingPrereq.checkApplicableForGIPatching(GIPatchingPrereq.java:592)
at oracle.opatchauto.gi.GIPatchingPrereq.checkApplicableForGIPatching(GIPatchingPrereq.java:561)
at oracle.opatchauto.gi.OPatchauto.runInitialPrereqs(OPatchauto.java:117)
at oracle.opatchauto.gi.OPatchauto.main(OPatchauto.java:90)
判断可能是之前运行opatchauto异常终止后写入的，不追究原因了，rm -r locks直接删除目录后安装得以继续
提示在两个节点上依次运行rootupgrade.sh，但是运行./rootupgrade.sh的时候又报错了：
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /oracle/app/grid/product/12.1.0.2/grid_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/grid/product/12.1.0.2/grid_1/crs/install/crsconfig_params
/oracle/app/grid/product/12.1.0.2/grid_1/bin/sqlplus -V ... failed rc=255 with message:
exec(): 0509-036 Cannot load program sqlplus because of the following errors: rtld: 0712-001 Symbol CreateIoCompletionPort was referenced from module /oracle/app/grid/product/12.1.0.2/grid_1/lib/libttsh12.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol GetMultipleCompletionStatus was referenced from module /oracle/app/grid/product/12.1.0.2/grid_1/lib/libttsh12.so(), but a runtime definition of the symbol was not found. rtld: 0712-002 fatal error: exiting.
Died at /oracle/app/grid/product/12.1.0.2/grid_1/crs/install/crsupgrade.pm line 1589.
The command '/oracle/app/grid/product/12.1.0.2/grid_1/perl/bin/perl -I/oracle/app/grid/product/12.1.0.2/grid_1/perl/lib -I/oracle/app/grid/product/12.1.0.2/grid_1/crs/install /oracle/app/grid/product/12.1.0.2/grid_1/crs/install/rootcrs.pl -upgrade' execution failed
参考1949184.1，AIX里有一个名为” I/O Completion Ports”的component必须打开，默认是Define状态的
root@qc550705:/oracle/app/grid/product/12.1.0.2/grid_1>lsdev | grep iocp
iocp0 Defined I/O Completion Ports
通过smitty iocp->” Configure Defined I/O Completion Ports”将这个component打开
root@qc550707:/oracle/app/grid/product/12.1.0.2/grid_1>lsdev | grep iocp
iocp0 Available I/O Completion Ports
再次运行./rootupgrade.sh报错消失，终于能往下走了
root@qc550705:/oracle/app/grid/product/12.1.0.2/grid_1>./rootupgrade.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /oracle/app/grid/product/12.1.0.2/grid_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/grid/product/12.1.0.2/grid_1/crs/install/crsconfig_params
2015/07/30 15:04:58 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2015/07/30 15:05:23 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2015/07/30 15:06:02 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2015/07/30 15:06:16 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2015/07/30 15:06:37 CLSRSC-464: Starting retrieval of the cluster configuration data
2015/07/30 15:07:30 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2015/07/30 15:07:30 CLSRSC-363: User ignored prerequisites during installation
2015/07/30 15:08:14 CLSRSC-515: Starting OCR manual backup.
2015/07/30 15:08:23 CLSRSC-516: OCR manual backup successful.
2015/07/30 15:08:44 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2015/07/30 15:08:45 CLSRSC-482: Running command: '/oracle/app/grid/product/12.1.0/grid_1/bin/crsctl start rollingupgrade 12 .1.0.2.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2015/07/30 15:09:13 CLSRSC-482: Running command: '/oracle/app/grid/product/12.1.0.2/grid_1/bin/asmca -silent -upgradeNodeAS M -nonRolling false -oldCRSHome /oracle/app/grid/product/12.1.0/grid_1 -oldCRSVersion 12.1.0.1.0 -nodeNumber 1 -firstNode t rue -startRolling false'
ASM configuration upgraded in local node successfully.
2015/07/30 15:09:23 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2015/07/30 15:09:23 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2015/07/30 15:11:39 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
2015/07/30 15:14:55 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2015/07/30 15:22:13 CLSRSC-472: Attempting to export the OCR
2015/07/30 15:22:13 CLSRSC-482: Running command: 'ocrconfig -upgrade grid oinstall'
2015/07/30 15:22:21 CLSRSC-473: Successfully exported the OCR
2015/07/30 15:22:47 CLSRSC-486:
At this stage of upgrade, the OCR has changed.
Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
2015/07/30 15:22:47 CLSRSC-541:
To downgrade the cluster:
1. All nodes that have been upgraded must be downgraded.
2015/07/30 15:22:48 CLSRSC-542:
2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
2015/07/30 15:22:48 CLSRSC-543:
3. The downgrade command must be run on the node qc550707 with the '-lastnode' option to restore global configuration data .
2015/07/30 15:23:12 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
2015/07/30 15:23:56 CLSRSC-474: Initiating upgrade of resource types
2015/07/30 15:24:51 CLSRSC-482: Running command: 'upgrade model -s 12.1.0.1.0 -d 12.1.0.2.0 -p first'
2015/07/30 15:24:52 CLSRSC-475: Upgrade of resource types successfully initiated.
2015/07/30 15:25:07 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
rootupgrade.sh运行后意味着滚动升级已经开始，期间会启停该台主机的clusterware，所有连接该节点的应用将会切换到另一个节点运行，以下是升级期间Clusterware的状态记录：
###由于采用的是rolling upgrade可以看到节点qc550705上的资源在升级期间正在被逐个offline掉
root@qc550707:/>crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFSVOL1.advm
ONLINE ONLINE qc550705 Volume device /dev/a
sm/acfsvol1-48 is on
line,STOPPING
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol1-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL3.advm
ONLINE ONLINE qc550705 Volume device /dev/a
sm/acfsvol3-48 is on
line,STOPPING
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol3-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL_STD1.advm
OFFLINE OFFLINE qc550705 Unable to connect to
ASM,STABLE
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol_std1-48 i
s online,STABLE
ora.ACFSDG.dg
ONLINE OFFLINE qc550705 STABLE
ONLINE ONLINE qc550707 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE OFFLINE qc550705 STABLE
ONLINE ONLINE qc550707 STABLE
ora.LISTENER.lsnr
ONLINE OFFLINE qc550705 STABLE
ONLINE ONLINE qc550707 STABLE
ora.SYSDG.dg
ONLINE OFFLINE qc550705 STABLE
ONLINE ONLINE qc550707 STABLE
ora.acfsdg.acfsvol1.acfs
ONLINE OFFLINE qc550705 admin unmounted /acf
s1,STABLE
ONLINE ONLINE qc550707 mounted on /acfs1,ST
ABLE
ora.acfsdg.acfsvol3.acfs
ONLINE OFFLINE qc550705 admin unmounted /acf
s3,STABLE
OFFLINE OFFLINE qc550707 (2) volume /acfs3 of
fline,STABLE
ora.net1.network
ONLINE ONLINE qc550705 STABLE
ONLINE ONLINE qc550707 STABLE
ora.ons
ONLINE ONLINE qc550705 STABLE
ONLINE ONLINE qc550707 STABLE
ora.proxy_advm
ONLINE ONLINE qc550705 STABLE
ONLINE ONLINE qc550707 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE qc550707 STABLE
ora.acfsdb.acfsdb_svc.svc
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
ora.acfsdb.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.acfsdg.acfsvol_std1.acfs
1 OFFLINE OFFLINE (2) volume /acfsstd1
offline,STABLE
ora.asm
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE qc550707 STABLE
ora.cdb550.cdb550svc.svc
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE qc550707 STABLE
ora.cdb550.db
1 ONLINE ONLINE qc550707 Open,STABLE
2 ONLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cvu
1 ONLINE ONLINE qc550707 STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.qc550705.vip
1 ONLINE INTERMEDIATE qc550707 FAILED OVER,STABLE
ora.qc550707.vip
1 ONLINE ONLINE qc550707 STABLE
ora.scan1.vip
1 ONLINE ONLINE qc550707 STABLE
###过不久节点qc550705彻底停掉后crsctl输出里只有qc550707了
root@qc550707:/>crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFSVOL1.advm
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol1-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL3.advm
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol3-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL_STD1.advm
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol_std1-48 i
s online,STABLE
ora.ACFSDG.dg
ONLINE ONLINE qc550707 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE qc550707 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE qc550707 STABLE
ora.SYSDG.dg
ONLINE ONLINE qc550707 STABLE
ora.acfsdg.acfsvol1.acfs
ONLINE ONLINE qc550707 mounted on /acfs1,ST
ABLE
ora.acfsdg.acfsvol3.acfs
OFFLINE OFFLINE qc550707 (2) volume /acfs3 of
fline,STABLE
ora.net1.network
ONLINE ONLINE qc550707 STABLE
ora.ons
ONLINE ONLINE qc550707 STABLE
ora.proxy_advm
ONLINE ONLINE qc550707 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE qc550707 STABLE
ora.acfsdb.acfsdb_svc.svc
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
ora.acfsdb.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.acfsdg.acfsvol_std1.acfs
1 OFFLINE OFFLINE (2) volume /acfsstd1
offline,STABLE
ora.asm
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE qc550707 STABLE
ora.cdb550.cdb550svc.svc
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE qc550707 STABLE
ora.cdb550.db
1 ONLINE ONLINE qc550707 Open,STABLE
2 ONLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cvu
1 ONLINE ONLINE qc550707 STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.qc550705.vip
1 ONLINE INTERMEDIATE qc550707 FAILED OVER,STABLE
ora.qc550707.vip
1 ONLINE ONLINE qc550707 STABLE
ora.scan1.vip
1 ONLINE ONLINE qc550707 STABLE
--------------------------------------------------------------------------------
root@qc550707:/>crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFSVOL1.advm
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol1-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL3.advm
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol3-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL_STD1.advm
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol_std1-48 i
s online,STABLE
ora.ACFSDG.dg
ONLINE ONLINE qc550707 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE qc550707 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE qc550707 STABLE
ora.SYSDG.dg
ONLINE ONLINE qc550707 STABLE
ora.acfsdg.acfsvol1.acfs
ONLINE ONLINE qc550707 mounted on /acfs1,ST
ABLE
ora.acfsdg.acfsvol3.acfs
OFFLINE OFFLINE qc550707 (2) volume /acfs3 of
fline,STABLE
ora.net1.network
ONLINE ONLINE qc550707 STABLE
ora.ons
ONLINE ONLINE qc550707 STABLE
ora.proxy_advm
ONLINE ONLINE qc550707 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE qc550707 STABLE
ora.acfsdb.acfsdb_svc.svc
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
ora.acfsdb.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.acfsdg.acfsvol_std1.acfs
1 OFFLINE OFFLINE (2) volume /acfsstd1
offline,STABLE
ora.asm
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE qc550707 STABLE
ora.cdb550.cdb550svc.svc
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE qc550707 STABLE
ora.cdb550.db
1 ONLINE ONLINE qc550707 Open,STABLE
2 ONLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cvu
1 ONLINE ONLINE qc550707 STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.qc550705.vip
1 ONLINE INTERMEDIATE qc550707 FAILED OVER,STABLE
ora.qc550707.vip
1 ONLINE ONLINE qc550707 STABLE
ora.scan1.vip
1 ONLINE ONLINE qc550707 STABLE
--------------------------------------------------------------------------------
节点qc550705升级完成后上面的资源组都已经启动：
root@qc550705:/oracle/app/grid/product/12.1.0.2/grid_1>crsctl status res -t -n qc550705
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFSVOL1.advm
ONLINE ONLINE qc550705 Volume device /dev/a
sm/acfsvol1-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL3.advm
ONLINE ONLINE qc550705 Volume device /dev/a
sm/acfsvol3-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL_STD1.advm
OFFLINE OFFLINE qc550705 Volume device /dev/a
sm/acfsvol_std1-48 i
s offline,STABLE
ora.ACFSDG.dg
ONLINE ONLINE qc550705 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE qc550705 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE qc550705 STABLE
ora.SYSDG.dg
ONLINE ONLINE qc550705 STABLE
ora.acfsdg.acfsvol1.acfs
ONLINE ONLINE qc550705 mounted on /acfs1,ST
ABLE
ora.acfsdg.acfsvol3.acfs
ONLINE ONLINE qc550705 mounted on /acfs3,ST
ABLE
ora.net1.network
ONLINE ONLINE qc550705 STABLE
ora.ons
ONLINE ONLINE qc550705 STABLE
ora.proxy_advm
ONLINE ONLINE qc550705 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE qc550705 Started,STABLE
ora.cdb550.cdb550svc.svc
1 ONLINE ONLINE qc550705 STABLE
ora.cdb550.db
2 ONLINE ONLINE qc550705 Open,STABLE
ora.qc550705.vip
1 ONLINE ONLINE qc550705 STABLE
--------------------------------------------------------------------------------
检查Cluster后台进程都已经从12.1.0.2所在的$ORACLE_HOME启动
grid@qc550705:/home/grid>ps -ef|grep d.bin
grid 4456482 3407988 0 15:42:09 pts/3 0:00 grep d.bin
root 5898368 1 0 15:20:30 - 0:06 /oracle/app/grid/product/12.1.0.2/grid_1/bin/octssd.bin reboot
grid 10223722 1 0 15:19:49 - 0:06 /oracle/app/grid/product/12.1.0.2/grid_1/bin/evmd.bin
root 12648692 1 0 15:17:22 - 0:23 /oracle/app/grid/product/12.1.0.2/grid_1/bin/ohasd.bin exclusive
grid 12714096 1 0 15:19:54 - 0:01 /oracle/app/grid/product/12.1.0.2/grid_1/bin/gpnpd.bin
root 5177706 1 0 15:22:22 - 0:14 /oracle/app/grid/product/12.1.0.2/grid_1/bin/crsd.bin reboot
grid 8454582 7471162 0 15:20:05 - 0:19 /oracle/app/grid/product/12.1.0.2/grid_1/bin/ocssd.bin
root 9240838 1 2 15:20:32 - 0:08 /oracle/app/grid/product/12.1.0.2/grid_1/bin/osysmond.bin
grid 9437466 1 2 15:20:03 - 0:22 /oracle/app/grid/product/12.1.0.2/grid_1/bin/gipcd.bin
grid 10289418 1 0 15:19:52 - 0:01 /oracle/app/grid/product/12.1.0.2/grid_1/bin/mdnsd.bin
接着在另个节点qc550707上运行./rootupgrade.sh，同样qc550707上的clusterware会自动重启，所有连接qc550707的应用将会切换到qc550705运行，以下是运行完成后的进程检查，确认是从12.1.0.2目录启动的
root@qc550707:/oracle/app/grid/product/12.1.0.2/grid_1>ps -ef|grep d.bin
root 4653140 1 0 15:46:42 - 0:21 /oracle/app/grid/product/12.1.0.2/grid_1/bin/ohasd.bin exclusive
grid 6619272 1 3 15:49:13 - 0:13 /oracle/app/grid/product/12.1.0.2/grid_1/bin/evmd.bin
grid 11599950 1 2 15:49:28 - 0:16 /oracle/app/grid/product/12.1.0.2/grid_1/bin/gipcd.bin
grid 11993244 4194578 1 15:49:30 - 0:13 /oracle/app/grid/product/12.1.0.2/grid_1/bin/ocssd.bin
root 2425302 1 0 15:49:48 - 0:06 /oracle/app/grid/product/12.1.0.2/grid_1/bin/osysmond.bin
grid 3735964 1 0 15:49:18 - 0:01 /oracle/app/grid/product/12.1.0.2/grid_1/bin/gpnpd.bin
root 4129266 1 3 15:50:27 - 0:24 /oracle/app/grid/product/12.1.0.2/grid_1/bin/crsd.bin reboot
grid 5505392 1 0 15:49:16 - 0:01 /oracle/app/grid/product/12.1.0.2/grid_1/bin/mdnsd.bin
root 8454420 1 3 15:49:45 - 0:14 /oracle/app/grid/product/12.1.0.2/grid_1/bin/octssd.bin reboot
root 10748284 4849798 0 16:06:29 pts/0 0:00 grep d.bin
root@qc550707:/oracle/app/grid/product/12.1.0.2/grid_1>crsctl status res -t -n qc550707
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFSVOL1.advm
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol1-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL3.advm
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol3-48 is on
line,STABLE
ora.ACFSDG.ACFSVOL_STD1.advm
ONLINE ONLINE qc550707 Volume device /dev/a
sm/acfsvol_std1-48 i
s online,STABLE
ora.ACFSDG.dg
ONLINE ONLINE qc550707 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE qc550707 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE qc550707 STABLE
ora.SYSDG.dg
ONLINE ONLINE qc550707 STABLE
ora.acfsdg.acfsvol1.acfs
ONLINE ONLINE qc550707 mounted on /acfs1,ST
ABLE
ora.acfsdg.acfsvol3.acfs
ONLINE ONLINE qc550707 mounted on /acfs3,ST
ABLE
ora.net1.network
ONLINE ONLINE qc550707 STABLE
ora.ons
ONLINE ONLINE qc550707 STABLE
ora.proxy_advm
ONLINE ONLINE qc550707 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.acfsdg.acfsvol_std1.acfs
1 ONLINE ONLINE qc550707 mounted on /acfsstd1
,STABLE
ora.asm
2 ONLINE ONLINE qc550707 Started,STABLE
ora.cdb550.cdb550svc.svc
2 ONLINE ONLINE qc550707 STABLE
ora.cdb550.db
1 ONLINE ONLINE qc550707 Open,STABLE
ora.oc4j
1 ONLINE ONLINE qc550707 STABLE
ora.qc550707.vip
1 ONLINE ONLINE qc550707 STABLE
--------------------------------------------------------------------------------
回到这个久违的界面，点击”OK”
又来了一个错误
从提示的.log文件里观察到：
。。。。
INFO: Executing MGMTDBCDB
INFO: Command /oracle/app/grid/product/12.1.0.2/grid_1/bin/dbca -silent -createDatabase -createAsContainerDatabase
true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName SYSDG -dataf
ileJarLocation /oracle/app/grid/product/12.1.0.2/grid_1/assistants/dbca/templates -characterset AL32UTF8 -autoGener
atePasswords -skipUserTemplateCheck -oui_internal
INFO: ... GenericInternalPlugIn.handleProcess() entered.
INFO: ... GenericInternalPlugIn: getting configAssistantParmas.
INFO: ... GenericInternalPlugIn: checking secretArguments.
INFO: No arguments to pass to stdin
INFO: ... GenericInternalPlugIn: starting read loop.
INFO: Read: PRCR-1154 : Failed to create file output stream with file name: /oracle/app/oracle/cfgtoollogs/dbca/tra
ce.log_OraGI12Home2_2015-07-30_04-12-27-PM/oracle/app/oracle/cfgtoollogs/dbca/trace.log_OraGI12Home2_2015-07-30_04-
12-27-PM.lck (The file access permissions do not allow the specified action.)
WARNING: Skipping line: PRCR-1154 : Failed to create file output stream with file name: /oracle/app/oracle/cfgtooll
ogs/dbca/trace.log_OraGI12Home2_2015-07-30_04-12-27-PM/oracle/app/oracle/cfgtoollogs/dbca/trace.log_OraGI12Home2_20
15-07-30_04-12-27-PM.lck (The file access permissions do not allow the specified action.)
INFO: End of argument passing to stdin
INFO: Read:
WARNING: Skipping line:
因为执行的dbca是从grid用户的$ORACLE_HOME下发起的：/oracle/app/grid/product/12.1.0.2/grid_1/bin/dbca，上述信息表明/oracle/app/oracle/cfgtoollogs/dbca/目录没有写权限，
grid@qc550705:/oracle/app/oracle/cfgtoollogs/dbca>ls -ld
drwxr-xr-x 5 oracle oinstall 4096 Jul 30 16:54
给同组的grid用户加上写权限，两个节点上都要执行：
Chmod g+w /oracle/app/oracle/cfgtoollogs/dbca
同时参照1940246.1的要求确保两个节点的sqlnet.ora文件里有下面这一行，如果DIAG_ADR_ENABLED=OFF的话也会引起[INS-20802]错误
DIAG_ADR_ENABLED=ON。
完成上述调整后点击”Retry”继续
最后一步Cluster Verification时出现报错：
按照1949615.1 为网卡加上了monitor属性
Ifconfig en0 monitor
Ifconfig en4 monitor
root@qc550705:/oracle/app/oraInventory/logs>ifconfig -a
en0: flags=e080863,280<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHAIN,MONITOR>
inet 3.3.104.16 netmask 0xffffff00 broadcast 3.3.104.255
inet 169.254.166.7 netmask 0xffff0000 broadcast 169.254.255.255
tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en4: flags=e080863,280<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHAIN,MONITOR>
inet 10.10.104.16 netmask 0xffffff80 broadcast 10.10.104.127
inet 10.10.104.45 netmask 0xffffff80 broadcast 10.10.104.127
inet 10.10.104.116 netmask 0xffffff80 broadcast 10.10.104.127
tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
但再次verification还是有报错
检查安装InstallAction*.log里有如下报错，因为用的是/etc/hosts而没有使用域名服务器去解析scan ip，所以报了这个错，可以忽略掉
INFO: ERROR:
INFO: PRVG-1101 : SCAN name "ora12c-scan" failed to resolve
INFO: ERROR:
INFO: PRVF-4657 : Name resolution setup check for "ora12c-scan" (IP address: 10.10.104.116) failed
INFO: ERROR:
INFO: PRVF-4663 : Found configuration issue with the 'hosts' entry in the /etc/nsswitch.conf file
点击Yes
完成GI的升级
修改Grid用户的环境变量将$ORACLE_HOME指向新的路径：
export ORACLE_HOME=/oracle/app/grid/product/12.1.0/grid_1
改成
export ORACLE_HOME=/oracle/app/grid/product/12.1.0.2/grid_1
使用root用户重启clusterware，验证GI升级后能正常启动
Su - root
crsctl stop crs –f
crsctl start crs
opatch检查GI版本为12.1.0.2
grid@qc550707:/home/grid>opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.3
Copyright (c) 2015, Oracle Corporation. All rights reserved.
Oracle Home : /oracle/app/grid/product/12.1.0.2/grid_1
Central Inventory : /oracle/app/oraInventory
from : /oracle/app/grid/product/12.1.0.2/grid_1/oraInst.loc
OPatch version : 12.1.0.1.3
OUI version : 12.1.0.2.0
Log file location : /oracle/app/grid/product/12.1.0.2/grid_1/cfgtoollogs/opatch/opatch2015-08-04_10-01-30AM_1.log
Lsinventory Output file location : /oracle/app/grid/product/12.1.0.2/grid_1/cfgtoollogs/opatch/lsinv/lsinventory2015-08-04_10-01-30AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c 12.1.0.2.0
There are 1 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
0 qc550707,qc550705
--------------------------------------------------------------------------------
现在可以放心删除12.1.0.1所在的GI用户的ORACLE_HOME目录，以给之后的DB rolling upgrade留出空间，删除之前最好备份一下，以防万一
rm -rf /oracle/app/grid/product/12.1.0
// 第二部分：DB升级
///
DB升级我们同样使用rolling upgrade的方法，注意rolling upgrade仅针对DB软件升级而言，对于db instance也就是数据字典的升级两个节点上的实例必须全停或者全起，不存在一个做完再作另外一个的情况。
1、创建软件安装目录, RAC两个节点上都要创建su - oracle
cd /oracle/app/oracle/product/
mkdir -p 12.1.0.2/db_1
2、确保/oracle目录有至少20G的空闲空间
3、运行./runInstaller，见下图
选择”Upgrade an exist database”
RAC选择第二项
ORACLE_HOME指定为/oracle/app/oracle/product/12.1.0.2/db_1目录
按提示依次在两节点上执行root.sh
root@qc550705:/oracle/app/oracle/product/12.1.0.2/db_1>/oracle/app/oracle/product/12.1.0.2/db_1/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/app/oracle/product/12.1.0.2/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: n
Copying oraenv to /usr/local/bin ...
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
选择”Upgrade Oracle Database”
这里进入了DBUA界面，检测到当前环境下有acfsdb、cdb550两个库，其中cdb550处于运行状态，acfsdb处于shutdown状态，我们选择
cdb550作为升级对象
Cdb550下还有三个pdb，也被列了出来
预检时找到了两个错误一个是数据库的recyclebin未清空（此错误是可以自动修复的），另一个提示Storage Check未通过（需要人工干预）
关于Storage Check未能通过的具体原因是：SYSAUX表空间的数据文件需要额外的118MB空间，但由于+SYSDG/sysaux01.dbf这个数据文件是autoextend off的，无法自动扩展
SQL> col file_name format a40
SQL> select file_name,autoextensible from dba_data_files where file_name like '+SYSDG/sysaux01.dbf';
FILE_NAME AUT
---------------------------------------- ---
+SYSDG/sysaux01.dbf NO
打开+SYSDG/sysaux01.dbf文件的自动扩展属性：
SQL> alter database datafile '+SYSDG/sysaux01.dbf' autoextend on;
Database altered.
SQL> select file_name,autoextensible from dba_data_files where file_name like '+SYSDG/sysaux01.dbf';
FILE_NAME AUT
---------------------------------------- ---
+SYSDG/sysaux01.dbf YES
选择”Fix & Check Again”自动修复再次检查后通过了，来到了如下界面，这里需要将Audit file destination改为新路径” /oracle/app/oracle/product/12.1.0.2/db_1/rdbms/audit”，其它维持缺省值
这里选择不配置EM DB Express
升级前提供了备份的选项(有RMAN Backup和Flashback guarantee point两种)，为节省时间我们选择使用自己的备份策略
安装概要界面出现意味着离成功不远了
升级进度的界面，两个数据库：
每一个步骤都有时长记录，总耗时1 hrs 40 mins左右
DB Upgrade升级结果报告：
最后修改oracle用户下ORACLE_HOME环境变量，验证补丁版本
ORACLE_HOME=$ORACLE_BASE/product/12.1.0 /db_1
改成
ORACLE_HOME=$ORACLE_BASE/product/12.1.0.2/db_1
oracle@qc550707:/home/oracle>opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.3
Copyright (c) 2015, Oracle Corporation. All rights reserved.
Oracle Home : /oracle/app/oracle/product/12.1.0.2/db_1
Central Inventory : /oracle/app/oraInventory
from : /oracle/app/oracle/product/12.1.0.2/db_1/oraInst.loc
OPatch version : 12.1.0.1.3
OUI version : 12.1.0.2.0
Log file location : /oracle/app/oracle/product/12.1.0.2/db_1/cfgtoollogs/opatch/opatch2015-08-04_19-46-08PM_1.log
Lsinventory Output file location : /oracle/app/oracle/product/12.1.0.2/db_1/cfgtoollogs/opatch/lsinv/lsinventory2015-08-04_19-46-08PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Database 12c 12.1.0.2.0
There are 1 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.
Rac system comprising of multiple nodes
Local node = qc550707
Remote node = qc550705
--------------------------------------------------------------------------------
OPatch succeeded.
来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/53956/viewspace-1761912/，如需转载，请注明出处，否则将追究法律责任。
转载于:http://blog.itpub.net/53956/viewspace-1761912/
扫一扫
591
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
