How do you apply a Patchset,PSU or CPU in a Data Guard Physical Standby configuration [ID 278641.1] | |||||
| |||||
修改时间 04-JUN-2010 类型 HOWTO 状态 PUBLISHED |
In this Document
Goal
Solution
References
Applies to:
Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 11.2.0.1.0 - Release: 10.1 to 11.2Information in this document applies to any platform.
Goal
To apply an Interim Patchset for example the 10.2.0.4 Patchset or January 2010 10.2.0.4 RDBMS PSU, to both the Primary and Standby Sites.
Configurations with a logical standby should refer instead to My Oracle Support Article "Upgrading Oracle Database with a Logical Standby Database In Place (Doc ID 437276.1)".
The process for applying the Patchset/PSU/CPU to a Primary and Standby site will be the same as that used to apply a patchset. The following is one approach to performing this task:
Solution
Process Overview:
1. In the primary disable log shipping to the standby site.
2. Shutdown the standby site and apply interim patchsets to the RDBMS binaries as per the README. This includes Patchset/Patchset Update(PSU)/Critical Patch Update (CPU). You will not be able to and do not need to run the scripts(catpatch.sql etc) against the standby rdbms itself.
3. Shutdown the primary site and apply the Patchset/PSU/CPU patch to the RDBMS binaries as per the README.
4. Start the primary site, re-enable log shipping to the standby and patch the RDBMS itself using the instructions in the README (run catpatch/catbundle/catcpu etc).
5. At the standby site restart redo apply/managed recovery and the RDBMS changes implemented in the Primary Site through catpatch/catbundle/catcpu will also be applied to the standby.
The Interim Patchset application in detail - in this case 10.2.0.4 PSU 3 or Patch 9119284
1. If you are using a Data Guard Broker configuration, in the Primary Site disable archiving to Standby Site.
If this is RAC you will need to ensure that archiving is disabled from all Primary Site Instances. When a Data Guard Broker configuration is in place this will be managed through dgmgrl. If you are using sqlplus and DO NOT have a broker configuration this would need to be performed from the sql prompt.
1.1 Disable log shipping using DGMGRL.
DGMGRL> connect /
Connected.
DGMGRL> show database verbose plb_prm
Database
Name: plb_prm
Role: PRIMARY
Enabled: YES
Intended State: ONLINE
Instance(s):
plb
Properties:
InitialConnectIdentifier = 'plb_prm_dgmgrl'
..
.
Current status for "plb_prm":
SUCCESS
DGMGRL> edit database plb_prm set state='LOG-TRANSPORT-OFF';
Succeeded.
DGMGRL> exit
Connected.
DGMGRL> show database verbose plb_prm
Database
Name: plb_prm
Role: PRIMARY
Enabled: YES
Intended State: ONLINE
Instance(s):
plb
Properties:
InitialConnectIdentifier = 'plb_prm_dgmgrl'
..
.
Current status for "plb_prm":
SUCCESS
DGMGRL> edit database plb_prm set state='LOG-TRANSPORT-OFF';
Succeeded.
DGMGRL> exit
OR
1.2 If you are managing the environment using sqlplus
In this case you DO NOT have a Data Guard Broker configuration disable the destination used to ship archives from the primary to the standby site using sqlplus.
SQL> alter system set log_archive_dest_state_X=defer scope=both sid='*'
2. Apply the patch at the Standby Site binaries.
If this is a RAC standby this patch application would occur across all nodes.
2.1.1 If this is a single instance environment shutdown ALL processes running from the home.
This will include all listeners, database instances, asm instances etc running from the home to patched.
[oracle@aulnx9 patches]$ lsnrctl stop lsnrplb
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 04-FEB-2010 08:41:29
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=aulnx9.au.oracle.com)(PORT=1666)))
The command completed successfully
[oracle@aulnx9 patches]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 08:42:02 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 04-FEB-2010 08:41:29
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=aulnx9.au.oracle.com)(PORT=1666)))
The command completed successfully
[oracle@aulnx9 patches]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 08:42:02 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
OR
2.1.2 If this a RAC standby shutdown all the RAC services that are running from the ORACLE_HOME to be patched.
In this case the RAC database is called plb and the listener name for the listener running from the ORACLE_HOME to patched is lsnrplb.
[oracle@aulnx13 ~] srvctl stop database -d plb_std
[oracle@aulnx13 ~] srvctl stop listener -n aulnx13.au.oracle.com -l lsnrplb_aulnx13
[oracle@aulnx13 ~] srvctl stop listener -n aulnx14.au.oracle.com -l lsnrplb_aulnx14
[oracle@aulnx13 ~] srvctl stop listener -n aulnx13.au.oracle.com -l lsnrplb_aulnx13
[oracle@aulnx13 ~] srvctl stop listener -n aulnx14.au.oracle.com -l lsnrplb_aulnx14
2.2 The release of OPatch that is supplied with 10.2.0.4 will not be able to apply a PSU as seen through the errors below.
[oracle@aulnx9 9119284]$ pwd
/home/oracle/patches/9119284
[oracle@aulnx9 9119284]$ which opatch
/u01/oracle/product/10.2.0/db_plb/OPatch/opatch
[oracle@aulnx9 9119284]$ opatch apply
Invoking OPatch 10.2.0.4.2
Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/product/10.2.0/db_plb
Central Inventory : /u01/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /u01/oracle/product/10.2.0/db_plb/oui
Log file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch2010-02-04_08-45-30AM.log
ApplySession failed: Patch ID is null.
System intact, OPatch will not attempt to restore the system
OPatch failed with error code 73
/home/oracle/patches/9119284
[oracle@aulnx9 9119284]$ which opatch
/u01/oracle/product/10.2.0/db_plb/OPatch/opatch
[oracle@aulnx9 9119284]$ opatch apply
Invoking OPatch 10.2.0.4.2
Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/product/10.2.0/db_plb
Central Inventory : /u01/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /u01/oracle/product/10.2.0/db_plb/oui
Log file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch2010-02-04_08-45-30AM.log
ApplySession failed: Patch ID is null.
System intact, OPatch will not attempt to restore the system
OPatch failed with error code 73
2.2.1 To overcome the patch issue apply the latest copy of OPatch to the home via Patch 6880880.
If you already have a later release of OPatch then it is not necessary to perform. this step.
Download Patch 6880880 and following the install instructions in the README
[oracle@aulnx9 patches]$ mv /u01/oracle/product/10.2.0/db_plb/OPatch /u01/oracle/product/10.2.0/db_plb/OPatch_10204
[oracle@aulnx9 patches]$ pwd
/home/oracle/patches
[oracle@aulnx9 patches]$ ls
9119284 OPatch
[oracle@aulnx9 patches]$ cp -rp OPatch/ /u01/oracle/product/10.2.0/db_plb/
[oracle@aulnx9 patches]$ pwd
/home/oracle/patches
[oracle@aulnx9 patches]$ ls
9119284 OPatch
[oracle@aulnx9 patches]$ cp -rp OPatch/ /u01/oracle/product/10.2.0/db_plb/
2.3 Once the new release of OPatch is in place apply the patch to the Standby Site
Please Note: In the Standby Site only the patching of the binaries is performed, there is no need to run the catupgrade/catcpu/catbundle.sql script. as this will be performed through redo apply at the Standby Site.
The example below is applying the patch to a Single Instance Standby Site and its applying PSU Patch 9119284.
[oracle@aulnx9 9119284]$ opatch apply
Invoking OPatch 11.2.0.1.1
Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/product/10.2.0/db_plb
Central Inventory : /u01/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 10.2.0.4.0
OUI location : /u01/oracle/product/10.2.0/db_plb/oui
Log file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch2010-02-04_08-57-57AM.log
Patch history file: /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch_history.txt
ApplySession applying interim patch '9119284' to OH '/u01/oracle/product/10.2.0/db_plb'
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/oracle/product/10.2.0/db_plb')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '9119284' for restore. This might take a while...
Backing up files affected by the patch '9119284' for rollback. This might take a while...
Execution of 'sh /home/oracle/patches/9119284/custom/scripts/pre -apply 9119284 ':
Return Code = 0
Patching component oracle.rdbms.rsf, 10.2.0.4.0...
Updating archive file "/u01/oracle/product/10.2.0/db_plb/lib/libgeneric10.a" with "lib/libgeneric10.a/qcodfdef.o"
..
.
Updating jar file "/u01/oracle/product/10.2.0/db_plb/rdbms/jlib/qsma.jar" with "/rdbms/jlib/qsma.jar/oracle/qsma/QsmaFileManager.class"
Copying file to "/u01/oracle/product/10.2.0/db_plb/rdbms/lib/env_rdbms.mk"
..
.
Running make for target idgmgrl
Running make for target ioracle
Running make for target client_sharedlib
Running make for target itnslsnr
Running make for target iwrap
Running make for target genplusso
ApplySession adding interim patch '9119284' to inventory
Verifying the update...
Inventory check OK: Patch ID 9119284 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9119284 are present in Oracle Home.
--------------------------------------------------------------------------------
********************************************************************************
********************************************************************************
** ATTENTION **
** **
** Please note that the Patch Set Update Installation (PSU Deinstallation) **
** is not complete until all the Post Installation (Post Deinstallation) **
** instructions noted in the Readme accompanying this PSU, have been **
** successfully completed. **
** **
********************************************************************************
********************************************************************************
--------------------------------------------------------------------------------
Execution of 'sh /home/oracle/patches/9119284/custom/scripts/post -apply 9119284 ':
Return Code = 0
The local system has been patched and can be restarted.
OPatch succeeded.
Invoking OPatch 11.2.0.1.1
Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/product/10.2.0/db_plb
Central Inventory : /u01/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 10.2.0.4.0
OUI location : /u01/oracle/product/10.2.0/db_plb/oui
Log file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch2010-02-04_08-57-57AM.log
Patch history file: /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch_history.txt
ApplySession applying interim patch '9119284' to OH '/u01/oracle/product/10.2.0/db_plb'
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/oracle/product/10.2.0/db_plb')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '9119284' for restore. This might take a while...
Backing up files affected by the patch '9119284' for rollback. This might take a while...
Execution of 'sh /home/oracle/patches/9119284/custom/scripts/pre -apply 9119284 ':
Return Code = 0
Patching component oracle.rdbms.rsf, 10.2.0.4.0...
Updating archive file "/u01/oracle/product/10.2.0/db_plb/lib/libgeneric10.a" with "lib/libgeneric10.a/qcodfdef.o"
..
.
Updating jar file "/u01/oracle/product/10.2.0/db_plb/rdbms/jlib/qsma.jar" with "/rdbms/jlib/qsma.jar/oracle/qsma/QsmaFileManager.class"
Copying file to "/u01/oracle/product/10.2.0/db_plb/rdbms/lib/env_rdbms.mk"
..
.
Running make for target idgmgrl
Running make for target ioracle
Running make for target client_sharedlib
Running make for target itnslsnr
Running make for target iwrap
Running make for target genplusso
ApplySession adding interim patch '9119284' to inventory
Verifying the update...
Inventory check OK: Patch ID 9119284 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9119284 are present in Oracle Home.
--------------------------------------------------------------------------------
********************************************************************************
********************************************************************************
** ATTENTION **
** **
** Please note that the Patch Set Update Installation (PSU Deinstallation) **
** is not complete until all the Post Installation (Post Deinstallation) **
** instructions noted in the Readme accompanying this PSU, have been **
** successfully completed. **
** **
********************************************************************************
********************************************************************************
--------------------------------------------------------------------------------
Execution of 'sh /home/oracle/patches/9119284/custom/scripts/post -apply 9119284 ':
Return Code = 0
The local system has been patched and can be restarted.
OPatch succeeded.
2.4 Start the Standby Site database to mount and restart the listener(s).
2.4.1 If the environment is a single instance (non-RAC standby)
Start the listener:
[oracle@aulnx9 9119284]$ lsnrctl start lsnrplb
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 04-FEB-2010 09:06:02
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Starting /u01/oracle/product/10.2.0/db_plb/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 10.2.0.4.0 - Production
System parameter file is /u01/oracle/product/10.2.0/db_plb/network/admin/listener.ora
Log messages written to /u01/oracle/product/10.2.0/db_plb/network/log/lsnrplb.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=aulnx9.au.oracle.com)(PORT=1666)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=aulnx9.au.oracle.com)(PORT=1666)))
STATUS of the LISTENER
------------------------
Alias lsnrplb
Version TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date 04-FEB-2010 09:06:02
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/oracle/product/10.2.0/db_plb/network/admin/listener.ora
Listener Log File /u01/oracle/product/10.2.0/db_plb/network/log/lsnrplb.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=aulnx9.au.oracle.com)(PORT=1666)))
Services Summary...
Service "plb_std_DGMGRL.au.oracle.com" has 1 instance(s).
Instance "plb_std", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 04-FEB-2010 09:06:02
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Starting /u01/oracle/product/10.2.0/db_plb/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 10.2.0.4.0 - Production
System parameter file is /u01/oracle/product/10.2.0/db_plb/network/admin/listener.ora
Log messages written to /u01/oracle/product/10.2.0/db_plb/network/log/lsnrplb.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=aulnx9.au.oracle.com)(PORT=1666)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=aulnx9.au.oracle.com)(PORT=1666)))
STATUS of the LISTENER
------------------------
Alias lsnrplb
Version TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date 04-FEB-2010 09:06:02
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/oracle/product/10.2.0/db_plb/network/admin/listener.ora
Listener Log File /u01/oracle/product/10.2.0/db_plb/network/log/lsnrplb.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=aulnx9.au.oracle.com)(PORT=1666)))
Services Summary...
Service "plb_std_DGMGRL.au.oracle.com" has 1 instance(s).
Instance "plb_std", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
Start the database:
[oracle@aulnx9 9119284]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 15:12:24 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> startup mount
ORACLE instance started.
Total System Global Area 268435456 bytes
Fixed Size 1266968 bytes
Variable Size 138414824 bytes
Database Buffers 125829120 bytes
Redo Buffers 2924544 bytes
Database mounted.
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 15:12:24 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> startup mount
ORACLE instance started.
Total System Global Area 268435456 bytes
Fixed Size 1266968 bytes
Variable Size 138414824 bytes
Database Buffers 125829120 bytes
Redo Buffers 2924544 bytes
Database mounted.
OR
2.4.2 If this is a RAC standby
[oracle@aulnx13 ~]$ srvctl start listener -n aulnx11 -l lsnrplb_aulnx13
[oracle@aulnx13 ~]$ srvctl start listener -n aulnx12 -l lsnrplb_aulnx14
[oracle@aulnx13 ~]$ srvctl start database -d plb_std -o mount
[oracle@aulnx13 ~]$ srvctl start listener -n aulnx12 -l lsnrplb_aulnx14
[oracle@aulnx13 ~]$ srvctl start database -d plb_std -o mount
3. Apply the Patch to the Primary.
Please Note: the primary site may also require that Opatch be upgraded in the same way as it was in the Standby Site via Patch 6880880.
Change to the patch directory at the Primary Site and apply the PSU here
[oracle@aulnx8 patches]$ cd 9119284/
[oracle@aulnx8 9119284]$ pwd
/home/oracle/patches/9119284
[oracle@aulnx8 9119284]$ pwd
/home/oracle/patches/9119284
3.1 Stop all processes running from the home being patched.
This will include listeners and databases etc.
3.1.1 If this is a single instance Primary (non-RAC)
[oracle@aulnx8 9119284]$ lsnrctl stop lsnrplb
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 04-FEB-2010 09:06:28
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=aulnx8.au.oracle.com)(PORT=1666)))
The command completed successfully
[oracle@aulnx8 9119284]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 09:06:36 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 04-FEB-2010 09:06:28
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=aulnx8.au.oracle.com)(PORT=1666)))
The command completed successfully
[oracle@aulnx8 9119284]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 09:06:36 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
OR
3.1.2 If this is a RAC Primary Site
[oracle@aulnx11 ~]$ srvctl stop listener -n aulnx11 -l lsnrplb_aulnx11
[oracle@aulnx11 ~]$ srvctl stop listener -n aulnx12 -l lsnrplb_aulnx12
[oracle@aulnx11 ~]$ srvctl stop database -d plb
[oracle@aulnx11 ~]$ srvctl stop listener -n aulnx12 -l lsnrplb_aulnx12
[oracle@aulnx11 ~]$ srvctl stop database -d plb
3.2 Apply the patch to the Primary Site binaries.
As per the standby site this example is applying Patchset Update 9119284 to a single instance Primary Site.
[oracle@aulnx8 9119284]$ opatch apply
Invoking OPatch 11.2.0.1.1
Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/product/10.2.0/db_plb
Central Inventory : /u01/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 10.2.0.4.0
OUI location : /u01/oracle/product/10.2.0/db_plb/oui
Log file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch2010-02-04_09-10-28AM.log
Patch history file: /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch_history.txt
ApplySession applying interim patch '9119284' to OH '/u01/oracle/product/10.2.0/db_plb'
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/oracle/product/10.2.0/db_plb')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '9119284' for restore. This might take a while...
Backing up files affected by the patch '9119284' for rollback. This might take a while...
Execution of 'sh /home/oracle/patches/9119284/custom/scripts/pre -apply 9119284 ':
Return Code = 0
Patching component oracle.rdbms.rsf, 10.2.0.4.0...
Updating archive file "/u01/oracle/product/10.2.0/db_plb/lib/libgeneric10.a" with "lib/libgeneric10.a/qcodfdef.o"
..
.
Updating jar file "/u01/oracle/product/10.2.0/db_plb/rdbms/jlib/qsma.jar" with "/rdbms/jlib/qsma.jar/oracle/qsma/QsmaFileManager.class"
..
.
Running make for target client_sharedlib
Running make for target idgmgrl
Running make for target ioracle
Running make for target client_sharedlib
Running make for target itnslsnr
Running make for target iwrap
Running make for target genplusso
ApplySession adding interim patch '9119284' to inventory
Verifying the update...
Inventory check OK: Patch ID 9119284 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9119284 are present in Oracle Home.
--------------------------------------------------------------------------------
********************************************************************************
********************************************************************************
** ATTENTION **
** **
** Please note that the Patch Set Update Installation (PSU Deinstallation) **
** is not complete until all the Post Installation (Post Deinstallation) **
** instructions noted in the Readme accompanying this PSU, have been **
** successfully completed. **
** **
********************************************************************************
********************************************************************************
--------------------------------------------------------------------------------
Execution of 'sh /home/oracle/patches/9119284/custom/scripts/post -apply 9119284 ':
Return Code = 0
The local system has been patched and can be restarted.
OPatch succeeded.
Invoking OPatch 11.2.0.1.1
Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/product/10.2.0/db_plb
Central Inventory : /u01/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 10.2.0.4.0
OUI location : /u01/oracle/product/10.2.0/db_plb/oui
Log file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch2010-02-04_09-10-28AM.log
Patch history file: /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch_history.txt
ApplySession applying interim patch '9119284' to OH '/u01/oracle/product/10.2.0/db_plb'
Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/oracle/product/10.2.0/db_plb')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '9119284' for restore. This might take a while...
Backing up files affected by the patch '9119284' for rollback. This might take a while...
Execution of 'sh /home/oracle/patches/9119284/custom/scripts/pre -apply 9119284 ':
Return Code = 0
Patching component oracle.rdbms.rsf, 10.2.0.4.0...
Updating archive file "/u01/oracle/product/10.2.0/db_plb/lib/libgeneric10.a" with "lib/libgeneric10.a/qcodfdef.o"
..
.
Updating jar file "/u01/oracle/product/10.2.0/db_plb/rdbms/jlib/qsma.jar" with "/rdbms/jlib/qsma.jar/oracle/qsma/QsmaFileManager.class"
..
.
Running make for target client_sharedlib
Running make for target idgmgrl
Running make for target ioracle
Running make for target client_sharedlib
Running make for target itnslsnr
Running make for target iwrap
Running make for target genplusso
ApplySession adding interim patch '9119284' to inventory
Verifying the update...
Inventory check OK: Patch ID 9119284 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9119284 are present in Oracle Home.
--------------------------------------------------------------------------------
********************************************************************************
********************************************************************************
** ATTENTION **
** **
** Please note that the Patch Set Update Installation (PSU Deinstallation) **
** is not complete until all the Post Installation (Post Deinstallation) **
** instructions noted in the Readme accompanying this PSU, have been **
** successfully completed. **
** **
********************************************************************************
********************************************************************************
--------------------------------------------------------------------------------
Execution of 'sh /home/oracle/patches/9119284/custom/scripts/post -apply 9119284 ':
Return Code = 0
The local system has been patched and can be restarted.
OPatch succeeded.
3.3 Upgrade/Patch the RDBMS and dictionary objects.
In a RAC environment this is performed from once instance only with the remaining cluster instances down and not running. The example below once again shows patch application to a single instance (non-RAC) Primary Site and is applying PSU 9119284.
Change to the $ORACLE_HOME/rdbms/admin directory where the patch has been applied and perform. the following.
[oracle@aulnx8 9119284]$ cd $ORACLE_HOME/rdbms/admin
[oracle@aulnx8 admin]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 09:25:09 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> startup restrict
ORACLE instance started.
Total System Global Area 268435456 bytes
Fixed Size 1266968 bytes
Variable Size 138414824 bytes
Database Buffers 125829120 bytes
Redo Buffers 2924544 bytes
Database mounted.
Database opened.
SQL> @catbundle psu apply
PL/SQL procedure successfully completed.
..
.
SQL> COMMIT;
Commit complete.
SQL> SPOOL off
SQL> SET echo off
Check the following log file for errors:
/u01/oracle/product/10.2.0/db_plb/cfgtoollogs/catbundle/catbundle_PSU_PLB_APPLY_2010Feb04_09_25_28.log
[oracle@aulnx8 admin]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 09:25:09 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> startup restrict
ORACLE instance started.
Total System Global Area 268435456 bytes
Fixed Size 1266968 bytes
Variable Size 138414824 bytes
Database Buffers 125829120 bytes
Redo Buffers 2924544 bytes
Database mounted.
Database opened.
SQL> @catbundle psu apply
PL/SQL procedure successfully completed.
..
.
SQL> COMMIT;
Commit complete.
SQL> SPOOL off
SQL> SET echo off
Check the following log file for errors:
/u01/oracle/product/10.2.0/db_plb/cfgtoollogs/catbundle/catbundle_PSU_PLB_APPLY_2010Feb04_09_25_28.log
Note: If this was a normal Patchset you would run:
SQL>@catupgrade
If this were a Critical Patch Update you would run:
SQL>@catcpu
SQL>@catupgrade
If this were a Critical Patch Update you would run:
SQL>@catcpu
4. Re-establish the Data Guard environment at the Primary Site.
4.1.1 In a Single Instance Primary restart processes running from the patched ORACLE_HOME.
This will include the listener, database instances, ASM instance and any other processes that were previously running from this ORACLE_HOME.
[oracle@aulnx8 admin]$ lsnrctl start lsnrplb
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 04-FEB-2010 09:45:43
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Starting /u01/oracle/product/10.2.0/db_plb/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 10.2.0.4.0 - Production
System parameter file is /u01/oracle/product/10.2.0/db_plb/network/admin/listener.ora
Log messages written to /u01/oracle/product/10.2.0/db_plb/network/log/lsnrplb.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=aulnx8.au.oracle.com)(PORT=1666)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=aulnx8.au.oracle.com)(PORT=1666)))
STATUS of the LISTENER
------------------------
Alias lsnrplb
Version TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date 04-FEB-2010 09:45:43
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/oracle/product/10.2.0/db_plb/network/admin/listener.ora
Listener Log File /u01/oracle/product/10.2.0/db_plb/network/log/lsnrplb.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=aulnx8.au.oracle.com)(PORT=1666)))
Services Summary...
Service "plb_prm_DGMGRL.au.oracle.com" has 1 instance(s).
Instance "plb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 04-FEB-2010 09:45:43
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Starting /u01/oracle/product/10.2.0/db_plb/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 10.2.0.4.0 - Production
System parameter file is /u01/oracle/product/10.2.0/db_plb/network/admin/listener.ora
Log messages written to /u01/oracle/product/10.2.0/db_plb/network/log/lsnrplb.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=aulnx8.au.oracle.com)(PORT=1666)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=aulnx8.au.oracle.com)(PORT=1666)))
STATUS of the LISTENER
------------------------
Alias lsnrplb
Version TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date 04-FEB-2010 09:45:43
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/oracle/product/10.2.0/db_plb/network/admin/listener.ora
Listener Log File /u01/oracle/product/10.2.0/db_plb/network/log/lsnrplb.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=aulnx8.au.oracle.com)(PORT=1666)))
Services Summary...
Service "plb_prm_DGMGRL.au.oracle.com" has 1 instance(s).
Instance "plb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
OR
4.1.2 If this were a RAC environment restart the listeners on each node running from the ORACLE_HOME being patched using srvctl.
[oracle@aulnx11 ~]$ srvctl start listener -n aulnx11 -l lsnrplb_aulnx11
[oracle@aulnx11 ~]$ srvctl start listener -n aulnx12 -l lsnrplb_aulnx12
[oracle@aulnx11 ~]$ srvctl start listener -n aulnx12 -l lsnrplb_aulnx12
4.1.3. Force the Primary to register its services with the listener.
[oracle@aulnx8 admin]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 09:45:50 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> alter system register;
System altered.
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Feb 4 09:45:50 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> alter system register;
System altered.
4.2.1 In a single instance (non-RAC) disable restricted session to allow end user connectivity.
SQL> alter system disable restricted session;
System altered.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
System altered.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
OR
4.2.2 In a RAC Primary Site restart the primary site and all its instances
[oracle@aulnx11 ~] srvctl stop database -d plb
[oracle@aulnx11 ~] srvctl start database -d plb
[oracle@aulnx11 ~] srvctl start database -d plb
If there are additional RAC services used to connect to this database these will need to be restarted using srvctl
[oracle@aulnx11 ~] srvctl start service -d plb
4.3 Re-enable log shipping to the Standby Site.
This will allow the RDBMS changes made through running catupgrade/catbundle/catcpu can be applied to the standby.
4.3.1 If you are using a Data Guard Broker configuration
[oracle@aulnx8 admin]$ dgmgrl
DGMGRL for Linux: Version 10.2.0.4.0 - Production
Copyright (c) 2000, 2005, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
DGMGRL> connect /
Connected.
DGMGRL> edit database plb_prm set state='ONLINE';
Succeeded.
DGMGRL> exit
DGMGRL for Linux: Version 10.2.0.4.0 - Production
Copyright (c) 2000, 2005, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
DGMGRL> connect /
Connected.
DGMGRL> edit database plb_prm set state='ONLINE';
Succeeded.
DGMGRL> exit
OR
4.3.2 If you are using sqlplus re-enable the archive destination used to ship archives from the Primary Site to the Standby Site
SQL> alter system set log_archive_dest_state_X=enable scope=both sid='*'
Where X is the number of the destination used for shipping redo to the standby site.
5. The Primary will then resume shipping and media recovery will continue at the Standby site and apply the changes made through the patch application at the Primary Site.
From the alert log at the Standby you will see logs generated during the running of the cat scripts shipped and applied to the standby:
Thu Feb 4 09:45:57 2010
RFS[1]: Archived Log: '/oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_79_5pmz52s2_.arc'
RFS[1]: Archived Log: '/oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_76_5pmz55l4_.arc'
RFS[1]: Archived Log: '/oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_77_5pmz55oo_.arc'
RFS[1]: Archived Log: '/oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_78_5pmz561c_.arc'
Thu Feb 4 09:46:24 2010
Media Recovery Log /oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_76_5pmz55l4_.arc
Media Recovery Log /oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_77_5pmz55oo_.arc
Media Recovery Log /oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_78_5pmz561c_.arc
Thu Feb 4 09:46:44 2010
Media Recovery Log /oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_79_5pmz52s2_.arc
Media Recovery Waiting for thread 1 sequence 80
RFS[1]: Archived Log: '/oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_79_5pmz52s2_.arc'
RFS[1]: Archived Log: '/oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_76_5pmz55l4_.arc'
RFS[1]: Archived Log: '/oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_77_5pmz55oo_.arc'
RFS[1]: Archived Log: '/oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_78_5pmz561c_.arc'
Thu Feb 4 09:46:24 2010
Media Recovery Log /oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_76_5pmz55l4_.arc
Media Recovery Log /oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_77_5pmz55oo_.arc
Media Recovery Log /oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_78_5pmz561c_.arc
Thu Feb 4 09:46:44 2010
Media Recovery Log /oradata/flash_recovery_area/PLB_STD/archivelog/2010_02_04/o1_mf_1_79_5pmz52s2_.arc
Media Recovery Waiting for thread 1 sequence 80
6. To check the patch has been applied successfully at the primary and standby sites perform.
In this case it was the PSU Patch 9119284.
Please Note: Currently (10.2.0.4 PSU's) do not update the header information for tools like sqlplus nor the versioning information for the database. These will remain at 10.2.0.4.0 and NOT updated to 10.2.0.4.3 for example after the application of the PSU. You will only be able to see the PSU's application through the inventory or looking at the version
[oracle@aulnx8 admin]$ opatch lsinventory
Invoking OPatch 11.2.0.1.1
Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/product/10.2.0/db_plb
Central Inventory : /u01/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 10.2.0.4.0
OUI location : /u01/oracle/product/10.2.0/db_plb/oui
Log file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch2010-02-04_10-01-18AM.log
Patch history file: /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch_history.txt
Lsinventory Output file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/lsinv/lsinventory2010-02-04_10-01-18AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (2):
Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 3 10.2.0.4.0
There are 2 products installed in this Oracle Home.
Interim patches (1) :
Patch 9119284 : applied on Thu Feb 04 10:15:51 EST 2010
Unique Patch ID: 11999265
Created on 17 Dec 2009, 03:32:08 hrs PST8PDT
Bugs fixed:
6418420, 7835247, 7207654, 7592346, 6724797, 7936993, 7331867, 9093300
..
.
7523787, 6029179, 5231155, 6455659
--------------------------------------------------------------------------------
OPatch succeeded.
Invoking OPatch 11.2.0.1.1
Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/product/10.2.0/db_plb
Central Inventory : /u01/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.1
OUI version : 10.2.0.4.0
OUI location : /u01/oracle/product/10.2.0/db_plb/oui
Log file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch2010-02-04_10-01-18AM.log
Patch history file: /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/opatch_history.txt
Lsinventory Output file location : /u01/oracle/product/10.2.0/db_plb/cfgtoollogs/opatch/lsinv/lsinventory2010-02-04_10-01-18AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (2):
Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 3 10.2.0.4.0
There are 2 products installed in this Oracle Home.
Interim patches (1) :
Patch 9119284 : applied on Thu Feb 04 10:15:51 EST 2010
Unique Patch ID: 11999265
Created on 17 Dec 2009, 03:32:08 hrs PST8PDT
Bugs fixed:
6418420, 7835247, 7207654, 7592346, 6724797, 7936993, 7331867, 9093300
..
.
7523787, 6029179, 5231155, 6455659
--------------------------------------------------------------------------------
OPatch succeeded.
OR
In the case of a PSU examine the registry history view.
SQL> select ACTION_TIME,ACTION,NAMESPACE,VERSION,BUNDLE_SERIES,ID from registry$history;
ACTION_TIME
---------------------------------------------------------------------------
ACTION NAMESPACE
------------------------------ ------------------------------
VERSION BUNDLE_SERIES ID
------------------------------ ------------------------------ ----------
04-FEB-10 09.26.25.026334 AM
APPLY SERVER
10.2.0.4 PSU 3
ACTION_TIME
---------------------------------------------------------------------------
ACTION NAMESPACE
------------------------------ ------------------------------
VERSION BUNDLE_SERIES ID
------------------------------ ------------------------------ ----------
04-FEB-10 09.26.25.026334 AM
APPLY SERVER
10.2.0.4 PSU 3
References
NOTE:241374.1 - Script. to Collect Data Guard Primary Site Diagnostic InformationNOTE:241438.1 - Script. to Collect Data Guard Physical Standby Diagnostic Information
NOTE:437276.1 - Upgrading Oracle Database with a Logical Standby Database In Place
NOTE:756671.1 - Oracle Recommended Patches -- Oracle Database
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/230160/viewspace-672978/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/230160/viewspace-672978/