In this Document
| Goal |
| Solution |
| 1.) If the filesystem gets full in the middle of applying a patch, what is the process to recover? |
| 2.) Can any of the space be reclaimed from $ORACLE_HOME/.patch_storage? |
| 3.) Usage of 'opatch util cleanup' to reclaim space from $ORACLE_HOME/.patch_storage |
| 4.) In preparation for patching systems, is there any way to predict the amount of space that will be required in $ORACLE_HOME/.patch_storage? |
| References |
APPLIES TO:
Oracle Universal Installer - Version 10.2.0.1 and laterInformation in this document applies to any platform.
***Checked for relevance on 03-Jan-2013***
GOAL
When applying interim patches or Security Patch Update (SPU) (formerly Critical Patch Update (CPU)) patches, OPatch takes a big amount of disk space under $ORACLE_HOME/.patch_storage to store a backup of the affected libraries and modules that have been updated. It even could happen that a disk becomes full in the middle of the patching process.
Regarding this topic, the following questions will be answered in this article:
1. ) If the filesystem gets full in the middle of applying a patch, what is the process to recover?
2. ) Can any of the space be reclaimed from $ORACLE_HOME/.patch_storage?
3. ) Usage of 'opatch util cleanup' to reclaim space from $ORACLE_HOME/.patch_storage
4. ) In preparation for patching systems, is there any way to predict the amount of space that will be required in $ORACLE_HOME/.patch_storage?
SOLUTION
1.) If the filesystem gets full in the middle of applying a patch, what is the process to recover?
If the filesystem gets full in the middle of applying a patch, Opatch will get a 'No space left on device' error from the Operating system and will show a message on the screen and log file, similar to the following (the exact first message depends on the operation being done at the moment the filesystem gets full - in the case below it was trying to create a new directory):
SEVERE: OUI-67073:UtilSession failed: ApplySession failed in system modification phase... ' PatchObject::createFP() throws IOException: failed to create dir "<ORACLE_HOME_PATH>/.patch_storage/<SUBDIRECTORIES>"'
INFO:--------------------------------------------------------------------------------
INFO:The following warnings have occurred during OPatch execution:
INFO:1) OUI-67124:ApplySession failed in system modification phase... ' PatchObject::createFP() throws IOException: failed to create dir "<ORACLE_HOME_PATH>/.patch_storage/<SUBDIRECTORIES>"'
INFO:2) OUI-67124:
NApply was not able to restore the home. Please invoke the following scripts:
- restore.[sh,bat]
- make.txt (Unix only)
to restore the ORACLE_HOME. They are located under
"<ORACLE_HOME_PATH>/.patch_storage/NApply/<TIMESTAMP>"
INFO:--------------------------------------------------------------------------------
.....
Run 'restore.sh' as instructed by OPatch. It will perform a restore of the ORACLE_HOME and the local inventory of the ORACLE_HOME as well, to the state they were before starting to apply the patch.
Run also make.txt if that file is non-empty. Make.txt will have contents only if the linking phase of patch apply has been reached. If error is raised before linking , make.txt will be an empty file and should be ignored.
After running restore.sh (and make.txt if needed), read section 2 and reclaim the space from $ORACLE_HOME/.patch_storage using the 'opatch util cleanup' (as explained in section 3).
In the above situation, sometimes, Opatch will be able to restore the system by itself. In this case, you will receive a final message similar to the following, and you should proceed to section 2 directly, as no run of restore.sh is needed.
INFO:Restoring "<ORACLE_HOME_PATH>" to the state prior to running NApply...
INFO:Checking if OPatch needs to invoke 'make' to restore some binaries...
INFO:OPatch was able to restore your system. Look at log file and timestamp of each file to make sure your system is in the state prior to applying the patch.
WARNING:OUI-67124:
NApply restored the home. Please check your ORACLE_HOME to make sure:
- files are restored properly.
- binaries are re-linked correctly.
(use restore.[sh,bat] and make.txt (Unix only) as a reference. They are located under
"<ORACLE_HOME_PATH>/.patch_storage/NApply/<TIMESTAMP>"
A different behavior happens in case the 'no space left on device' condition is flagged while doing archive operations with OS command 'ar', OPatch reports a 'Warning' instead of an 'Error', and ask for confirmation to continue or not:
WARNING:OUI-67160:ApplySession for patch ID '<PATCH#>': Could not back up following actions for patch rollback:
1) File not backed up from <ORACLE_HOME_PATH>/.patch_storage/<SUBDIRECTORIES>/<MODULE> to <ORACLE_HOME_PATH>/.patch_storage/<SUBDIRECTORIES>/<MODULE>... 'ar: <MODULE>: cannot write
No space left on device
'
2) File not backed up from <ORACLE_HOME_PATH>/.patch_storage/<SUBDIRECTORIES>/<MODULE> to <ORACLE_HOME_PATH>/.patch_storage/<SUBDIRECTORIES>/<MODULE>... 'ar: <MODULE>: cannot write
No space left on device
'
INFO:
Do you want to proceed? [y|n]
.....
In this case, the answer should be 'N' to stop patch apply. Even if you try to make room while OPatch is waiting for input, the ar operations that failed will not be retried, and you will have to end doing the restore manually later.
After answering 'N', OPatch will try to restore the system:
INFO:User Responded with: N
WARNING:OUI-67124:ApplySession failed in system modification phase... 'User requested not to continue.'
INFO:
Restoring "<ORACLE_HOME_PATH>" to the state prior to running NApply...
INFO:Checking if OPatch needs to invoke 'make' to restore some binaries...
INFO:OPatch was able to restore your system. Look at log file and timestamp of each file to make sure your system is in the state prior to applying the patch.
WARNING:OUI-67124:
NApply restored the home. Please check your ORACLE_HOME to make sure:
- files are restored properly.
- binaries are re-linked correctly.
(use restore.[sh,bat] and make.txt (Unix only) as a reference. They are located under
"<ORACLE_HOME_PATH>/.patch_storage/NApply/<TIMESTAMP>"
If OPatch succeeds as above, you could proceed to read section 2 and reclaim the space from $ORACLE_HOME/.patch_storage using the 'opatch util cleanup' (as explained in section 3).
If not, raise a Service Request with Oracle Global Customer Support to investigate it further before proceeding.
2.) Can any of the space be reclaimed from $ORACLE_HOME/.patch_storage?
Starting with 10.2, Opatch does not backup only the affected modules, it also takes a backup of the complete affected libraries to $ORACLE_HOME/.patch_storage/<patchid>/backup/<lib_directory_name>/<library_name>.
If patch does not modify any library, the backup subdirectory will not contain the lib subdirectories, just the needed subdirectories to hold the affected modules.
This causes the .patch_storage directory to grow accordingly to the size of the libraries and modules being patched.
The biggest library in $ORACLE_HOME/lib is libserver10.a and it is one of the most commonly affected by RDBMS patches.
As an example, in a Sun SPARC Solaris system, it has a size around 150-170 Mb. This means that if 10 patches affecting libserver10.a are installed, more than 1.5-1.7 Gb of free space will be needed in the filesystem where $ORACLE_HOME/.patch_storage will reside to avoid a 'disk full' error.
Some of these backup directories are needed to allow rollback of the interim patches applied and should be kept until they are not needed anymore (as when you install a patchset).
For more information on the reasons why they cannot be deleted, please see:
This new OPatch backup behavior is to allow an immediate restore in case of failure while patching.
The drawback of this method appears when applying an SPU/CPU patch, as it is composed of several patches (named molecules), but because the patch is applied as a whole, the space required for backup corresponds to the addition of all the molecules applied.
Several defect reports have just been raised to Development to modify this behavior and allow an SPU/CPU patch to be applied with much less free space requirements.
This issue was logged in:
Bug 6798330 OPATCH 10.2.0.X BACKUP THE COMPLETE LIBRARY, NOT THE OBJECT MODULE
and has since been resolved in opatch versions 10.2.0.4.3. To download the latest version of OPATCH for a particular Oracle release review:
Also as mentioned before, when a patchset is installed, these backups are no longer needed, but currently neither Oracle Universal Installer (OUI) nor OPatch do an automatic cleanup of unneeded backups after a patchset install.
For example:
Interim patches have been installed on top of 10.2.0.1.
Opatch saves a backup in the $ORACLE_HOME/.patch_storage directory before applying an interim patch (for later rollback if necessary).
Then a patchset 10.2.0.5 is installed, during which all 10.2.0.1 patches are deinstalled.
The backups for all previously installed 10.2.0.1 interim patches still exist in the .patch_storage area, but these backups are not needed anymore (its not possible to rollback a 10.2.0.1 patch from a 10.2.0.5 environment), and can be deleted.
There was an enhancement request logged about the lack of this feature in:
Bug 5256432 PATCHSET INSTALLATION DOES NOT CLEAN UP .PATCH_STORAGE
and is now available in opatch versions 10.2.0.4.3 and above.
3.) Usage of 'opatch util cleanup' to reclaim space from $ORACLE_HOME/.patch_storage
With the latest OPatch versions for 10.2.0.4.3 and above, available for download as Patch 6880880, there is a new command 'util' which can perform several utility actions.
One of them is the cleanup of all the backup directories under .patch_storage not needed for rollback purposes.
To see all the different option for the util command issue the following command:
To delete the directories not needed for rollback interim patches use the 'cleanup' option.
'opatch util cleanup -help' will show the associated help for the cleanup option:
$ opatch util cleanup -help
Invoking OPatch 10.2.0.4.3
Oracle Interim Patch Installer version 10.2.0.4.3
Copyright (c) 2007, Oracle Corporation. All rights reserved.
UTIL session
DESCRIPTION
This utility cleans up 'restore.sh,make.txt' files and 'rac,scratch,backup'
directories of the.patch_storage directory of Oracle Home.If -ps option is used,
then, it cleans the above specified areas only for that patch, else for all
patches under ORACLE_HOME/.patch_storage. You will be still able to
rollback patches after this cleanup.
SYNTAX
opatch util cleanup [-invPtrLoc <Path to oraInst.loc> ]
[-jre <LOC> ] [-oh <ORACLE_HOME> ]
[-silent] [-force] [-report]
[-ps patch ID with time stamp, this will
be located under ORACLE_HOME/.patch_storage/]
....
An example of usage:
Invoking OPatch 10.2.0.4.3
Oracle Interim Patch Installer version 10.2.0.4.3
Copyright (c) 2007, Oracle Corporation. All rights reserved.
UTIL session
Oracle Home : /home/oracle/10.2.0.3
Central Inventory : /home/oracle/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.3
OUI version : 10.2.0.3.0
OUI location : /home/oracle/10.2.0.3/oui
Log file location : /home/oracle/10.2.0.3/cfgtoollogs/opatch/opatch2008-02-12_10-33-40AM.log
Invoking utility "cleanup"
OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
You will be still able to rollback patches after this cleanup.
Do you want to proceed? [y|n]
y
User Responded with: Y
Size of directory "/home/oracle/10.2.0.3/.patch_storage" before cleanup is 438057972 bytes.
Size of directory "/home/oracle/10.2.0.3/.patch_storage" after cleanup is 198415929 bytes.
UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
deleted, Please refer log file.
OPatch succeeded.
$
Additionally, further cleanup of the $ORACLE_HOME/.patch_storage is possible if there are directories from patches applied to previous versions. This can be done manually as follows:
1. run command:
$ opatch lsinventory
2. Remove all the sub-directories from $ORACLE_HOME/.patch_storage that are not present in the list of installed patches. Directory names would be prefaced with the patchid for example:
13343438_<timestamp>
If there is no possibility of obtaining enough free space in the filesystem through cleaning, you should consider a solution at Operating System level, such as adding more disk space to the filesystem, or to create a symbolic link to another filesystem where enough free space exists and with the right OS permissions.
4.) In preparation for patching systems, is there any way to predict the amount of space that will be required in $ORACLE_HOME/.patch_storage?
Doing some manual calculations could give an estimate of how much free space will be needed to apply a certain patch. The method is:
a) Uncompress the patch to be applied and analyze its directory structure.
b) Note each library name under lib, lib32, rdbms/lib, rdbms/lib32, sysman/lib32, etc subdirectories.
c) Look for the size of the libraries affected in the corresponding subdirectories on $ORACLE_HOME.(i.e lib corresponds to $ORACLE_HOME/lib, rdbms/lib32 corresponds to $ORACLE_HOME/rdbms/lib32.
d) Sum the sizes of all the libs involved, plus the 4 times the size of the uncompressed patch tree itself, plus 2 Mb for the backup size of the needed local inventory directories for the $ORACLE_HOME being patched.
e) You have a roughly estimate of the free space needed.
f) Add always an additional 50-100 Mb security overhead to allow backups done by the relink operations. These relink backups(i.e 'oracleO', 'expO', etc) are overwritten each time a relink is performed, so there will be no grow on that branch.
Lets see a couple of examples
1) A simple interim patch, like the fix for 4087161 :
-4087161
+---etc
+ +---config
+ +---xml
+---files
+---rdbms
+---admin
+---lib
+ +---libdbtools10.a
+---lib32
+ +---libdbtools10.a
+---xml
+---xsl
So, in this patch, libdbtools10.a is modified in both 32 and 64 bit version.
Now we note the size of libdbtools10.a on both directories: 8 Mb in lib and 6 Mb in lib32. Uncompressed patch size is 1.5 Mb and we use the fixed 2 Mb size for local Inventory.
Sum the sizes:
4*Patch : 6.0 Mb
Local Inv : 2.0 Mb
--------
22.5 Mb
Backup directory will require around 22 Mb
2) A CPU January 2008 patch like the fix 6646853
lib
+---libn10.a 3 times
lib32
+---libn10.a 3 times
lib
+---libserver10.a 11 times
sysman
+---lib32
+---libnmehl.a once
+---libnmem.a once
+---libnmexml.a once
lib
+---libordsdo10.a 3 times
Other molecules do not touch libraries, and for these, its size is computed in the size of the uncompressed patch itself.
Please note that each time a library is updated, it is fully backed up. The computations follow:
Size of lib/libn10.a: 4.5 * 3 = 13.5 Mb
Size of lib32/libn10.a: 3.7 * 3 = 11.1 Mb
Size of lib/libserver10.a: 151.0 * 11 = 1661.0 Mb
Size of sysman/lib32/libnmehl.a: 0.25 * 1 = 0.25 Mb
Size of sysman/lib32/libnmem.a: 0.15 * 1 = 0.15 Mb
Size of sysman/lib32/libnmexml.a: 0.05 * 1 = 0.05 Mb
Size of lib/libordsdo10.a: 5.0 * 3 = 15.0 Mb
Libraries : 1701.05 Mb
4*Patch : 152.0 Mb
Local Inv : 2.0 Mb
-----------
1859.05 Mb
Hence, on the system used for the examples, backup directory will need around 1.8 Gb of free space as a maximum. The real space needed could be much less if previous CPU was already applied, as many molecules will be present on the system.
As you could see in the last example, if libserver10.a is updated by the patch, all the other calculations become superfluous, so for a fast rough estimate, it will be enough to multiply the number of times libserver10.a is updated by the size of the library, plus a discretionary 10%-15% safety margin.
REFERENCES
BUG:6798330 - OPATCH 10.2.0.X BACKUP THE COMPLETE LIBRARY, NOT THE OBJECT MODULE
BUG:6806976 - CPUJAN2008 IS TAKING 2.1 GB SPACE IN ORACLE HOME
BUG:6811008 - CPU JANUARY 2008 CONSUMES NEARLY 1621 MB OF DISK SPACE
NOTE:403218.1 - Can You Delete $ORACLE_HOME/.patch_storage Directory ?
NOTE:1351428.1 - Information Center: Patching and Maintaining Oracle Database Server/Client Installations
扫一扫
2253
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
