Today I will step on mined territory. Can I cleanup the previous Patch Bundles with opatch? Or can I just wipe out the old ones? That is a question which will bring people with temper quickly close to explosion. But let me start step by step, especially for those of you who aren’t familiar with this topic.
The secret directories
You L O V E secrets, don’t you? No? Well … then let me start with a secret which isn’t a secret to most people making it even more obscure.
When you install a patch with opatch or opatchauto, there a lot of files will be stored in a hidden directory:
- $ORACLE_HOME/.patch_storage
- $ORACLE_HOME/.opatchauto_storage
Why are these directories hidden? I seriously have no idea. Maybe to indicate “Don’t touch anything! No Trespassing! Stay Away! Private Property“. But as usual when you try to hide something, you just make it more interesting. I just doubt that this was the intention.
Anyhow, when you look closer, especially when you have some sort of patch history like I have in my environments, you may quickly realized: There is a lot of space used.
In my case, these directories all together consume a bit over 10GB in my environment:
[CDB2] oracle@hol:/u01/app/oracle/product/19/.patch_storage $ ls 32218454_Jan_14_2021_16_24_38 32545013_Apr_16_2021_07_40_05 32904851_Jul_20_2021_09_21_24 33192694_Sep_14_2021_16_48_27 33192793_Oct_15_2021_07_09_52 33197296_Oct_4_2021_00_44_36 33515361_Jan_13_2022_06_14_07 33561310_Jan_5_2022_08_13_10 interim_inventory.txt LatestOPatchSession.properties NApply newdirs.txt record_inventory.txt 10017056
Don’t wipe it out!
Rule no.1 given by the patching owners: Don’t wipe out this directory. It may be needed to rollback a patch later on.
Still I know from some people who remove the entire directory from time to time to free up space. And so far, I haven’t heard anywhere that this will break a future patch. If I did this in my environment, I made sure that the last two subdirectories were always kept – just in case. But I don’t have many one-off or merge patches. Hence, I can’t really tell you whether this was clever or just dumb. And this brings me back to …
Rule no.1:
Don’t wipe out .patch_storage and .opatchauto_storage.
opatch util cleanup
Let me make this chapter pretty short.
Rule no.2:
opatch util cleanup does not do what you’d expect it to do. And sometimes it does a bit more than you’d expect it to do.
What is the solution?
Addition May 16, 2023:
Before you read on, better jump to this blog post. With opatch 37, released in April 2023, a cleanup option is available – and this significantly improves performance:
For a long long time many customers asked for a solution. Just briefly browsing the bug database reveals a decent number of bugs and enhancement request filed against or for opatch to do a proper cleanup.
The most interesting one I could find was:
- Enh 31138227 – $OH/.PATCH_STORAGE KEEP N-1 VERSIONS ONLY FOR BETTER SPACE CONSUMPTION
I would have hoped for such a solution since I usually do this on my Linux system:
# package-cleanup --oldkernels --count=1
which removes older kernels I won’t need or use anymore.
From above enhancement request I learned that there is a new opatch option available from version 26 on:
./opatch util archive -patches 123456,1122334 -archive_dir /dirx
The first problem here is that the usual -help option in opatch doesn’t tell you about it. And since the Enhancement Request is non-public you can’t see the details. But the secret keyword is: opatch util archive -help
Annotation:
Actually I learned from the feature owners that there is a simple reason why the help didn’t show you the command options on the help’s higher level – the feature was not “live” yet. Since I learned about it through a bug I filed, I tried it out and wasn’t aware that it’s not public yet. Still, it works fine.
$ ./opatch util archive -help
Oracle Interim Patch Installer version 12.2.0.1.28
Copyright (c) 2022, Oracle Corporation. All rights reserved. DESCRIPTION
This operation helps to archive the patches stored under
ORACLE_HOME/.patch_storage to custom storage location.
NOTE: Patch directories in .patch_storage will be used during
rollback operation.
SYNTAX
opatch util archive [-patches <List of Patch ids with comma separated>]
[-archive_dir <path to store the archive files>]
OPTIONS
-patches
A comma separated list of patches to be given for
archive.
-archive_dir
Specify custom storage path to store the patches
which are going to be archived.
NOTE: All options are mandatory.
RECOMMENDATION :
To archive patches, opatch recommends the following
1. Archiving inactive patches :
Run the following command to get inactive patches : "OPatch/opatch lsinv -inactive"
Example
opatch util archive -patches 123456,1122334 -archive_dir /dir1/dir2/
OPatch succeeded.
So this is the new command: opatch util archive -patches … -archive_dir …
Don’t get puzzled by the brackets in the command syntax. All options are MANDATORY, and not optional.
This is the promise to solve the problem of gigabytes of old patch content just wasting space on your hard drive. Let’s see whether this solves your problem as well.
Which patch directories are candidates?
When you search for inactive homes, then this opatch command is most helpful but gives a lengthy output:
$ ./opatch lsinv -inactive
It tells you the patch numbers of the inactive ones. Those are the ones you want to pass on to the archive command below.
opatch util archive
As you read before, even though I cleaned up illegally by myself before, still more than 10GB get consumed on my machine.
Still, the archive command will do a space check (see below in What if …)
$ ./opatch util archive -patches 32218454,32545013,32904851,33192694,33192793,33197296,33515361,33561310 -archive_dir /media/sf_TEMP/ Oracle Interim Patch Installer version 12.2.0.1.28 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.28 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-01-20_14-08-15PM_1.log Invoking utility "archive" Archiving Patch - 33515361 .... Sucessfully archived patch - 33515361 in /media/sf_TEMP/33515361_Jan_13_2022_06_14_07.zip Archiving Patch - 32545013 .... Sucessfully archived patch - 32545013 in /media/sf_TEMP/32545013_Apr_16_2021_07_40_05.zip Archiving Patch - 32904851 .... Sucessfully archived patch - 32904851 in /media/sf_TEMP/32904851_Jul_20_2021_09_21_24.zip Archiving Patch - 33197296 .... Sucessfully archived patch - 33197296 in /media/sf_TEMP/33197296_Oct_4_2021_00_44_36.zip Archiving Patch - 33192793 .... Sucessfully archived patch - 33192793 in /media/sf_TEMP/33192793_Oct_15_2021_07_09_52.zip Archiving Patch - 33192694 .... Sucessfully archived patch - 33192694 in /media/sf_TEMP/33192694_Sep_14_2021_16_48_27.zip Archiving Patch - 33561310 .... Sucessfully archived patch - 33561310 in /media/sf_TEMP/33561310_Jan_5_2022_08_13_10.zip Archiving Patch - 32218454 .... Sucessfully archived patch - 32218454 in /media/sf_TEMP/32218454_Jan_14_2021_16_24_38.zip Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/19/.patch_storage/33515361_Jan_13_2022_06_14_07 Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/19/.patch_storage/32545013_Apr_16_2021_07_40_05 Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/19/.patch_storage/32904851_Jul_20_2021_09_21_24 Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/19/.patch_storage/33197296_Oct_4_2021_00_44_36 Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/19/.patch_storage/33192793_Oct_15_2021_07_09_52 Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/19/.patch_storage/33192694_Sep_14_2021_16_48_27 Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/19/.patch_storage/33561310_Jan_5_2022_08_13_10 Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/19/.patch_storage/32218454_Jan_14_2021_16_24_38 Patch archive operation completed successfully. OPatch succeeded.
What if …
Of course, with opatch there is always plenty of fun with opatch error messages.
My first attempt failed:
$ ./opatch util archive -patches 32218454,32545013,32904851,33192694,33192793,33197296,33515361,33561310 -archive_dir /home/oracle/patch_storage Oracle Interim Patch Installer version 12.2.0.1.28 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.28 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-01-20_11-48-35AM_1.log Invoking utility "archive" Patch archive operation failed. UtilSession failed to Archive the patches UtilSession failed: Patch archive operation failed. OPatch failed with error code 73
Why did it fail? I needed to check the logfile to find out:
[Jan 20, 2022 11:48:37 AM] [INFO] Archive directory: /home/oracle/patch_storage has required permission to archive the patches
[Jan 20, 2022 11:48:37 AM] [INFO] Space for patch :: /u01/app/oracle/product/19/.patch_storage/32218454_Jan_14_2021_16_24_38 :: 1715352980
[Jan 20, 2022 11:48:37 AM] [INFO] Space for patch :: /u01/app/oracle/product/19/.patch_storage/32545013_Apr_16_2021_07_40_05 :: 1974092970
[Jan 20, 2022 11:48:37 AM] [INFO] Space for patch :: /u01/app/oracle/product/19/.patch_storage/32904851_Jul_20_2021_09_21_24 :: 1927088358
[Jan 20, 2022 11:48:38 AM] [INFO] Space for patch :: /u01/app/oracle/product/19/.patch_storage/33192793_Oct_15_2021_07_09_52 :: 1818503734
[Jan 20, 2022 11:48:38 AM] [INFO] Space for patch :: /u01/app/oracle/product/19/.patch_storage/33192694_Sep_14_2021_16_48_27 :: 444972030
[Jan 20, 2022 11:48:38 AM] [INFO] Space for patch :: /u01/app/oracle/product/19/.patch_storage/33197296_Oct_4_2021_00_44_36 :: 126355388
[Jan 20, 2022 11:48:38 AM] [INFO] Space for patch :: /u01/app/oracle/product/19/.patch_storage/33515361_Jan_13_2022_06_14_07 :: 1750455909
[Jan 20, 2022 11:48:38 AM] [INFO] Space for patch :: /u01/app/oracle/product/19/.patch_storage/33561310_Jan_5_2022_08_13_10 :: 452566990
[Jan 20, 2022 11:48:38 AM] [INFO] Archive directory: /home/oracle/patch_storage needs 10209388359 bytes.
[Jan 20, 2022 11:48:38 AM] [SEVERE] No space to archive given patches in /home/oracle/patch_storage
[Jan 20, 2022 11:48:38 AM] [INFO] Patch archive operation failed.
[Jan 20, 2022 11:48:38 AM] [SEVERE] Patch archive operation failed.
[Jan 20, 2022 11:48:38 AM] [INFO] UtilSession failed to Archive the patches
[Jan 20, 2022 11:48:38 AM] [SEVERE] UtilSession failed to Archive the patches
[Jan 20, 2022 11:48:38 AM] [SEVERE] OUI-67073:UtilSession failed: Patch archive operation failed.
[Jan 20, 2022 11:48:38 AM] [INFO] Finishing UtilSession at Thu Jan 20 11:48:38 CET 2022
[Jan 20, 2022 11:48:38 AM] [INFO] Stack Description: java.lang.RuntimeException: Patch archive operation failed.
at oracle.opatch.opatchutil.OUSession.archive(OUSession.java:3491)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.opatch.UtilSession.process(UtilSession.java:355)
at oracle.opatch.OPatchSession.process(OPatchSession.java:2660)
at oracle.opatch.OPatch.process(OPatch.java:870)
at oracle.opatch.OPatch.main(OPatch.java:927)
Caused by: java.lang.RuntimeException: Patch archive operation failed.
... 9 more
You think this is a bit hard to read? I agree – and I did a quick translation since a size of 190423787 is not easy to consume. I removed a bit of the output too. Now this works much better:
Archive directory: /home/oracle/patch_storage has required permission to archive the patches 32218454_Jan_14_2021_16_24_38 :: 1.715 GB 32545013_Apr_16_2021_07_40_05 :: 1.974 GB 32904851_Jul_20_2021_09_21_24 :: 1.927 GB 33192793_Oct_15_2021_07_09_52 :: 1.818 GB 33192694_Sep_14_2021_16_48_27 :: 0.444 GB 33197296_Oct_4_2021_00_44_36 :: 0.126 GB 32515361_Jan_13_2022_06_14_07 :: 1.750 GB 33561310_Jan_5_2022_08_13_10 :: 0.452 GB Archive directory: /home/oracle/patch_storage needs 10.209 GB. No space to archive given patches in /home/oracle/patch_storage
Hah! Got it. The calculation takes the current sized of each patch directory (here between 444 MB and 1.9 GB, adds it up, calculates something around 10 GB – and says: “Hey Mike, you don’t have enough space on your device for another 10 GB“.
So I put the files on the disk of my host system knowing that I will have at least 80GB of free space. And this was the result once opatch was done with its duties:
688M Jan 20 14:10 33515361_Jan_13_2022_06_14_07.zip 948M Jan 20 14:12 32545013_Apr_16_2021_07_40_05.zip 824M Jan 20 14:13 32904851_Jul_20_2021_09_21_24.zip 119M Jan 20 14:13 33197296_Oct_4_2021_00_44_36.zip 717M Jan 20 14:15 33192793_Oct_15_2021_07_09_52.zip 118M Jan 20 14:15 33192694_Sep_14_2021_16_48_27.zip 120M Jan 20 14:15 33561310_Jan_5_2022_08_13_10.zip 725M Jan 20 14:17 32218454_Jan_14_2021_16_24_38.zip
In sum 4.26 GB used. This is means less than 50% space consumed.
Lesson no.1:
The space usage assumption is dead wrong and uses the current sizes instead at least a assumption.
Lesson no.2:
The opatch util archive command does a simple zip of the subdirectory for each patch.
And this brings me to the magic final question.
Will opatch util archive free up space in .patch_storage?
Since you saw the command output above already, you may not be curious anymore.
It does cleanup. Every patch subdirectory to be archived will be removed afterwards. This is how my .patch_storage directory looked afterwards:
$ ls -lhrt total 160K -rw-r--r--. 1 oracle dba 24K Jan 19 21:39 newdirs.txt drwxr-xr-x. 16 oracle dba 4.0K Jan 19 22:12 NApply -rw-r--r--. 1 oracle dba 64K Jan 19 22:12 record_inventory.txt -rw-r--r--. 1 oracle dba 64K Jan 19 22:12 interim_inventory.txt -rw-r--r--. 1 oracle dba 92 Jan 19 22:12 LatestOPatchSession.properties
All the subdirectories are gone. It accumulates now only 160K.
I think I can deal with this.
What happens if you want to rollback a patch?
I’m happy that I tried this out as well before deleting all the patches on my host environment where I archived them to:
$ ./opatch rollback -id 33516412 Oracle Interim Patch Installer version 12.2.0.1.28 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/21 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/21/oraInst.loc OPatch version : 12.2.0.1.28 OUI version : 12.2.0.9.0 Log file location : /u01/app/oracle/product/21/cfgtoollogs/opatch/opatch2022-01-20_15-38-47PM_1.log Patches will be rolled back in the following order: 33516412 Prerequisite check "CheckRollbackable" failed. Possible causes are: The patch can not be rolled back No read or write permission to ORACLE_HOME/.patch _storage The ORACLE_HOME/.patch_storage file may have been removed. The details are: *************************************************** Following patches are archived: 33516412 Please follow the instructions below to unarchive the patches before proceeding with current Opatch operation (as /u01/app/oracle/product/21 owner). Step 1: /u01/app/oracle/product/21/OPatch/opatch util unarchive -patches 33516412 Step 2 : Resume/Rerun current Opatch operation: /u01/app/oracle/product/21/OPatch/opatch rollback -id 33516412 -invPtrLoc /u01/app/oracle/product/21/oraInst.loc *************************************************** UtilSession failed: Prerequisite check "CheckRollbackable" failed. Log file location: /u01/app/oracle/product/21/cfgtoollogs/opatch/opatch2022-01-20_15-38-47PM_1.log OPatch failed with error code 73
So I would need to “unarchive” the patch before.
Let me try this with: opatch util unarchive -patches 33516412
$ /u01/app/oracle/product/21/OPatch/opatch util unarchive -patches 33516412 Oracle Interim Patch Installer version 12.2.0.1.28 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/21 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/21/oraInst.loc OPatch version : 12.2.0.1.28 OUI version : 12.2.0.9.0 Log file location : /u01/app/oracle/product/21/cfgtoollogs/opatch/opatch2022-01-20_15-40-58PM_1.log Invoking utility "unarchive" Unarchive given patches for given OH Started : unarchiving the patch dirs to OH/.patch_storage User specified list :[33516412] Total Space Needed for Unarchive :1511356677 Unarchive successful :/u01/app/oracle/product/21/.patch_storage/33516412_Jan_13_2022_06_54_26 Unarchive session is complete To clean up archives from custom storage path,Run the following command :'/u01/app/oracle/product/21/OPatch/opatch util cleanuparchives' OPatch succeeded.
And afterwards I could complete the rollback of my bundle patch for this exercise.
Learning no.3:
If you ever EVER plan to rollback a patch bundle, then keep the archives at the EXACT same location where you asked opatch to archive them to. Otherwise, you may not be able to rollback a bundle.
Other findings
Besides the fact that you need to know the command to get the “help” information, this is another pitfall I came across quickly:
Don’t put space characters between your patch numbers.
When you do this:
$ ./opatch util archive -patches 33197448, 33587128 -archive_dir /media/sf_TEMP Oracle Interim Patch Installer version 12.2.0.1.28 Copyright (c) 2022, Oracle Corporation. All rights reserved. Syntax Error... Unrecognized Option for util. Please use the option 'opatch util -help' to get correct syntax OPatch failed with error code 14
you see opatch failing with one of its mystical errors.
The fix is to remove the space between the patch number and the comma, such as:
$ ./opatch util archive -patches 33197448,33587128 -archive_dir /media/sf_TEMP Oracle Interim Patch Installer version 12.2.0.1.28 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.2.0.1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.2.0.1/oraInst.loc OPatch version : 12.2.0.1.28 OUI version : 12.2.0.1.4 Log file location : /u01/app/oracle/product/12.2.0.1/cfgtoollogs/opatch/opatch2022-01-20_15-21-41PM_1.log Invoking utility "archive" Archiving Patch - 33587128 .... Sucessfully archived patch - 33587128 in /media/sf_TEMP/33587128_Dec_27_2021_16_38_16.zip Archiving Patch - 33197448 .... Sucessfully archived patch - 33197448 in /media/sf_TEMP/33197448_Oct_4_2021_05_28_49.zip Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/12.2.0.1/.patch_storage/33587128_Dec_27_2021_16_38_16 Succesfully verified the created archive. Deleting the original patch from - /u01/app/oracle/product/12.2.0.1/.patch_storage/33197448_Oct_4_2021_05_28_49 Patch archive operation completed successfully. OPatch succeeded.
Then it works.
Learning no.4:
Don’t use space characters in the comma-separated patch number list for -patches
Learning no.5:
The new opatch util archive command frees up a lot of space. Unfortunately there is no automatism and still non n-1 solution. But most of you won’t be affected since you patch with new homes, and just flip from one home to another.
In my test I freed up all together about 15GB in my 3 Oracle Homes, 12.2.0.1, 19c and 21c. What I would still want is a n-1 option keeping automatically only the previous patch and wipe out older ones since I rarely will jump back more than one patch bundle, if ever. And in addition I would have expected a “all” switch which clean out every patch without the need to copy and paste individual patch numbers.
Cleaning Up the archives
And finally – thanks to Stephen Anderson who pointed me to this – the question remains:
Do you need to remove the archives now manually from disk?
Let’s see since there is another – more or less undocumented – command for this. You may have seen this already in the output above:
opatch util cleanuparchives
Let me show you how this works:
$ $ORACLE_HOME/OPatch/opatch util cleanuparchives Oracle Interim Patch Installer version 12.2.0.1.28 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.28 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-01-21_12-27-56PM_1.log Invoking utility "cleanuparchives" Started : Util cleauparchives session Patch id :33515361 Patch id :32545013 Patch id :32904851 Patch id :33197296 Patch id :33192793 Patch id :33192694 Patch id :33561310 Patch id :32218454 Nothing to cleanup Cleanuparchives from secondary storage is successful. OPatch succeeded.
Now knowing opatch for quite a while I was wondering about this “Nothing to cleanup” message followed by a “… is successful” output. What has happened?
At first, I check my archive storage – and all the files are still there.
And then as next step, I tried to restore one of the archives which are physically still on disk as I did this before:
$ /u01/app/oracle/product/21/OPatch/opatch util unarchive -patches 33515361 Oracle Interim Patch Installer version 12.2.0.1.28 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.28 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-01-21_12-31-31PM_1.log Invoking utility "unarchive" Unarchive given patches for given OH Started : unarchiving the patch dirs to OH/.patch_storage User specified list :[33515361] Total Space Needed for Unarchive :1750455909 Unarchive successful :/u01/app/oracle/product/19/.patch_storage/33515361_Jan_13_2022_06_14_07 Unarchive session is complete To clean up archives from custom storage path,Run the following command :'/u01/app/oracle/product/19/OPatch/opatch util cleanuparchives' OPatch succeeded.
So basically I can’t tell you what the “cleanuparchives” command does, or if it is supposed to do anything. The opatch logfile didn’t help me any further. And the command’s help tells me:
$ $ORACLE_HOME/OPatch/opatch util cleanuparchives -help
Oracle Interim Patch Installer version 12.2.0.1.28
Copyright (c) 2022, Oracle Corporation. All rights reserved.
DESCRIPTION
This operation helps to cleanup the archived files from custom storage
location.
NOTE: This will not cleanup any other directories/files in the system.
This will only cleanup the archives generated as part of
util archive operation.
SYNTAX
opatch util cleanuparchives
I can just confirm that it is not doing what it is supposed to do.
Further Links and Information
- opatch util cleanup problems
- Patching all my environments with the January 2022 Patch Bundles
- OPatch Download: Patch 6880880
- Cleaning up older patch artifacts – and improve opatch performance
–Mike
Awesome stuff! Also, what about the other command listed in your “unarchive” output: opatch util cleanuparchives
One might assume that it will permanently purge the non-OH storage area a bit like opatch util cleanup?
Hi Stephen,
thanks for the hint – I updated the blog post (and spoiler: The command unfortunately does nothing) 🙂
Cheers,
mike
This is a cool new command. Hope it can be fully documented soon. We’ve been doing cleanup according to Doc 550522.1. Basically, clean $ORACLE_HOME/.patch_storage by removing directories not shown in `opatch lsinventory’. I use a script like this http://yong321.freeshell.org/oranotes/clean_patch_storage.txt
It will be documented – actually I learned afterwards that it hasn’t been made official yet – so basically I was telling a “secret” 😉
But since this does not harm, everybody can try it out.
Thanks,
Mike
Hi Mike,
nice stuff. What about consolidation of old patches on a host with mulitple oracle homes (identical versions) into just one archive location? Does unarchive remove the patch from the archive location and so make it unavailable for the other homes?
Hi Robert,
since the feature isn’t officially available (yet), I can’t tell you about the effect other homes.
But as far as I see the files are kept – please scroll down in my blogpost: “At first, I check my archive storage – and all the files are still there.”
Cheers,
Mike
Hello, Mike !
The real name of this article should be “How to move this problem to another place” 😉
It is really practically popular question 🙂 !
Need to ask: do we really need n-1 and older quarterly patchsets in our OH to be able to apply future patchests?
If i install base release and every quarterly patchset. For examle, i install 19.3 then 19.11 then 19.12 then 19.13 and so on.
So, i definitely know that:
1) i need the latest patchset only.
2) i need to be able to install all the future patchsets: 19.14, 19.15 and so on.
Now many patchets i need to keep in OH ?
I once deleted a hidden directory and was unable to apply the next patchset.
Thank you!
Hi Yury,
I more or less agree with you – but I talked to the team and they confirmed that my (and your) n-1 idea is very complex to be realised. Hence, I doubt that we’ll get such a convenient solution the sooner or later.
Cheers,
Mike
Yeah, slow patching. It took 25 minutes to archive 19 patches. I’m assuming with the file naming structure, I will have to create a unique subdirectory for each ORACLE_HOME for which I archive patches? FYI, disk space usage went from 19G to 2.1G in the .patch_storage directory and the archive directory with the zip files is 6.7G. So, it saved just over 50% of the space.
Same here, Lance – I saved a bit more than 50%. And a central storage may be a good idea.
But let me learn over time about the usage patterns in reality.
Cheers,
Mike
Hi Mike,
thanks a lot for the post.
While I’m preparing for the patching, I thought to compile a command to get the inactive patches as a list, and here is what I come up with so far:
$ORACLE_HOME/OPatch/opatch lsinv -inactive | awk ‘/^Patch / {if (x==””) {x=$2} else {x=x”,”$2}} END {print x}’
Which gives me “32904851,31668882,31743771,31902534,31772784,31771877,29585399,29517242” in my environment.
opatch util archive -patches $(…) … should work.
I haven’t tested it though & hope Oracle will provide some built-in option for doing that directly via opatch arguments in the future.
Thanks!
Thank you, Timur!!
Cheers,
Mike
Hi Mike,
Unfortunately this approach works only once. Second time you run archival for inactive patches:
$ORACLE_HOME/OPatch/opatch util archive -patches \
$($ORACLE_HOME/OPatch/opatch lsinv -inactive | awk ‘/^Patch / && /applied/ {if (x==””) {x=$2} else {x=x”,”$2}} END {print x}’) \
-archive_dir $ARCHIVE_DIR
The opatch fails with an error like
UtilSession failed to Archive the patches
UtilSession failed: Following patches are invalid/not available in the .patch_storage dir : […]
Those patches that opatch is complaining about are already archived. Any chance to ignore this error?
Thanks!
OK a bit more hacking and the error is avoided:
[sourcecode]
$ORACLE_HOME/OPatch/opatch util archive -patches \
$({ $ORACLE_HOME/OPatch/opatch lsinv -inactive | awk ‘/^Patch / && /applied/ {print $2}’ & ls -1 $ARCHIVE_DIR | cut -d_ -f1 ; } |
sort | uniq -c | awk ‘$1==”1″ {if (x==””) {x=$2} else {x=x”,”$2}} END {print x}’) \
-archive_dir $ARCHIVE_DIR
[/sourcecode]
Hi Timur,
thanks – and sorry for the late reply.
I will give this a try and refer to you if this works.
And thanks again for the hacking and the hint.
Cheers
Mike
It bothers me that you can not change the Archive location after the fact, and if I try to unarchive from a new location, the system will require me to move the archived patches into the old archive location.
Is not a major issue, but I expected to be able to just indicate the “new Archive location” where the patches compressed in case I need to move them around.
Still, did a test and saved 50% space.. which is a massive difference, thanks for sharing this awesome feature!!
Thanks Victor – and I hope this feature will evolve over time.
Thanks,
Mike
I ran into the problem of opatch taking more and more time for any simple lsinv / apply / rollback operations in 19c and this is directly related to the number of inactive RU patches available in the home. More the inactive RUs and longer the patching activities. Came up with the out-of-the box solution and it is documented at https://www.linkedin.com/pulse/why-my-oracle-19c-db-patching-slow-balaji-govindu/
Thanks Balaji – you will see further blog posts from my side as well.
Thanks,
Mike
Hi Mike,
Thanks for your post. I am currently struggling with a 19c DB System in OCI that is regularly patched with the quarterly RU using the patch “automation” in OCI DB System. Now it takes 5-6 hours to patch a 19c DB System (DB System+RU). This is a bit hard to explain to our customers – that we need a maintenance window of 6 hours to due a quarterly RU patch that normally can be done within 1-2 hours. Oracle Support states they are working on another solution for patching DB Systems………………..
The problem definitely relates to poor performance of opatch operations due to many RU updated over the past months/years.
For me opatch should be able to handle this automatically without any manually archive commands to lower the amount of work done by opatch to do lsinv/apply/rollback. There is clearly something wrong with the implementation of opatch if a simple one-off patch takes 30-60 minutes to apply.
Do you know whether Oracle has any plans on fixing the core problem instead of symptom treatment with manual archive commands that makes the patch process even more complex?
Thanks.
Regards,
Brian
Hi Brian,
please watch out for blog posts coming soon.
There seems to be an issue with too many entries in the inventory.
Do you have an SR you can please share with me?
I would like to use this to increase the pressure internally since many customers seem to struggle with this.
Cheers,
Mike
regarding “cleanuparchives”
short description : after unarchive already archived patch, cleanuparchives will delete the zip
snip:
Deleting :/u01/sw/opatch_archive/29517242_Apr_17_2019_23_27_10.zipDeletion is Success and updating the XML object to remove entry:29517242
Patch id :30869156
Removed the entry of :29517242 from mappingXML file
Updated Central mapping file. Cleanuparchives operation is completed
—
raw log :
[
oracle@vm19rac02 dbhome_1]$ ls -la .patch_storagetotal 224
drwx------ 5 oracle oinstall 4096 Apr 13 15:09 .
drwxrwxr-x 72 oracle oinstall 4096 Mar 10 17:25 ..
drwxr-xr-x 4 oracle oinstall 87 Mar 10 17:26 33515361_Jan_13_2022_06_14_07
drwxr-xr-x 4 oracle oinstall 87 Mar 10 17:27 33529556_Jan_9_2022_21_15_36
-rw-r--r-- 1 oracle oinstall 97871 Mar 10 17:27 interim_inventory.txt
-rw-r--r-- 1 oracle oinstall 92 Mar 10 17:27 LatestOPatchSession.properties
drwxr-xr-x 14 oracle oinstall 4096 Mar 10 17:27 NApply
-rw-r--r-- 1 oracle oinstall 24221 Mar 10 17:27 newdirs.txt
-rw-r--r-- 1 oracle oinstall 1653 Apr 13 15:09 .patch_archive_mapping.xml
-rw-r--r-- 1 oracle oinstall 85550 Mar 10 17:27 record_inventory.txt
[oracle@vm19rac02 dbhome_1]$ du -sh .patch_storage
2.4G .patch_storage[oracle@vm19rac02 dbhome_1]$ vi /u01/sw/opatch_archive/raw_term_log.txt
[oracle@vm19rac02 dbhome_1]$ opatch util unarchive -patches 29517242
Oracle Interim Patch Installer version 12.2.0.1.29
Copyright (c) 2022, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/19.0.0/dbhome_1Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.29
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatch/opatch2022-04-13_15-39-08PM_1.log
Invoking utility "unarchive"
Unarchive given patches for given OH
Started : unarchiving the patch dirs to OH/.patch_storage
User specified list :[29517242]
Total Space Needed for Unarchive :244391
Unarchive successful :/u01/app/oracle/product/19.0.0/dbhome_1/.patch_storage/29517242_Apr_17_2019_23_27_10
Unarchive session is complete
To clean up archives from custom storage path,Run the following command :'/u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatch util cleanuparchives'
OPatch succeeded.
[oracle@vm19rac02 dbhome_1]$ ls -l .patch_storage
total 212
drwxr-xr-x 3 oracle oinstall 74 Apr 13 15:39 29517242_Apr_17_2019_23_27_10
drwxr-xr-x 4 oracle oinstall 87 Mar 10 17:26 33515361_Jan_13_2022_06_14_07
drwxr-xr-x 4 oracle oinstall 87 Mar 10 17:27 33529556_Jan_9_2022_21_15_36
-rw-r--r-- 1 oracle oinstall 97871 Mar 10 17:27 interim_inventory.txt
-rw-r--r-- 1 oracle oinstall 92 Mar 10 17:27 LatestOPatchSession.properties
drwxr-xr-x 14 oracle oinstall 4096 Mar 10 17:27 NApply
-rw-r--r-- 1 oracle oinstall 24221 Mar 10 17:27 newdirs.txt
-rw-r--r-- 1 oracle oinstall 85550 Mar 10 17:27 record_inventory.txt
[oracle@vm19rac02 dbhome_1]$ ls -l /u01/sw/opatch_archive/
total 567768
-rw-r--r-- 1 oracle oinstall 47382 Apr 13 15:08 29517242_Apr_17_2019_23_27_10.zip
-rw-r--r-- 1 oracle oinstall 23249 Apr 13 15:08 29585399_Apr_9_2019_19_12_47.zip
-rw-r--r-- 1 oracle oinstall 485071768 Apr 13 15:09 30869156_Apr_6_2020_23_20_53.zip
-rw-r--r-- 1 oracle oinstall 96193925 Apr 13 15:08 30894985_Apr_10_2020_05_35_01.zip
-rw-r--r-- 1 oracle oinstall 52874 Apr 13 15:28 raw_term_log.txt
[oracle@vm19rac02 dbhome_1]$ opatch util cleanuparchives
Oracle Interim Patch Installer version 12.2.0.1.29
Copyright (c) 2022, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/19.0.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.29
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatch/opatch2022-04-13_15-39-55PM_1.log
Invoking utility "cleanuparchives"
Started : Util cleauparchives session
Patch id :30894985
Patch id :29585399
Patch id :29517242
Deleting :/u01/sw/opatch_archive/29517242_Apr_17_2019_23_27_10.zip
Deletion is Success and updating the XML object to remove entry:29517242
Patch id :30869156
Removed the entry of :29517242 from mappingXML file
Updated Central mapping file. Cleanuparchives operation is completed
Cleanuparchives from secondary storage is successful.
OPatch succeeded.
[oracle@vm19rac02 dbhome_1]$ ls -l /u01/sw/opatch_archive/
total 567720
-rw-r--r-- 1 oracle oinstall 23249 Apr 13 15:08 29585399_Apr_9_2019_19_12_47.zip
-rw-r--r-- 1 oracle oinstall 485071768 Apr 13 15:09 30869156_Apr_6_2020_23_20_53.zip
-rw-r--r-- 1 oracle oinstall 96193925 Apr 13 15:08 30894985_Apr_10_2020_05_35_01.zip
-rw-r--r-- 1 oracle oinstall 52874 Apr 13 15:28 raw_term_log.txt
[oracle@vm19rac02 dbhome_1]$
Thanks a lot!
Cheers,
Mike
Hello Mike!
Thank you for very interesting finding!
Do you know if same commands can be used to clean up GI Oracle Home?
Regards,
Dmytro
I think so, Dmytro.
Cheers
Mike
Yes it does. Actually I didn’t even think that it wouldn’t and while I’m waiting for archiving inactive patches to finish I came across this last comment…
Cheers
Holger
Hi,
great article.
I am using Oracle Interim Patch Installer version 13.9.4.2.2
and archive option is not available;
>opatch util archive -help
Oracle Interim Patch Installer version 13.9.4.2.2
Copyright (c) 2022, Oracle Corporation. All rights reserved.
Syntax Error… Unrecognized util command.
There must be some reason that oracle decided to remove that feature
Hi David,
which Oracle version is this? My opatch gives a very different splash screen:
Oracle Interim Patch Installer version 12.2.0.1.33Copyright (c) 2022, Oracle Corporation. All rights reserved.
Cheers
Mike
After running through this process, the same patches are still listed as inactive. Is there a way I can tell what patches have been archived outside of looking at the zip files in the archive directory?
Also, if I have multiple Oracle Homes, all created from the same Gold image, could archive them all but only keep one copy of the zip files and delete the rest, knowing if I need to unarchive them I would have to put them into the correctly named archive directory?
Hi Lance,
I work on finding out about the most recent cleanup options, and how to deal with the result.
Cheers,
Mike
good afternoon mike,
Great blog about cleaning up
THANKS!!
the version dawid mentions seems to be a version used with Weblogic.
my version in weblogic is now:
OPatch Version: 13.9.4.2.10
for my unix oracle servers with 19c databases the version is
OPatch Version: 12.2.0.1.34
in both versions i cannot find util archive
in both version i can find
./opatch util cleanup -help
is the cleanup the same as what you describe in your blog as:
./opatch util archive -patches 123456,1122334 -archive_dir /dirx
cheers
Kees
Hi Kees,
I have the 33 version of opatch on disk right now, and it delivers everything fine when I for instance check “./opatch util archive -help”
Can you please check what this command gives in your case?
Thanks
Mike
good morning mike; this is the output:
./opatch util archive -help
Oracle Interim Patch Installer version 12.2.0.1.34
Copyright (c) 2022, Oracle Corporation. All rights reserved.
DESCRIPTION
This operation helps to archive the patches stored under
ORACLE_HOME/.patch_storage to custom storage location.
NOTE: Patch directories in .patch_storage will be used during
rollback operation.
SYNTAX
opatch util archive [-patches ]
[-archive_dir ]
OPTIONS
-patches
A comma separated list of patches to be given for
archive.
-archive_dir
Specify custom storage path to store the patches
which are going to be archived.
NOTE: All options are mandatory.
RECOMMENDATION :
To archive patches, opatch recommends the following
1. Archiving inactive patches :
Run the following command to get inactive patches : “OPatch/opatch lsinv -inactive”
Example
opatch util archive -patches 123456,1122334 -archive_dir /dir1/dir2/
OPatch succeeded.
Hi Kees,
than it should work fine. If it doesn’t then you please need to alert Oracle Support about it. The existence of the help should mean that the feature is present as well.
Thanks
Mike
Funny how I always come back to your posts.
I just read https://dohdatabase.com/2023/05/08/how-to-clone-oracle-home-without-using-clone-pl/ and at the end I stumbled over the mention of a command “opatch util deleteinactivepatches” which I never heard of. While googling, the first hit was this page (even though you didn’t mention that command since it was introduced last month).
That said:
The new OPatch command is documented a little more here: https://support.oracle.com/epmos/faces/DocContentDisplay?id=2942102.1
I think, this is a wonderful addition to what you mentioned in this blog post, even if you don’t do an updated post. 🙂
Coming this week, Jan.
Just had no time to play with it. But we have even a MOS note out already this time:
Oracle Support Document 2942102.1 (OPatch 12.2.0.1.37+ Introduces a New Feature to Delete Inactive Patches in the ORACLE_HOME/.patch_storage Directory ) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=2942102.1
Cheers
Mike