Happy New Year to all of you at first. I hope you all had a good start – and for myself, I promise to blog a bit more again in 2024. Let me start with my first post since I received a question very often: Why should I patch out-of-place with a new Oracle Home? Since I need to patch a bit today I can easily show the downsides of in-place patching – and a patching lab.
It all started at OCW 2023
Ok, we didn’t really start it at OCW 2023 in Las Vegas. But we’ve had several patching sessions. And we launched our new lab: Patch Me If You Can. I will come back to the lab later. We published pictures with customers and speakers, and we told you to patch always Out-Of-Place with a new Oracle Home. Btw, this will become the default in Oracle Database 23c.
Still, people come back to us (which is good), and despite our stark statement “Don’t argue with us”, there are questions here and there, either asked directly or coming up in customer calls and even escalations. One very common complaint is that we promoted the Data Pump Bundle Patch a lot, but then it needs to be deinstalled before the next RU can be applied. To be honest, I heard some very strong wording about this fact.
This brings me right to the downsides of in-place patching.
The downsides of In-Place Patching
In-Place Patching means that you have one single Oracle Home, and you apply the next Release Update into it, and then the next one, and so on. The unfunny part starts when you add one-off patches, merge patches and patch bundles to this home. While the Release Updates are created to be merged into each other, the one-offs and others are built specifically on-top of a given RU.
This means: You need to remove all non-RUs from the Oracle Home before you can apply the next RU to your existing home. Hence, you have a lot more work to do. Now, some customers complain about this. But that is the concept of one-offs, merges and patch bundles (e.g., the Data Pump Bundle Patch).
And this brings me to one of the most annoying downsides of In-Place Patching: The downtime. At this point, not only you will have way more work. But your Oracle Home is not available for use while you do all these actions. You need to stop all database instances using this particular home, and here you just need to remove patches before you can apply the next RU. The downtime will be significant.
And even worse, if something goes wrong, you will end up building a new home from scratch anyways. Which is no fun when this happens on your prod environment.
Finally, you carry a lot of “memories” with your home. This tackles also the question, why you better don’t clone your home but instead start with a fresh install. Even when you clone your existing home into a separate one, you clone all the home’s history. We’ve seen in the past that this is not ideal.
Hence, our strong recommendation is always to patch Out-Of-Place with a brand new Oracle Home.
Today is a nice, very cold Wednesday morning. And I downloaded some patch bundles including the Release Update 19.21.0 but also a one-off I would like to try out. It has been created on-top of 19.21.0. And I will try the “forbidden” route, and patch in-place since this is my Hands-On Lab environment, and nobody else will be affected when something goes deeply wrong.
My starting point is:
$ $ORACLE_HOME/OPatch/opatch lspatches 35246710;HIGH DIRECT PATH READ AFTER 19.18 DBRU PATCHING 35213579;MERGE ON DATABASE RU 22.214.171.124.0 OF 35037877 35046819 35162446;NEED BEHAVIOR CHANGE TO BE SWITCHED OFF 35160800;GG IE FAILS WITH ORA-14400 AT SYSTEM.LOGMNRC_USER AFTER ORACLE DB UPGRADE TO 19.18DBRU 35156936;ORA-7445 [KFFBNEW()+351] AFTER CONVERT TO ASM FLEX DISKGROUP 34974052;DIRECT NFS CONNECTION RESET MESSAGES 34879016;ALL SESSIONS HANG DUE TO INST_RCV BUFFER IS NOT GETTING WRITE PERMISSION 34871935;SBI QUEUE BUILDUP - SESSIONS SPIKE WITH GC CURRENT REQUEST (6-DEC-2022) 34861493;RESYNC CATALOG FAILED IN ZDLRA CATALOG AFTER PROTECTED DATABASE PATCHED TO 19.17 34810252;SPIN OFF FOR BUG 34808861 [ORA-00600 INTERNAL ERROR CODE, ARGUMENTS [KFDS_GETSEGREUSEENQ01] TERMINATED ALL DB INSTANCES 34793099;STRESS FA CDB CREATION FAILS ON 19.17 WITH THE ORA-00704 BOOTSTRAP PROCESS FAILURE WHILE OPENING PDB$SEED 34783802;PARALLEL QUERY ON PARTITIONED TABLE RETURNS WRONG RESULT 34557500;CTWR CAUSED MULTIPLE INSTANCES IN HUNG STATE ON THE RAC STANDBY DATABASE 34340632;AQAH SMART MONITORING & RESILIENCY IN QUEUE KGL MEMORY USAGE 33973908;DBWR NOT PICKING UP WRITES FOR SOME TIME 32727143;TRANSACTION-LEVEL CONTENT ISOLATION FOR TRANSACTION-DURATION GLOBAL TEMPORARY TABLES 31222103;STRESS RAC ATPD FAN EVENTS ARE NOT GETTING PROCESSED WITH 21C GI AND 19.4 DB 34972375;DATAPUMP BUNDLE PATCH 126.96.36.199.0 34786990;OJVM RELEASE UPDATE: 188.8.131.52.230117 (34786990) 34765931;Database Release Update : 184.108.40.206.230117 (34765931) 29585399;OCW RELEASE UPDATE 220.127.116.11.0 (29585399) OPatch succeeded.
It is basically the 19.18.0 RU with the OJVM, the Data Pump Bundle Patch, and I think that I applied an MRP as well. MRPs are very hard if not impossible to identify from the inventory as you can see above.
What happens when I try to apply the 19.21.0 RU to this home?
You know it already, opatch gives me a nice complaint and refuses to continue:
- Please rollback the relevant overlay patches of the subset patches which are conflicting and apply the superset patch Log file location: /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2024-01-10_08-40-16AM_1.log OPatch failed with error code 73
Makes sense – and now my Wednesday morning will be less exiting. Before I can apply 19.21.0 to this home, I need to remove lots of stuff from it beforehand.
But as usual, the log is not as clear as I would wish it to be. It tells me:
[Jan 10, 2024 8:41:09 AM] [WARNING] OUI-67301: Following patches have conflicts: [ 34972375 35643107 ]
And while the first one listed is the Data Pump Bundle Patch, the second one has no entry in the inventory. Well, a quick MOS check tells me that 35643107 is the RU I want to apply, 19.21.0. So, it makes sense that it is not in my inventory. It just dawns me that this is not everything I need to remove. I can keep the OJVM in theory but I want to refresh it as well to 19.21, and not mix a 19.21 RU with a 19.18.0 OJVM.
The annoying removal exercise
Let me do a shortcut here – this is what I need to remove:
$ORACLE_HOME/OPatch/opatch nrollback -id 34879016,34871935,34810252,34793099,34783802,33973908
How do I know? Well, I’ve done this before – and I realized that the opatch 39 version I updated this morning at first gives me java stack trace errors in the log. Ouch …
Big downside again: This removal will take time.
Well, and never trust strangers. Let me do a conflict check, and you will realize that there is more removal to happen:
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./
leads to another conflict I have to resolve. More removal is required:
$ORACLE_HOME/OPatch/opatch nrollback -id 34340632,34972375,35213579
Believe me, there are much nicer things to do on a Wednesday morning. And digging through hard-to-read opatch logs with ten-thousand of fixes listed, messages embedded after hundreds of lines is not my favorite hobby.
Another conflict check tells me that I am good to go.
How long did it take?
Even if I remove the time it took me to write this up for your reading pleasure, the entire removal exercise has taken more than 30 minutes since I needed to run the removal in two stages, go through the logs, wait for the conflict check to complete again etc.
My database instance(s) would be all down while I did that.
To be very honest, this is not smart – and it is highly error-prone. Of course, we all never mistype – but it requires a high amount of concentration to go through this. And at least to me, it is highly frustrating.
It is so much nicer to have a brand new home provided, and then just stop my database (instance), and relocate it to the new home before I call datapatch, isn’t it?
I will save you the time to read through my patch apply. But I need to direct you to our new lab, especially those of you who are not convinced yet.
Patch Me If You Can
We launched the Patch Me If You Can lab at Oracle Cloud World 2023 in Las Vegas. And we created it with three goals in our mind. At first, we wanted to give everybody a playground environment where you can exercise database patching without downloading all the software. Then we wanted to share some cool techniques to quickly – and fully scripted and automated – deploy a new and patched home. And finally, for those you still believe in in-place patching, we wanted to show you how complex it is when you have several additional fixes in your current home.
Go for it – try it out.
It is a Green Button Lab, meaning, when you provision it via the Green Button, you need to wait 10-12 minutes, and then the environment is ready to go. You will connect via a noVNC connection within your browser, nothing needs to be installed or downloaded. And if your company firewall blocks the standard ports we are using, then tether over your cell or use your private internet connection from home instead.
This is what you can do – and it is all scripted and explained, with logs and everything you need to know and try.
If you are not convinced after completing the lab, we failed.
Some side notes on in-place patching
Well, things are seldomly black and white. There is some grey somewhere sometimes. And I know, there are some applications and setups which don’t support out-of-place patching, e.g. Oracle EBusiness Suite to name one. We see this – and this will change with Oracle Database 23c for sure.
Further Links and Information