Happy New Year to all of you at first. I hope you all had a good start – and for myself, I promise to blog a bit more again in 2024. Let me start with my first post since I received a question very often: Why should I patch out-of-place with a new Oracle Home? Since I need to patch a bit today I can easily show the downsides of in-place patching – and a patching lab.
Photo by Cederic Vandenberghe on Unsplash
It all started at OCW 2023
Ok, we didn’t really start it at OCW 2023 in Las Vegas. But we’ve had several patching sessions. And we launched our new lab: Patch Me If You Can. I will come back to the lab later. We published pictures with customers and speakers, and we told you to patch always Out-Of-Place with a new Oracle Home. Btw, this will become the default in Oracle Database 23c.
Still, people come back to us (which is good), and despite our stark statement “Don’t argue with us”, there are questions here and there, either asked directly or coming up in customer calls and even escalations. One very common complaint is that we promoted the Data Pump Bundle Patch a lot, but then it needs to be deinstalled before the next RU can be applied. To be honest, I heard some very strong wording about this fact.
This brings me right to the downsides of in-place patching.
The downsides of In-Place Patching
In-Place Patching means that you have one single Oracle Home, and you apply the next Release Update into it, and then the next one, and so on. The unfunny part starts when you add one-off patches, merge patches and patch bundles to this home. While the Release Updates are created to be merged into each other, the one-offs and others are built specifically on-top of a given RU.
This means: You need to remove all non-RUs from the Oracle Home before you can apply the next RU to your existing home. Hence, you have a lot more work to do. Now, some customers complain about this. But that is the concept of one-offs, merges and patch bundles (e.g., the Data Pump Bundle Patch).
And this brings me to one of the most annoying downsides of In-Place Patching: The downtime. At this point, not only you will have way more work. But your Oracle Home is not available for use while you do all these actions. You need to stop all database instances using this particular home, and here you just need to remove patches before you can apply the next RU. The downtime will be significant.
And even worse, if something goes wrong, you will end up building a new home from scratch anyways. Which is no fun when this happens on your prod environment.
Finally, you carry a lot of “memories” with your home. This tackles also the question, why you better don’t clone your home but instead start with a fresh install. Even when you clone your existing home into a separate one, you clone all the home’s history. We’ve seen in the past that this is not ideal.
Hence, our strong recommendation is always to patch Out-Of-Place with a brand new Oracle Home.
OPatch complains
Today is a nice, very cold Wednesday morning. And I downloaded some patch bundles including the Release Update 19.21.0 but also a one-off I would like to try out. It has been created on-top of 19.21.0. And I will try the “forbidden” route, and patch in-place since this is my Hands-On Lab environment, and nobody else will be affected when something goes deeply wrong.
My starting point is:
$ $ORACLE_HOME/OPatch/opatch lspatches 35246710;HIGH DIRECT PATH READ AFTER 19.18 DBRU PATCHING 35213579;MERGE ON DATABASE RU 19.18.0.0.0 OF 35037877 35046819 35162446;NEED BEHAVIOR CHANGE TO BE SWITCHED OFF 35160800;GG IE FAILS WITH ORA-14400 AT SYSTEM.LOGMNRC_USER AFTER ORACLE DB UPGRADE TO 19.18DBRU 35156936;ORA-7445 [KFFBNEW()+351] AFTER CONVERT TO ASM FLEX DISKGROUP 34974052;DIRECT NFS CONNECTION RESET MESSAGES 34879016;ALL SESSIONS HANG DUE TO INST_RCV BUFFER IS NOT GETTING WRITE PERMISSION 34871935;SBI QUEUE BUILDUP - SESSIONS SPIKE WITH GC CURRENT REQUEST (6-DEC-2022) 34861493;RESYNC CATALOG FAILED IN ZDLRA CATALOG AFTER PROTECTED DATABASE PATCHED TO 19.17 34810252;SPIN OFF FOR BUG 34808861 [ORA-00600 INTERNAL ERROR CODE, ARGUMENTS [KFDS_GETSEGREUSEENQ01] TERMINATED ALL DB INSTANCES 34793099;STRESS FA CDB CREATION FAILS ON 19.17 WITH THE ORA-00704 BOOTSTRAP PROCESS FAILURE WHILE OPENING PDB$SEED 34783802;PARALLEL QUERY ON PARTITIONED TABLE RETURNS WRONG RESULT 34557500;CTWR CAUSED MULTIPLE INSTANCES IN HUNG STATE ON THE RAC STANDBY DATABASE 34340632;AQAH SMART MONITORING & RESILIENCY IN QUEUE KGL MEMORY USAGE 33973908;DBWR NOT PICKING UP WRITES FOR SOME TIME 32727143;TRANSACTION-LEVEL CONTENT ISOLATION FOR TRANSACTION-DURATION GLOBAL TEMPORARY TABLES 31222103;STRESS RAC ATPD FAN EVENTS ARE NOT GETTING PROCESSED WITH 21C GI AND 19.4 DB 34972375;DATAPUMP BUNDLE PATCH 19.18.0.0.0 34786990;OJVM RELEASE UPDATE: 19.18.0.0.230117 (34786990) 34765931;Database Release Update : 19.18.0.0.230117 (34765931) 29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399) OPatch succeeded.
It is basically the 19.18.0 RU with the OJVM, the Data Pump Bundle Patch, and I think that I applied an MRP as well. MRPs are very hard if not impossible to identify from the inventory as you can see above.
What happens when I try to apply the 19.21.0 RU to this home?
You know it already, opatch gives me a nice complaint and refuses to continue:
- Please rollback the relevant overlay patches of the subset patches which are conflicting and apply the superset patch Log file location: /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2024-01-10_08-40-16AM_1.log OPatch failed with error code 73
Makes sense – and now my Wednesday morning will be less exiting. Before I can apply 19.21.0 to this home, I need to remove lots of stuff from it beforehand.
But as usual, the log is not as clear as I would wish it to be. It tells me:
[Jan 10, 2024 8:41:09 AM] [WARNING] OUI-67301:
Following patches have conflicts: [ 34972375 35643107 ]
And while the first one listed is the Data Pump Bundle Patch, the second one has no entry in the inventory. Well, a quick MOS check tells me that 35643107 is the RU I want to apply, 19.21.0. So, it makes sense that it is not in my inventory. It just dawns me that this is not everything I need to remove. I can keep the OJVM in theory but I want to refresh it as well to 19.21, and not mix a 19.21 RU with a 19.18.0 OJVM.
The annoying removal exercise
Let me do a shortcut here – this is what I need to remove:
$ORACLE_HOME/OPatch/opatch nrollback -id 34879016,34871935,34810252,34793099,34783802,33973908
How do I know? Well, I’ve done this before – and I realized that the opatch 39 version I updated this morning at first gives me java stack trace errors in the log. Ouch …
Big downside again: This removal will take time.
Well, and never trust strangers. Let me do a conflict check, and you will realize that there is more removal to happen:
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./
leads to another conflict I have to resolve. More removal is required:
$ORACLE_HOME/OPatch/opatch nrollback -id 34340632,34972375,35213579
Believe me, there are much nicer things to do on a Wednesday morning. And digging through hard-to-read opatch logs with ten-thousand of fixes listed, messages embedded after hundreds of lines is not my favorite hobby.
Another conflict check tells me that I am good to go.
How long did it take?
Even if I remove the time it took me to write this up for your reading pleasure, the entire removal exercise has taken more than 30 minutes since I needed to run the removal in two stages, go through the logs, wait for the conflict check to complete again etc.
My database instance(s) would be all down while I did that.
To be very honest, this is not smart – and it is highly error-prone. Of course, we all never mistype – but it requires a high amount of concentration to go through this. And at least to me, it is highly frustrating.
It is so much nicer to have a brand new home provided, and then just stop my database (instance), and relocate it to the new home before I call datapatch, isn’t it?
I will save you the time to read through my patch apply. But I need to direct you to our new lab, especially those of you who are not convinced yet.
Patch Me If You Can
We launched the Patch Me If You Can lab at Oracle Cloud World 2023 in Las Vegas. And we created it with three goals in our mind. At first, we wanted to give everybody a playground environment where you can exercise database patching without downloading all the software. Then we wanted to share some cool techniques to quickly – and fully scripted and automated – deploy a new and patched home. And finally, for those you still believe in in-place patching, we wanted to show you how complex it is when you have several additional fixes in your current home.
Go for it – try it out.
It is a Green Button Lab, meaning, when you provision it via the Green Button, you need to wait 10-12 minutes, and then the environment is ready to go. You will connect via a noVNC connection within your browser, nothing needs to be installed or downloaded. And if your company firewall blocks the standard ports we are using, then tether over your cell or use your private internet connection from home instead.
This is what you can do – and it is all scripted and explained, with logs and everything you need to know and try.
LiveLabs: Patch Me If You Can Hands-On Lab
If you are not convinced after completing the lab, we failed.
Some side notes on in-place patching
Well, things are seldomly black and white. There is some grey somewhere sometimes. And I know, there are some applications and setups which don’t support out-of-place patching, e.g. Oracle EBusiness Suite to name one. We see this – and this will change with Oracle Database 23c for sure.
Further Links and Information
–Mike
I’ve been meaning to comment on in-place / out of-place patching with how we do it. Maybe it’ll help some and maybe you’ll help me find even smarter ways.
Out strategy is a combination of patching in-place, but in an “out of place” way. We patch “in-place” for many reasons, some of which are that we use Enterprise Manager (not to patch, not using fleet maintenance) but for managing and monitoring databases, and database code trees.
I don’t want to need to discover multiple oracle_homes. I want to rely on having the same home name, the same home path, consistently, exclusively, throughout my entire enterprise.
That enterprise currently consists of 227 Oracle 19c server code trees and 344 19c database instances. 220 databases have been patched to 19.21.
Our code trees consist of RU, with the OJVM, the Data Pump Bundle Patch, (no MRPs). It also includes several one-offs that our vendors require, or that we find applicable to our environment (by reviewing MOS 555.1).
1. We begin each new patching cycle by uploading our current opatch inventory to MOS (Analyze with OPatch). MOS 1091294.1 has more info here.
1a – this identifies the new versions of one off patches that we need to request. We request those and wait. Once we get what we need…
2. We begin by installing 19c base release, the RU we want, OJVM, Data Pump Bundle, and all one offs and updated RU specific one offs. This happens on our “code tree server”.
2a. – once this code tree is good to go, we bundle it up ourselves (zip it). This first process sometimes takes 2-6 weeks depending on how many one off updates we need to request from Oracle and how busy we are (only 6 dbas here managing all those databases).
We now have our specific “gold image”.
3. We write our own EM jobs (OS Command jobs) consisting of bash scripts.
3a. – the first one performs “pre-patching” actions. Our servers do have enough free space to hold 2 oracle_homes. This “stages” that oracle home. It backs up the existing home and it extracts the new home to a temporary location. No downtime and can be done at leisure and can be done en mass across the entire enterprise.
Downtime starts.
3b. – patching. Another bash script performs: shutdown everything, detaches the existing home from inventory, deletes that home (should mention that we use read only oracle homes to keep the networking and database init.ora/spfiles under ORACLE_BASE). Moves the new home into the old path. Runs clone.pl (to attach it to inventory and relink everything), runs root.sh, enabled read only oracle home. This takes about 5-10 minutes. That’s right, we just accomplished “patching our existing ORACLE_HOME location ” in 10 minutes or less.
3c. – post-patching. Another bash script performs: starts all databases normal and opens all pdbs restricted, runs datapatch, runs catcon, and restarts everything normal. We do this in parallel so all databases on a server are performing these actions at the same time. This now takes about 20 minutes when datapatch is 100% successful.
Downtime ends. 30 minutes typically – we schedule 60 minute windows allowing a human to review log files.
In our minds, we are getting the best of both worlds. In-place patching = no changes to home names, file system paths, /etc/oratab modifications, no new EM target discovery. Out-of-place saftey and speed.
It’s not simple, but it works really well for us.
TR
Thanks a lot, Todd!
This is great write-up, and I hope it stays readable in the comment section on the blog as well.
Cheers,
Mike
Thank you, Todd!
You developed the patching approach of my dream.
I was not sure about all the steps requied on the 3b stage (detaching the old home and delete it, then clone.pl for the new OH on the same path).
Can you ouline the rollback plan for your approach?
It’s a process that evolves and gets better every iteration. Backout plan would be to unzip the old home and put it back – detach new – reattach old – clone.pl – datapatch – catcon and we’re back to 19.19 (eg).
We’ve never had to do it though.
This sounds perfect, you should write a blog about this. The devil is in steps 3b and 3c
Changing ORACLE_HOME is no fun either. It affects all your scripts, cron jobs, 3rd party backup utilities, etc. that depend on your OH.
Well, but your scripts and cron jobs should use env variables either – and they should point to the current home.
At least, this is what I do in my envs when I change. It is one env file, and it handles everything.
For external tools, I know that some struggle with this.
And I don’t have to look that far – we by ourselves have a list of products who can’t deal with OOP patching, just to name Oracle EBS as one example.
Cheers,
Mike
Hi Mike,
Re the EBS being unable to handle OOP, do you have details of what the issue actually is?
I myself have done it and got it working (from 19.21 to 19.23) – but appreciate that EBS Support may have an issue with it.
Yes there are additional steps to do and components to deploy in the new OH, but nothing that can’t be scripted and made semi-automated – or even just documented.
Just wondering if you have any further insight into the EBS teams thought on OOP?
Thanks
Des
Hi Des,
I don’t unfortunately.
But this is likely to change with 23c.
Cheers,
Mike
Hello Mike. We have been patching out of place for years, however we use to make a clone of the previous home, with the drawback you are talking about: copy of the entire history (.patch_storage) of the home. I wonder what can happen if you need to rollback a one-off patch that required postinstall. If we move the database to a home without this patch installed, how can datapatch rollback? We think it’s mandatory roll back the patch on db catalog *before* the switch. Are we wrong?
Hi David,
datapatch keeps the rollback information in a ZIP inside your database. I will publish a blog post the coming days explaining this in more detail.
Cheers,
Mike
Nice2Know: 19.x:Clone.pl script is deprecated and how to clone using gold-image (Doc ID 2565006.1).
I really hope they keep dragging it along. It takes only minutes and is command line and no X forwarding required.
Or minimally, I hope it’s same functionality gets well built into the dbca and can be executed in silent mode, and performs the same actions, with more importantly, the same speed.
Hi TWR,
I think there are some changes in DBCA -silent coming in this direction with 23c.
Cheers,
Mike
There is some extra work attached to out-of-place-patching. We are doing patching as a fully automated task and, for the moment, this is done in-place. We plan to change that, but have seen a obstacle. The database home must be changed in Oracle Enterprise Manager when patching are finished. I do not know if this can be done through emcli? Do you know anything about this matter?
Hi Kurt,
to be very frank, I am not an EM expert. Another customer told me that the patch in-place because of EM and the agents. Otherwise, they need to rediscover the home again after patching since the agent is not finding it. I am not aware of any command in emcli – but I wonder about it since this question may not be brand-new.
Thanks
Mike
Read comment #1. We do it without changing home names and paths and EM just picks up on the patch changes.
But emcli might be doable. You can certainly use emcli to discover new targets, which is your out of place home. You can also use it to delete targets, so bye bye old home.
My problem is this. I only install and use emcli on our OMR host. I AM CERTAINLY NOT going to install it on all 227 db hosts that run databases for us.
But as long as you tracked all the names of the new homes and old homes, I think you could perform those emcli actions from a centralized install. But I’m not sure.
Hopefully it’s not the same problem that we get with blacking out pdbs.
1 – emctl – cannot do it, and oracle has no plans to help – so say I run bash scripts as os command jobs on db hosts, I have no way to properly blackout pdb targets unless I use the -nodeLevel flag. Which we do for quick running jobs, but not what I want to use for a 4 hour database clone job.
2 – Oracle says to use emcli as my workaround, except I’m trying to run an os command job, that runs against a db host, and I do not want to install emcli on all my database hosts. Maybe a multi-step EM job is the answer, with a first step being emcli os command job on my OMR target, but I can’t pass in the name of the pdb target I want blacked out. And now I’m feed up and irritated that I have to work this hard at it.
Hi TWR,
this goes a bit beyond of what I am able to do on the blog 🙁
Especially, since I am not an emcli expert.
Cheers,
Mike
Yes Kurt, we can update new home with emcli. We’re doing it in our environment. Command for that would be like below.
$EMCLI_HOME/emcli modify_target -name=”$TARGET_NAME” \
-type=”oracle_database” \
-properties=”OracleHome:$NEW_HOME” -on_agent
You do not need to run a rediscover. It is enough to just change the Oracle Home Path in the Monitoring Configuration. Still this is a manual task and with hundreds of databases this is not very fun. I am quite sure this might could be done with emcli, but I did not find the correct syntax yet.
Still a removal and rediscover can be scripted, including the Properties fields, but I find it both untidy and complicated compare to the litte task.
We will consider out-of-place later on, but this things must be solved first
Hi Mike,
Actually you don’t need to manually rollback the one-offs first, instead you can use -force_conflict option to let opatch do it for you.
-force_conflict
If a conflict exist which prevents the patch from being
applied, the -force_conflict flag can be used to apply the
patch. OPatch will remove all the conflicting patches before
applying the current patch. This will override the ‘silent’
behavior for conflicts and hence is meaningful only when used
with ‘silent’ option.
Yeki,
I think -force is for special cases, and not as a general weapon to avoid rollback.
Thanks,
Mike
I’ve been using force_conflict for over 3 years with no issues. I didn’t think it was for “special cases” as the “Oracle OPatch User’s Guide” says it can be used for that purpose
Bear in mind we only have OJVM patch on top of the RU, so I don’t know if it would work smoothly with other types of conflicting patches.
Hi Mike,
We’ve been doing out-of-place patching in our Oracle Restart environments for both grid and database homes. But we’re installing fresh home for database but not for grid. For grid patching, we use opatchauto out-of-place option which does cloning of existing home which is not recommended, correct?
Will patching time increase with each RU apply ? What is the recommended way for grid out of place patching? Is it performing a software only grid install with latest patches in a different location and switch to this new home.
Hi Arun,
Daniel from our team wrote a longer blog post series about the patching options, up/downsides.
See https://dohdatabase.com
Cheers,
Mike
Hello Mike
We’ve been doing out-of-place patching in our Oracle Restart environments for both grid and database homes. But we’re installing fresh home for database but not for grid. For grid patching, we use opatchauto out-of-place option which does cloning of existing home which is not recommended, correct?
Will patching time increase with each RU apply ?
What is the recommended way for grid out of place patching? Is it doing software only grid installation with latest patch and switch services to this new home??
Hi Arun,
please see the blog posts of my team mate Daniel:
https://dohdatabase.com
evaluating all types with advantages and disadvantages.
Cheers,
Mike
Hi Mike,
I’ve been playing around a bit with OOP using autoupgrade. I noticed that the glogin.sql file is not copied to the new Oracle Home. Would it be possible for you to add this to autoupgrade accordingly? What could speak against it?
Best regards
Frank
Hi Frank,
I can forward this to the team.
THanks
Mike
Hi Mike,
I expected that autoupgrade would also take care of the DBA directory paths in OOP. However, I found that after the OOP with Autpupgrade some paths still pointed to the old Oracle Home. I had to correct this manually by running “@?/rdbms/admin/utlfixdirs.sql”.
Is this an autoupgrade bug or is this intentional?
Regards
Frank
Hi Frank,
this is something we discussed, and I think the next version of AU should take care on this.
Cheers,
Mike
How To Update OEM Targets Oracle Home Configuration After Out of Place Patching without using Enterprise Manager Cloud Control (Doc ID 2451994.1)
Hello Mike,
When we CPU-patch a database system (DBCS) VM on OCI, it will do inplace-patching. Do you think that will change soon too?
Hi Paul,
I would hope so – but I have no influence on this unfortunately.
Cheers,
Mike
Hi,
After (actually decades!) doing in-place patching we no want to change to OOP mainly because of (down)time constraints. But we are facing the issue that 3rd party backup software (Networker and CommVault) need the correct ORACLE_HOME in each Client configuration. Editing 100+ Client settings after each Patching is simply a NO-CANDO for us. Are there any tweaks available? Just creating a softlink with the old home name pointing to the new one does not work.
Regards,
Sandra
Hi Sandra,
one customer I talked to had the idea of using alternating homes, for instance:
/u01/app/oracle/product/19a
/u01/app/oracle/product/19b
and having two scripts, one which turns all references to 19a, and the other one to be called a quarter later, turning everything to 19b.
Cheers,
Mike
I’ve used the Alternating Oracle_Homes method successfully for several years both on Windows and Linux servers for several clients. On Windows you need to re-create the SID service via oradim.exe to run from the alternate Oracle_Home
Yes – and soon we’ll do this all for you in AutoUpgrade as well.
Cheers,
Mike
“You need to remove all non-RUs from the Oracle Home before you can apply the next RU to your existing home. Hence, you have a lot more work to do. Now, some customers complain about this. But that is the concept of one-offs, merges and patch bundles (e.g., the Data Pump Bundle Patch).”
The above is not my experience. With the latest April 2025 Database 19c patches when I apply the 19.27 GI RU (37641958) to the grid home, which already has JDK Bundle Patch 37213431 applied, there is no complaint it just removes it:
[INFO] sub-set Patch [37213431] has become inactive due to the application of user-set patch [37642901].
…
[INFO] Start invoking ‘patchgen’ at Fri Jun 27 ….
[INFO] Command call is : /u01/app/oracle/product/19.0.0/grid/bin/patchgen commit -rb 37213431
But, when I apply the 19.27 GI RU (37641958) to the db home, which also already has JDK Bundle Patch 37213431 applied, it fails and I am forced to perform rollbacks:
Superset Patch 37642901 has Subset Patch 37260974 which has overlay patches [37213431] and these overlay patches conflict with Superset Patch or are not included in Superset Patch.
I don’t understand?
Hi Nelson,
for the database home, this is (I’d say) expetced with the OCW component. BINARY-wise, one-offs need to be removed from the DB home.
Simple solution: Patch out-of-place.
Cheers,
Mike