From all my quarterly blog posts you know that I patch usually in-place for space and other reasons. And you may have similar reasons when you patch. But it is not ideal – and just speaking about risk and downtime, you should always patch out-of-place. So let me show you how Patching to Oracle 19.19.0 out-of-place with additional patches works.
Photo by Stefano Ferretti on Unsplash
What do I need?
Actually, this is not complicated at all. And if you never done it this way, give it a try now please.
- Base release Oracle Database 19.3.0 image
- opatch 37 from April 2023
- Release Update 19.19.0 from April 2023
- Data Pump Bundle Patch on top of RU 19.19.0 from April 2023
- OJVM Patch Bundle 19.19.0 from April 2023
- MRP1 on top of 19.19.0 from May 2023
And of course, you can add more fixes. Just make sure they are built on top of 19.19.0. The above list is conflict free to each other. I described the process already here. But today I will add more patches, and then have AutoUpgrade patch my database to the new home.
Where do I start?
Starting point is my new home. I will unpack the base release 19.3.0 into this home. And of course, set your environment correctly before you start.
cd /u01/app/oracle/product mkdir 1919 cd 1919 cp /home/oracle/stage/LINUX.X64_193000_db_home.zip . unzip LINUX.X64_193000_db_home.zip rm LINUX.X64_193000_db_home.zip rm -rf OPatch cp /home/oracle/stage/p6880880_210000_Linux-x86-64.zip . unzip p6880880_210000_Linux-x86-64.zip rm p6880880_210000_Linux-x86-64.zip
I need to remove the existing OPatch subdirectory, and exchange it with the most current one. Always include this step, regardless of what the readmes tell you.
Let’s prepare our patches
It’s key that you separate all additional patches and bundles into separate subdirectories. Otherwise, you will overwrite the PatchSearch.xml file, and patches can’t be accessed and installed.
.
├── dpbp
│ ├── 35261302
│ └── PatchSearch.xml
├── mrp
│ ├── 35333937
│ └── PatchSearch.xml
├── ojvm
│ ├── 35050341
│ └── PatchSearch.xml
└── ru
├── 35042068
└── PatchSearch.xml
We can kick it off
Now let me install everything with just one command. And since the MRP currently is not known to the OUI (aka runInstaller), I need to add every of its subdirectories separately. This will be fixed in a later RU.
├── mrp │ ├── 35333937 │ │ ├── 34340632 │ │ ├── 35012562 │ │ ├── 35037877 │ │ ├── 35116995 │ │ ├── 35225526 │ │ ├── README.html │ │ └── README.txt │ └── PatchSearch.xml
Hence, my string will become a bit lengthy:
./runInstaller -applyRU /home/oracle/stage/ru/35042068 -applyOneOffs /home/oracle/stage/ojvm/35050341,/home/oracle/stage/dpbp/35261302,/home/oracle/stage/mrp/35333937/34340632,/home/oracle/stage/mrp/35333937/35012562,/home/oracle/stage/mrp/35333937/35037877,/home/oracle/stage/mrp/35333937/35116995,/home/oracle/stage/mrp/35333937/35225526
So what am I doing?
I use the command:
./runInstaller -applyRU ... -applyOneOffs ...
where I separate the one-offs (the OJVM and the Data Pump Bundle Patch are technically one-offs as well) by comma. Once Bug 35251760 – ADD SUPPORT FOR “NAPPLYONEOFFS” TO BE ABLE TO INSTALL MULTIPLE ONEOFFS OR MRP USING GRIDSETUP/RUNINSTALLER will be fixed, you’ll be able to pass on the main bug number for the MRP, and it will fetch the subpatches automatically.
And then it installs and applies
Now it will take a bit – but not too long usually.
Once we are here (in my VBox environment on my laptop’s SSD this finishes in 12 minutes):
$ ./runInstaller -applyRU /home/oracle/stage/ru/35042068 -applyOneOffs /home/oracle/stage/ojvm/35050341,/home/oracle/stage/dpbp/35261302,/home/oracle/stage/mrp/35333937/34340632,/home/oracle/stage/mrp/35333937/35012562,/home/oracle/stage/mrp/35333937/35037877,/home/oracle/stage/mrp/35333937/35116995,/home/oracle/stage/mrp/35333937/35225526 Preparing the home to patch... Applying the patch /home/oracle/stage/ru/35042068... Successfully applied the patch. Applying the patch /home/oracle/stage/ojvm/35050341... Successfully applied the patch. Applying the patch /home/oracle/stage/dpbp/35261302... Successfully applied the patch. Applying the patch /home/oracle/stage/mrp/35333937/34340632... Successfully applied the patch. Applying the patch /home/oracle/stage/mrp/35333937/35012562... Successfully applied the patch. Applying the patch /home/oracle/stage/mrp/35333937/35037877... Successfully applied the patch. Applying the patch /home/oracle/stage/mrp/35333937/35116995... Successfully applied the patch. Applying the patch /home/oracle/stage/mrp/35333937/35225526... Successfully applied the patch. The log can be found at: /u01/app/oraInventory/logs/InstallActions2023-05-22_02-57-50PM/installerPatchActions_2023-05-22_02-57-50PM.log Launching Oracle Database Setup Wizard...
I will need to pass in my root command in the OUI’s screen:
And another screen:
And another screen …:
And more … I am silently hoping that there are ways to suppress these screens …
And now the important screen:
And then we are (almost) done
Well, finally after all these clicks, I’m getting to the OUI’s summary page where I can click the “Install” button:
Which I happily do – but just to get to a screen I will never understand:
Didn’t I say before (and confirm by passing on my root credentials) that I want the OUI do this for me? I have no idea why this screen is coming up …
Installation with all patches complete.
Let me do a quick check before we continue:
$ $ORACLE_HOME/OPatch/opatch lspatches 35225526;FRA E18POD EPM | ORA-00942 TABLE OR VIEW DOES NOT EXIST WHEN RUNNING SELECT QUERY 35116995;19.X DATAPATCH FAILS WITH ORA-01422 ,ORA-20004 NO FILE FOUND IN SCRIPT RDBMS/ADMIN/NOTHING.SQL 35037877;EM CODE IS BROKEN IN 19.18 35012562;OR EXPANSION IS NOT PICKED UP IN QUERY WITH CONTAINS OPERATOR AFTER UPGRADE TO 19C 34340632;AQAH SMART MONITORING & RESILIENCY IN QUEUE KGL MEMORY USAGE 35261302;DATAPUMP BUNDLE PATCH 19.19.0.0.0 35050341;OJVM RELEASE UPDATE: 19.19.0.0.230418 (35050341) 35042068;Database Release Update : 19.19.0.0.230418 (35042068) 29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399) OPatch succeeded.
This looks good.
But I am still not done
Final steps is to lift my databases to the new home. You can – of course – do this database-by-database. But we are in a modern IT environment and hence, we will use AutoUpgrade certainly and automate this task.
Let’s have a look at the config file at first:
global.autoupg_log_dir=/home/oracle/upg_logs global.timezone_upg=no global.restoration=no upg1.source_home=/u01/app/oracle/product/19 upg1.target_home=/u01/app/oracle/product/1919 upg1.sid=CDB2
And then I can kickoff AutoUpgrade to lift my database from 19.18.0 to 19.19.0 and execute all the necessary checks and steps.
$ java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar -config PATCH.cfg -mode deploy AutoUpgrade is not fully tested on OpenJDK 64-Bit Server VM, Oracle recommends to use Java HotSpot(TM) AutoUpgrade 22.5.221011 launched with default internal options Processing config file ... +--------------------------------+ | Starting AutoUpgrade execution | +--------------------------------+ 1 CDB(s) plus 1 PDB(s) will be processed Type 'help' to list console commands upg>
And shortly later …
upg> lsj +----+-------+---------+---------+-------+----------+-------+-------------------+ |Job#|DB_NAME| STAGE|OPERATION| STATUS|START_TIME|UPDATED| MESSAGE| +----+-------+---------+---------+-------+----------+-------+-------------------+ | 100| CDB2|DBUPGRADE|EXECUTING|RUNNING| 23:04:44|24s ago|0%Patching CDB$ROOT| +----+-------+---------+---------+-------+----------+-------+-------------------+ Total jobs 1
Now AutoUpgrade completed the patching process successfully:
upg> Job 100 completed ------------------- Final Summary -------------------- Number of databases [ 1 ] Jobs finished [1] Jobs failed [0] Jobs restored [0] Jobs pending [0]
All set. And you can automate this perfectly well for all your databases.
Are we done now?
Not fully. Once you are fine with the new home, all is working well, you may want to cleanup the previous, old home. And in addition, you may also cleanup the patch leftovers from the previous patch run as I explained in Cleaning up patch artifacts and improving patching performance.
One of the real benefits with out-of-place patching is that you can go back to the previous home easily – while when you patch in-place, you will have a lot of extra work to do rolling out patches in the right order.
And there may be more …
Please take into account certain files you may need to move around by yourself:
Further Links and Information
- Patching Process with MRPs
- MOS Note: 888.1 – Primary Note for Database Proactive Patch Program
- MOS Note: 555.1 – Oracle Database 19c Important Recommended One-off Patches
- Simple database installation with runInstaller -applyRU -applyOneOffs
–Mike
This is a very nice compressed guideline of all we need to do.
Even including the “rm -f *.zip”, which is often disregarded 🙂
Thank you for that, Mike.
And don’t forget to update the listener-config (my own favourite mistake)
Thanks for the hint, Dieter – let me check if AutoUpgrade does update the listener config as well 🙂
But I added your hint to the blog post, too!
Cheers,
Mike
I finally got my answer. we must install MRPs, one by one via applyOneOffs option. it would be nice to use only MRP patch number instead of all single patches. Thanks for explaining Mike.
Hi Mustafa,
I updated the blog post with the bug number the OUI team is supposed to fix soon.
Then you can add the root patch number of the MRP instead of each subdirectory.
Cheers
Mike
Hi Mike!
Some time ago I found amazing thing:
The MRP patch can still be installed with a single command, without having to list all the patches that make it up. I just checked on DBRU version 19.19, GI MRP and OPatch version 12.2.0.1.37 with this command:
./runInstaller -applyRU $HOME/INSTALL/35037840/35042068 -applyOneOffs $HOME/INSTALL/35333960,$HOME/INSTALL/35050341,$HOME/INSTALL/29213893
And the result:
29213893;DBMS_STATS FAILING WITH ERROR ORA-01422 WHEN GATHERING STATS FOR USER$ TABLE
35050341;OJVM RELEASE UPDATE: 19.19.0.0.230418 (35050341)
35225526;FRA E18POD EPM | ORA-00942 TABLE OR VIEW DOES NOT EXIST WHEN RUNNING SELECT QUERY
35116995;19.X DATAPATCH FAILS WITH ORA-01422,ORA-20004 NO FILE FOUND IN SCRIPT RDBMS/ADMIN/NOTHING.SQL
35037877;EM CODE IS BROKEN IN 19.18
35012562;OR EXPANSION IS NOT PICKED UP IN QUERY WITH CONTAINS OPERATOR AFTER UPGRADE TO 19C
34340632;AQAH SMART MONITORING & RESILIENCY IN QUEUE KGL MEMORY USAGE
33872599;OCW Interim patch for 33872599
35042068;Database Release Update : 19.19.0.0.230418 (35042068)
So maybe OIU already knows how to put MRP patches?
Regards, Vadim
That is interesting – so you apply the GI MRP to the database home. Is this a Oracle Restart (Single Instance High Availability) environment?
Thanks
Mike
Hi Mike!
No it is not Oracle Restart, just simple DB Home. I use GI MRP, because it contains same patches as DB MRP and OCW patch in addition (I checked it before installing of course). I prefer to install OCW on DB Home, because sometimes I need to convert DB-only instance with datafiles in filesystem to DB + GI with ASM datafiles storage. And my DB will be already ready for GI with OCW patch installed.
Regards, Vadim
Hi Vadim,
got you – I assumed that you do this with a Oracle Restart home. The teams are still evaluating things, and I think you are in contact with Support as well about this.
Cheers,
Mike
it was working for database home too if MRP is installed via opatchauto. when MRPs are installed via opatch napply, then applyOneOffs stopped to work for Database home
To get rid of the GUI dialogs I am doing the command “….runInstaller -silent -responsefile /home/oracle/database.rsp…….” where the rsp is generated from the GUI-installer in advance. By doing this the job can be repeated on many servers through scripting or whatever you like.
Oh thanks – I silently assumed there would be a way – I will give this a try.
But does this work in combination with -applyRU and -applyOneOffs ?
Thanks for sharing!!!
Mike
Hi Mike,
In case you not try it yet, -silent does work in combination with -applyRU and -applyOneOffs, that’s very convenience.
Thanks Yeki,
I am trying it out right now:
./runInstaller -applyRU /home/oracle/stage/ru/35042068 -applyOneOffs /home/oracle/stage/ojvm/35050341,/home/oracle/stage/dpbp/35261302,/home/oracle/stage/mrp/35333937/34340632,/home/oracle/stage/mrp/35333937/35012562,/home/oracle/stage/mrp/35333937/35037877,/home/oracle/stage/mrp/35333937/35116995,/home/oracle/stage/mrp/35333937/35225526 -silentCheers,
Mike
which does not work – it needs more parameters … 🙂
Hi Mike,
The -silent parameter need -responsefile together 😉
Haha … I made it work as well 🙂
./runInstaller -applyRU /home/oracle/stage/ru/35042068 \-applyOneOffs /home/oracle/stage/ojvm/35050341,/home/oracle/stage/dpbp/35261302,/home/oracle/stage/mrp/35333937/34340632,/home/oracle/stage/mrp/35333937/35012562,/home/oracle/stage/mrp/35333937/35037877,/home/oracle/stage/mrp/35333937/35116995,/home/oracle/stage/mrp/35333937/35225526 \
-silent -ignorePrereqFailure -waitforcompletion \
oracle.install.option=INSTALL_DB_SWONLY \
UNIX_GROUP_NAME=oinstall \
INVENTORY_LOCATION=/u01/app/oraInventory \
ORACLE_HOME=/u01/app/oracle/product/1919 \
ORACLE_BASE=/u01/app/oracle \
oracle.install.db.InstallEdition=EE \
oracle.install.db.OSDBA_GROUP=dba \
oracle.install.db.OSOPER_GROUP=dba \
oracle.install.db.OSBACKUPDBA_GROUP=dba \
oracle.install.db.OSDGDBA_GROUP=dba \
oracle.install.db.OSKMDBA_GROUP=dba \
oracle.install.db.OSRACDBA_GROUP=dba \
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false \
DECLINE_SECURITY_UPDATES=true
Seems like it might be more efficient to include the OJVM and data pump patches in the Database release update.
Are there defensible reasons why someone patching the database would want to avoid those two patches?
OJVM is already included in 21c – and will be in 23c.
We are working on having DPBP included as well once it is RAC Rolling – but it isn’t yet.
Cheers
Mike
Hi Mike, may I ask you if autoupgrade does already support RAC (including GI) for the mentioned OOP-switch-step? So far my preferred way for applying RUs on our RACs was using the OOP-patching-feature of opatchauto with the disadvantage that additional one-off-patches have to be installed ‘manually’ afterwards.
Thanks & regards
Axel D.
Hi Alex,
yes, AU supports this – with one important restriction. It does not do the “rolling” approach right now. Thus, it restarts the DB – which is not intended and wanted in many cases. Future version of AU will support the rolling approach as well. And ALWAYS ONLY out-of-place.
Cheers
Mike
Hi Mike ,
What about the datapatch -verbose ,will autoupgrade utility will take care,
what if we have DG setup ? which home we need to switch like this first standby or primary ?
Regards,
Balaji
Sure, it does. Try it out please.
Otherwise, this would be really imperfect 🙂
DG setup:
At the moment, it takes care on the PROD environment only. But we work with high priority on the DG setups as well.
Normally, in a DG environment, you would flip the standby over to the new home already, then start redo apply again (that works nicely), and then you invoke AU on the primary and let it do its work.
Cheers,
Mike
Hi Mike,
I am not sure if this is in your zone. There are some patches like 29259068 or 26749785 etc. These are listed as a warning in orachk report. why they are not added into MRPs?
Very simple – please see the readmes:
https://updates.oracle.com/Orion/Services/download?type=readme&aru=25195443
==> This patch is non-Data Guard Standby-First Installable
https://updates.oracle.com/Orion/Services/download?type=readme&aru=24595385
==> This patch is non-RAC Rolling Installable.
Hence, they both can’t be included.
RUs are always guaranteed RAC Rolling and Standby First.
Cheers,
Mike
many thanks, you said “RUs are always guaranteed RAC Rolling and Standby First” so MRPs are too? I though they can be added into MRPs not RUs.
MRPs are RAC Rolling and Standby First as well.
Cheers
Mike
Hi Mike,
What about read-only Oracle Homes? Would this process work with this configuration?
Thanks,
Steve
Yes!
Cheers
Mike
Hi,
Can I install Combo RU 35058163 in the same manner (one PatchSearh.xml file for both patches) with option -applyRU?
I don’t think so – I think you must point to the individual subdirectories. But I have not tried it …
Can you give me feedback please once you had a chance to try it out?
Cheers
Mike
You have right, option -applyRU doesn’t work with Combo.
I didn’t try installation with point to the subdirectories.
I just tried and if you point to the subdirectories it works.
Hi Mike ,
I will have to patch my env from 19.16 to 19.19 and ive been doing in our test env the following order with Opatch apply (not from the runInstaller) .
The following is the order of my patching the binaries on new ORACLE_HOME (out of place)
LINUX.X64_193000_db_home.zip ==> 19.3
p35058163_190000_Linux-x86-64.zip ==> 19.19 (OJVM + DB Update (35058163))
Patch 35261302 – DATAPUMP BUNDLE PATCH 19.19.0.0.0
Patch 34614081 – AH DIAGNOSTIC : NOT DUMP THE SQL_ID FOR USER QUERIES IN ALERT.LOG (BUG 33118355)
Patch 34832725 – ORA-4031 “KSU:STATS_FREELIST” AND “KGLSESHTTABLE” ERRORS IN 19C
Patch 35012562 – OR EXPANSION IS NOT PICKED UP IN QUERY WITH CONTAINS OPERATOR AFTER UPGRADE TO 19C
Patch 35095748 – NO SEMI JOIN IF ONLY THE COLUMNS USED IN THE GROUP BY CLAUSE ARE PROJECTED
When iam trying to apply the patch 35095748 ,while performing the “opatch prereq CheckConflictAgainstOHWithDetail -ph ./” , i got the below conflict
Invoking prereq “checkconflictagainstohwithdetail”
ZOP-40: The patch(es) has conflicts with other patches installed in the Oracle Home (or) among themselves.
Prereq “checkConflictAgainstOHWithDetail” failed.
Summary of Conflict Analysis:
There are no patches that can be applied now.
Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
35012562, 35095748
Conflicts/Supersets for each patch are:
Patch : 35095748
Conflict with 35012562
Conflict details:
/u01/app/oracle/product/19.19.0/db_1/lib/libserver19.a:kko.o
OPatch succeeded.
Please suggest/advise.
Thanks,
Kumar
Hi Kumar,
the message tells it already:
Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
35012562, 35095748
Please open an SR and request a merge.
Cheers
Mike
Hi Kishore,
Here is what you need: Patch 35391697: MERGE ON DATABASE RU 19.19.0.0.0 OF 35012562 35095748
p.s. simply use patch conflict tool for such cases
BR, Nariman
When you try to rollback the 19.19.0 patches the system breaks because files are missing.
Patch 35042068 rollback (pdb CDB$ROOT): WITH ERRORS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/35042068/25183678/35042068_rollback_JAMCDB4_CDBROOT_2023Jun07_11_11_28.log (errors)
-> Error at line 63694: script rdbms/admin/dbmscr.sql
– SP2-0310: unable to open file “/u01/app/oracle/product/19.19.0/db_se2/sqlpatch/35042068/25183678/rollback_files/19.1.0.0.0/rdbms/admin/dbms_registry_basic.sql”
-> Error at line 63696: script rdbms/admin/dbmscr.sql
– SP2-0310: unable to open file “/u01/app/oracle/product/19.19.0/db_se2/sqlpatch/35042068/25183678/rollback_files/19.1.0.0.0/rdbms/admin/dbms_registry_extended.sql”
-> Error at line 71133: script rdbms/admin/catbctab.sql
– ORA-00904: “K”.”CERTIFICATE_ID”: invalid identifier
I think this should not be an issue anymore – sorry for approving your comment this late, me bad.
Cheers,
Mike
HI Mike ,
Seems like oracle 19.19 RU have some issues ?? After applying RU 19.19 , we observed node reboots in some and slowness in other dbs (Both are RAC env) . After reading couple of oracle doc they mentioned few known issues . Can you please check and confirm on this ASAP. Thanks.
Hi Luckyreddy,
not that I am aware of. But I may be the wrong person to ask. You please need to check with Oracle Support, or as last resort, with @KnownAsMarkus and @RACMasterPM on Twitter for issues such as node reboots.
Cheers,
Mike
Hi Mike,
Your base image is 19.3 but we are on RedHat 8 which is first supported in 19.11. Should 19.11 be the base image for patching in our environment?
Thanks!
Hi Ken,
I am not aware of 19.11.0 being released as a full release. I would guess that it is an RU – and hence, you need to start with 19.3.0.
Cheers
Mike
Hi Mike,
In theory for RAC databases we should install GI RU (for 19.19 it’s patch 35037840) witch contains patches for GI and DB. For others we can install DB patch (for 19.19 it’s 35042068 like above instruction).
I did two tests with database patching from 19.11 to 19.19. Both environments are SIHA (Oracle Restart).
On first env I installed using opatchauto with option -outofplace pointing to both OHs (GI and DB) using RU 35037840.
On second I did manual installation only DB OH using RU patch 35042068.
And a big difffrance with part of result opatch lsinventory is OCW component:
– first env shows 35050331;OCW RELEASE UPDATE 19.19.0.0.0 (35050331)
– second env shows old one 29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
So questions are:
– for what DB OH shows OCW component, why Oracle install it in DB OH?
– what is the best approach for patching DB OH on environment with GI OH (like SIHA), which patch should we install, what is better?
Thanks.
Hi Konrad,
by default, the OCW component got installed into every 19c DATABASE home.
Why has this been done? To make sure that OCW is there in case you use the DATABASE home within a Grid Infrastructure environment, regardless whether it is a RAC or a SIHA environment.
Normally, in RAC/SIHA environments you need to keep the OCW in DATABASE home updated as well. When you install only the DATABASE Release Update, you will miss that. So ideally, you would add OCW to this particular DB home as well. I think “opatchauto” does this when you feed it with the complete patch having everything in separate subdirectories.
Cheers
Mike
Is your OS OL7 or OL8?
Is setting environment variable CV_ASSUME_DISTID=OL7 not necessary anymore when installing a 19c DB HOME on OL8?
Hi Mane,
mine is still OL7. When I move our lab to OL8 I have several restrictions with the various releases. And in order to demo all the releases from 11.2.0.4 to 21c, OL7 is the only OS version I can settle on.
Cheers,
Mike
Hi Mike
Tried several times, with or without ORACLE_HOME env. But always getting invalid_oracle_home error during RU opatch session. I’m on OEL8. It would not even start (just ignoring patching) until i fixed swap size. So please can you clarify what kind of environment needs to be setup before starting ./runInstaller.
Eventually I gave up and did it std way, all worked fine so home was fine (soft install) obviously.
kr
Sorry to hear that 🙁
I’d hope that Oracle Support could help you then.
Cheers,
Mike
Hi Mike, so far so good… i tried today an out of place patching like described.
at first, patched a new oracle_home (/oracle/product/19.19)
Fine!
Running autoupgrade.jar (newest one) with a config file in deploy mode.
Error:
ERROR The following checks have ERROR severity and no fixup is available or
the fixup failed to resolve the issue.
Fix them manually before continuing:
SC_SYS_PUBLIC_GRANTS
2023-08-10 15:25:31.442 ERROR Dispatcher failed: AutoUpgException [A failed check has an ERROR severity but the fixup is unavailable or failed to correct the problem. Manually fix the problem and rerun AutoUpgrade]
As you mentioned in your Blogpost from march23, this is a precheck for checking right sys permissions.
What should i do? i probably could use an earlier version without this check but..naah.. 🙂
Hope for an Answer!
Hi Dennis,
sorry for my very late reply. I just found time to go through the comments which came in during my vacation or during OCW time.
Could you confirm this with the last version of AutoUpgrade?
If yes, and if you are still interested, then please:
1. open an SR
2. upload the AU logs (java -jar autoupgrade.jar -config yourconfig.cfg -zip)
3. send me the SR number or just use the comment function to this thread
Thanks,
Mike
Hi Mike,
Can you please look at Bug 35497444 – ORA-1873: the leading precision of the interval is too small after each db restart in 19.19 (Doc ID 35497444.8) and provide some insight? Even 19.20 is affected and there is no one-off for Linux.
Thanks,
Arun
Hi Arun,
I see several backports on several platforms being available already.
Can you please check with Support or request a backport for your platform?
Thanks,
Mike
Hi Mike,
Can I confirm that for an environment with Oracle Restart (GI), I could either
1. Apply patch 35037840 for both GI and Database
or
2. Apply patch 35037840 for GI and then apply patch 35042068 for the Database?
Thanks
Hi Mike. “Otherwise, you will overwrite the PatchSearch.xml file, and patches can’t be accessed and installed.” – PatchSearch.xml files are not needed
Well, but you overwrite them with the next one, the process will fail, regardless of whether they are needed or not. Put the patches all under one tree, and then it will fail with the error that is can’t find PatchSearch.xml file.
Cheers,
Mike
you need to delete all PatchSearch.xml files
I will try this the next time . thanks!
Mike
This all seems great in theory, but my team has raised a few questions about its practical use, especially where we have OEM, ZDLRA, OKV for TDE, etc.
1. out of place patching we have fresh install home to different location and apply MRP + one offs recommend patches. –> what about the patches which we are been Applying in our existing oracle homes for non-recommended or issue specific Database patches like SR suggestions or ERP related issues fix.
2. ERP oracle_home install we need more post steps to work as ERP database oracle_home.
3. ERP side we have context files and they refer oracle_home locations, switch over oracle_home is not simple task.
4. If we maintain 2 oracle_homes how to Monitor in OEM target.
5. OKV keys are installed/enrolled under oracle_home and if we change to new oracle_home we may need to re-enroll the endpoints every time.
6. As we are using automated Clone tool for clones , we may have to rewrite all the script to point correct active oracle_home during the clone.
7. ZDLRA wallet location also be in the dbs location and rman configuration we may need to reconfigure every time when we switch the homes.
Hi Jeff,
to be frank, I learned the hard way recently that some of our core products (EBS for example) don’t support out-of-place patching. We try to change this for 23c. And I confess that I was not aware of it. So, thanks for your summary – I need to share this with the team.
I will get back as soon as I collect more answers, at least for the products we can give answer on.
Cheers,
Mike
Hi Mike,
I do my patches/upgrades as out of place. Today I realized that some of Oracle defined directories are showing old home directory paths. These are the directories that point to old homes:
ORACLE_HOME
ORACLE_OCM_CONFIG_DIR2
ORACLE_OCM_CONFIG_DIR
DBMS_OPTIM_LOGDIR
SDO_DIR_ADMIN
DBMS_OPTIM_ADMINDIR
XMLDIR
XSDDIR
Should I recreate them with new paths? thanks.
Hi Mustafa,
I think this here is the solution:
https://mikedietrichde.com/2021/08/12/dbms_optim_bundle-and-out-of-place-patching/
==> $ORACLE_HOME/rdbms/admin/utlfixdirs.sql
And I thought this would happen now automatically.
It is supposed to be fixed when you kick off “datapatch -verbose”
Cheers,
Mike
thanks Mike, my databases are 19.20 and I have this issue, next Saturday I will patch 19.21 and then check if they are fixed or not. if not then I can add utlfixdirs.sql into my out of place patching steps.
Cheers,
Thanks Mustafa – and let me know how it worked.
The actions are supposed to be taken by datapatch. If this is not the case, then I can check directly with the team.
But most likely we will need an SR with the logs – so please keep the logs from the patching run (most of them are supposed to be in $ORACLE_HOME/cfgtoollogs/sqlpatch if I remember correctly).
Cheers,
Mike
Hi Mike, I just patched my dev databases to 19.21 and after patching directories were still showing the old home path. I have two dev databases and I run utlfixdirs.sql after patching. interestingly, directories on one of the databases are fixed but on the other one, DBMS_OPTIM_LOGDIR and
DBMS_OPTIM_ADMINDIR is still pointing old home. I will replace them manually.
Hi Mustafa,
this sounds like a bug to me. Especially, when DBMS_OPTIM_ADMINDIR is pointing to the wrong path, the new list of fixes won’t be found.
I’ll try to check this in my env as well as soon as I apply 19.21.0.
Cheers,
Mike
Hi Mike,
Sorry to ask this question in 19.19 RU but it pertains to 19.21 RU. Has Oracle changed the value of max_pdbs from 254 to 5 in 19.21? A third party vendor has informed us so I wanted to double check.
Thanks,
Arun
Hi Arun,
I did check with oradiff.oracle.com – and I did not see any default change.
So I guess this is a wrong information.
Cheers,
Mike
I have the same problem with MAX_PDBS, changed during update to 19.21. Instance of Oracle Rac can’t start because of difference beetwin instances ( 5 and 254 )
Hi Cameel,
where do you see this change? I checked with ORAdiff.oracle.com, and max_pdbs=4096 across all releases and patches.
So, please let me know more about it – I am curious. Or did this happen in the cloud, or on-prem??
Thanks
Mike
Hello, we’re currently testing out-of-place patching and I’ve question
why do you install the new ORACLE_HOME using runInstaller ?
why do you not copy the old ORACLE_HOME to the new location and apply the patches on that ?
thanks in advance for your answer
regards
Chris
Hi Chris,
very simple:
When you clone your existing home you carry everything with you – this is not wanted.
And you need to deinstall one-offs and patch bundles such as the data pump BP before.
So, it will take longer.
Cheers,
Mike
ok, thanks for the explanation.
Regards
Chris
Suppose I have used autoupgrade to move from 19.19 to 19.22. After a week has passed, I need to switch back to 19.19.
How can I do this without data loss? I don’t see any examples of how to handle this scenario.
Thanks
Hi Dave,
this is VERY simple luckily – and one rule applies: Have the EXACT same COMPATIBLE across all your CDBs if you do this with the CDB architecture.
But generally, just stop your database, start it up again in the old end and invoke “datapatch -verbose”. Since the database carries the rollback files with it, datapatch will do the right thing.
Cheers,
Mike
Hello Mike,
Do we have to follow any sequence for patching ? Like for 4 node cluster, we should be begin with node 1 first and in sequence complete it. Have you seen any issues where we patch node 2 first the 4 then 3 then 1?
Thanks
Bhushan Altekar
Hi Bushar,
please follow our recommendations in:
https://www.youtube.com/watch?v=sF-rmD78zIo&t=3514s
Cheers,
Mike
Mike, Your posts are always full of good information and pretty thorough. Unfortunately, I run on a Windows platform and the steps do not always work, the same. My question has this approach been tested out on a Windows environment? In my experience having 2 Oracle homes on one server is not a good thing.
Hi Angela,
Windows is … different 🙂
You know that, I know it, too.
So, at first, I have heard from customers that the parameter options on Win don’t work as they do on Linux or any other Unix.
Furthermore, OOP patching on Windows is a bit more tricky since you have to take care on the registry keys as well (and some other things such as the services running from a given home). We are trying to automate this in AutoUpgrade in one of the coming versions.
But in summary, Windows will need some extra steps.
Cheers,
Mike
Just an add-on note for out of place patching on windows. About the autoupgrade cfg file, the upg.source_home and upg.target_home must match exactly what stand in the inventory.xml file : it is case sensitive. For example if d:\oracle\19c is specified (in cfg file) instead of D:\oracle\19c (as in inventory.xml file), the autoupgrade deploy will fail with an error about SC_CENTRAL_INVENTORY check : ” an issue with the control inventory has been found : d:\oracle\19c is not present in inventory.xml”
Oh thanks – this is a good hint.
I will forward this to our “master of windows” and ask whether he can double check.
Cheers,
Mike
Hi Mike, thanks for all the good information you put it out there and it is quite helpful. Just wondering if a patch rollback is needed, other than repointing the database to the old Oracle Home and re-running the Datapatch from the old home is there any necessity for taking a backup of any sort before the datapatch runs? thanks
Hi Prema,
when you patch OOO, then you don’t need to rollback patches. Your new home will be clean binary-wise without conflicts, and datapatch will do all the magic since the database carries the specific rollback files in a BLOB.
Cheers,
Mike
Hi Mike
A question about applying misplaced patches in Grid/DB: I see that it’s being done since base version 19.3, but what happens if the homepage already has certain necessary patches applied in addition to the RUs? Would these be lost?
Thanks for the support.
Victor Guzmán
Hi Victor,
if you have one-off patches on top, you need to add them to your config file for AutoUpgrade.
Then AU will download the matching version on top of the new RU (as long as they exist already).
Thanks
Mike
tried OOP on a 19c test VM and when trying to switch home, I get:
[FATAL] [INS-45101] Clusterware is not running on the local node.
ACTION: Ensure that the Clusterware is configured and is running on local node before proceeding.
Found KB125082 (19.x: gridSetup.sh -switchGridHome fails with ” [INS-45101] Clusterware is not running on the local node “)
Solution: As switchGridHome is currently not supported on oracle restart environment,Apply patch on existing GI_HOME .
This means, OOP for GI on Non-RAC is not supported with 19c ???
Hi Mane,
please see Daniel’s blog posts about GI patching on dohdatabase.com
https://dohdatabase.com/2024/11/12/how-to-patch-oracle-restart-19c-and-oracle-database-using-out-of-place-switch-home/
https://dohdatabase.com/2024/11/20/how-to-patch-oracle-restart-19c-and-oracle-data-guard-using-out-of-place-switch-home/
Cheers,
Mike
Hi Mike ,
Does Opatchauto prepare clone and then switch home is supported in Restart environment on 19.25 ?
we have tested it works in lab machine ,but raised SR for confirmation says its not supported ,however no documentation or supporting note is provided
Please share the SR number with me.
Daniel has shared all the techniques on his blog: dohdatabase.com
No idea why Support said that something is not supported.
When this happens, ask for the MOS note or the documentation where it says “not supported” or whatever proof.
Cheers,
Mike
Are there any changes for this installation method for RU 19.32 according Oracle Support recommend to separate the OJVM installation and do the later manually (OJVM, DPBP, JAVA, RU) ?
Sound strange according installation completes with only one error line (think not critical error line) in the logfile on one server but many different strange logfile errors on other server where prereq. Check failed according swap space (javax,….).
Never heard of this – where did Support recommend to “do it later manually”?
Was this via an SR? Please don’t hesitate to share it with me.
Cheers,
Mike
Hello Mike, the SR Number: 4-0003383484 and last reply:
”
do not apply OJVM patch while performing installation
First do the install and then install OJVM manually ”
## Logfile Extract from Binary Installation with (RU 19.32, OJVM, DPBP, JDK, OPatch)
# 1. Example Logfile from Server where only one line contains an WARNING without prereq. failed
“WARNING: Could not locate file: /u01/app19/oracle/product/dbhome_32/install/files.lst”
# 2. Example of some lines (more exists) of the logfile from Server – where it seems precheck fails – but why complain missing lib files ?
…
WARNING: [Jul 29, 2026 8:35:28 AM] Cannot set feature: http://javax.xml.XMLConstants/feature/secure-processing: java.lang.AbstractMethodError: javax.xml.parsers.DocumentBuilderFactory.setFeature(Ljava/lang/String;Z)V
…
INFO: WARNING: The action did not find source file /u01/app19/oracle/product/dbhome_32/lib/naeet_ee.o.dbl. The destination of the action was not modified.
…
WARNING: [Jul 29, 2026 8:36:09 AM] Could not locate file: /u01/app19/oracle/product/dbhome_32/install/files.lst
WARNING: [Jul 29, 2026 8:36:09 AM] Validation disabled for the state finish
INFO: [Jul 29, 2026 8:36:09 AM] Adding ExitStatus SUCCESS_WITH_WARNINGS to the exit status set
INFO: [Jul 29, 2026 8:36:09 AM] PREREQS_FAILED_WITH_WARNING
Hi Jochen,
I read the SR now, and I see why the support engineer asked to:
do not apply OJVM patch while performing installation
First do the install and then install OJVM manually
Share the results
This is NOT a general guideline but proposed to narrow down the issue.
I had some issues in the past with the separation of patch directories, and I went to separate them fully to avoid the overwriting of the patchsearch.xml.
Normally, I would ask you to try the installation with ./runInstaller -applyRU … -applyOneOffs …
See an example here:
https://mikedietrichde.com/2022/05/17/simple-database-installation-with-applyru-and-applyoneoffs/
But I see that you quoted it already in the SR 🙂 🙂 🙂
Did you try it out this way, and did it work?
Just to (again) to narrow down the problem.
Thanks,
Mike
Yes all patches extracted to own subdirectory and tested on 2 Server where only the “..install/files.lst” logfile line appear but on one other RHEL8 server with failed prereq. checks many error lines appear. Same installation calls scripted on this 3 Servers.
We think that the error could be ignored, but missing lib files in logfile sounds strange ?
INFO: We only speak here from binary installation (on datapatch or rollback we found no problem). I read that there are some security changes by javax in the new jdk versions… hope that will not interfere with apply OJVM and JDK with this method of runInstaller ?
Hi Joe,
I saw your comment in the SR about security changes – I am really curious about this as well.
But I have no good solution at this point.
Thanks
Mike