In an previous blog post we showed how to install and patch in one single action. And as I use this technique all the time when I setup new homes, would like to show you an Oracle 19c Installation with 19.11.0 RU, OJVM and some other fixes. The purpose of this blog post is mainly to demonstrate again how you can install and patch in one action. But also to help you avoiding a pitfall when you apply the OJVM RU from April 2021.
Photo by Sammy Williams on Unsplash
What do we need?
At first, let us clarify what we need as the minimum set of software and patch bundles. As I’d like to go to 19.11.0 directly, I will need to download at least:
Then in addition, I will download the 19.11.0 Release Update from April 2021.
For Linux, this will be:
- DB RU 19.11.0.0.210420 Patch 32545013
- Opatch .24 Patch 6880880
This will be the minimum set: Software, Release Update and the correct Opatch version.
What do we want in addition?
But I want more. MOS Note: 555.1 (Oracle Database 19c Important Recommended One-off Patches) lists additional fixes on top. Unfortunately the list for 19.11.0 is empty at the moment but I have a few things on my wish list. So lets see whether those are available already for 19.11.0:
OJVM Release Update 19.10.0.0.210119 Patch 32067171 for all platforms
Ups – wait a bit – this is the January one. I want APRIL. Luckily MOS is “intelligent” and recommends to use a newer one as soon as I click on the 19.10.0 one:
Hence, this is the correct download:
- OJVM Release Update 19.11.0.0.210420
- Patch 32399816 with README
I guess this will be corrected soon. And I silently ignore the fact that the README still asks you to do a STARTUP UPGRADE.
In addition, I would like to have the newest time zone patch and the most recent JDK patch as the RU delivers only n-1.
- DST V36 Update – DSTv36
- RDBMS Patch 32327201 with README
- OJVM Patch 32327208 with README
- JDK Bundle Patch 19.0.0.0.210420
- Patch 32490416 with README
- SIGNIFICANT INCREASE IN LIBRARY CACHE MUTEX X WAIT TIME AFTER 19c UPGRADE
- Patch 32356628 with README
- Highly recommended by customers on the blog and not in the 19.11.0 RU but available already as one-off patch on top of 19.11.0
So I have a list of 5 additional one-offs in addition including the OJVM 19.11.0 bundle.
Installation with Patch Apply – First Attempt
Spoiler alert – from the headline for this chapter you may sense that something will go wrong. Let’s see.
My sequence of tasks:
- Download and unzip 19.3.0 into its new Oracle Home
- Remove the OPatch directory and unzip the newest opatch into the new Oracle Home
- Download and unzip 19.11.0 into a totally separate subdirectory
- Download and unzip OJVM Patch 32399816 into a different subdirectory
- Download and unzip Patch 32327201, Patch 32327208, Patch 32490416 and Patch 32356628 all into separate subdirectories
- Call the OUI and pass on the patches I’d like to apply right away.
This is how I setup the patch area to avoid xml files being overwritten by the next patch:
$ tree -L 2
.
├── dst
│ └── 32327201
├── dstojvm
│ └── 32327208
├── jdk
│ ├── 32490416
│ └── PatchSearch.xml
├── mutex
│ ├── 32356628
│ └── PatchSearch.xml
├── ojvm
│ ├── 32399816
│ └── PatchSearch.xml
└── RU
├── 32545013
└── PatchSearch.xml
Then my installation command will be:
[HUGO] oracle@hol:/u01/app/oracle/product/19.11.0 $ ./runInstaller -applyRU patch/RU/32545013 -applyOneOffs patch/jdk/32490416,patch/mutex/32356628,patch/dst/32327201,patch/ojvm/32399816,patch/dstojvm/32327208
You can follow the OUI applying the patches, patch after patch, to the new home:
$ ./runInstaller -applyRU patch/RU/32545013 -applyOneOffs patch/jdk/32490416,patch/mutex/32356628,patch/dst/32327201,patch/ojvm/32399816,patch/dstojvm/32327208 Preparing the home to patch... Applying the patch patch/RU/32545013... Successfully applied the patch. Applying the patch patch/jdk/32490416... Successfully applied the patch. Applying the patch patch/mutex/32356628... Successfully applied the patch. Applying the patch patch/dst/32327201... Successfully applied the patch. Applying the patch patch/ojvm/32399816... Successfully applied the patch. Applying the patch patch/dstojvm/32327208... Successfully applied the patch. The log can be found at: /u01/app/oraInventory/logs/InstallActions2021-04-22_04-04-40PM/installerPatchActions_2021-04-22_04-04-40PM.log Launching Oracle Database Setup Wizard...
At this point, you can interact now with the GUI:
and later:
This looks fine, and I can kick it off.
But I shouldn’t count my chicken before they are hatched.
It fails during RMAN link. That is not nice. The log says:
You can find the log of this install session at: /u01/app/oraInventory/logs/InstallActions2021-04-22_04-04-40PM/installActions2021-04-22_04-04-40PM.log Error in invoking target 'irman ioracle idrdactl idrdalsnr idrdaproc' of makefile '/u01/app/oracle/product/19.11.0/rdbms/lib/ins_rdbms.mk'. See '/u01/app/oraInventory/logs/InstallActions2021-04-22_04-04-40PM/installActions2021-04-22_04-04-40PM.log' for details.
And this is from the install log:
INFO: /u01/app/oracle/product/19.11.0/lib//libserver19.a(joxwtp.o): In function `jox_eujs_nowait': joxwtp.c:(.text+0xf7b): undefined reference to `jox_eujs_nowait_' INFO: make: *** [/u01/app/oracle/product/19.11.0/rdbms/lib/oracle] Error 1 INFO: End output from spawned process. INFO: ---------------------------------- INFO: Exception thrown from action: make Exception Name: MakefileException Exception String: Error in invoking target 'irman ioracle idrdactl idrdalsnr idrdaproc' of makefile '/u01/app/oracle/product/19.11.0/rdbms/lib/ins_rdbms.mk'. See '/u01/app/oraInventory/logs/InstallActions2021-04-22_04-04-40PM/installActions2021-04-22_04-04-40PM.log' for details. Exception Severity: 1 INFO: Error in invoking target 'irman ioracle idrdactl idrdalsnr idrdaproc' of makefile '/u01/app/oracle/product/19.11.0/rdbms/lib/ins_rdbms.mk'. See '/u01/app/oraInventory/logs/InstallActions2021-04-22_04-04-40PM/installActions2021-04-22_04-04-40PM.log' for details. INFO: [Apr 22, 2021 4:21:54 PM] InstallProgressMonitor: Completed phase 4
This seems to be the failure:
/u01/app/oracle/product/19.11.0/lib//libserver19.a(joxwtp.o): In function `jox_eujs_nowait': joxwtp.c:(.text+0xf7b): undefined reference to `jox_eujs_nowait_'
And my feeling tells me: This is coming from OJVM.
With a quick double-check (and the help of my colleague Thomas from presales who ran into this failure a bit earlier yesterday already), it became clear that I need to install OJVM separately.
- Bug 32816171 – 19.11.0 INSTALLATION WITH “-APPLYRU -APPYONEOFFS” FAILS WHEN OJVM 19.11.0 INCLUDED: MAKEFILEEXCEPTION WITH JOX_EUJS_NOWAITFurther addition:
19.11 OJVM RU includes fix for bug 32124570
This requires “jox_on” linking target to be executed as part of “applying” the patch.
But “runInstaller” does not execute this specific option.Hence at this point there is no quick way to fix this issue.
If you plan to apply the OJVM 19.11.0 RU, you need to do it separately.As an alternate approach you may use the “create goldimage” procedure:
https://docs.oracle.com/en/database/oracle/oracle-database/19/ladbi/Chunk7030974.html#GUID-30C45B5F-58A3-4D4C-8475-40093C51650B
Installation with Patch Apply – Second Attempt
Ok, let me repeat the exercise but now I will not include OJVM April 2021 RU but install it afterwards manually:
./runInstaller -applyRU patch/RU/32545013 -applyOneOffs patch/jdk/32490416,patch/mutex/32356628,patch/dst/32327201,patch/dstojvm/32327208
And now it works flawlessly:
$ ./runInstaller -applyRU patch/RU/32545013 -applyOneOffs patch/jdk/32490416,patch/mutex/32356628,patch/dst/32327201,patch/dstojvm/32327208 Preparing the home to patch... Applying the patch patch/RU/32545013... Successfully applied the patch. Applying the patch patch/jdk/32490416... Successfully applied the patch. Applying the patch patch/mutex/32356628... Successfully applied the patch. Applying the patch patch/dst/32327201... Successfully applied the patch. Applying the patch patch/dstojvm/32327208... Successfully applied the patch. The log can be found at: /u01/app/oraInventory/logs/InstallActions2021-04-22_04-41-25PM/installerPatchActions_2021-04-22_04-41-25PM.log Launching Oracle Database Setup Wizard... The response file for this session can be found at: /u01/app/oracle/product/19.11/install/response/db_2021-04-22_04-41-25PM.rsp You can find the log of this install session at: /u01/app/oraInventory/logs/InstallActions2021-04-22_04-41-25PM/installActions2021-04-22_04-41-25PM.log
I just need to apply the OJVM RU manually afterwards.
[HUGO] oracle@hol:/u01/app/oracle/product/19.11/patch/ojvm/32399816 $ $ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.24 Copyright (c) 2021, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19.11 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19.11/oraInst.loc OPatch version : 12.2.0.1.24 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19.11/cfgtoollogs/opatch/opatch2021-04-22_16-55-58PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 32399816 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/19.11') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '32399816' to OH '/u01/app/oracle/product/19.11' Patching component oracle.javavm.server, 19.0.0.0.0... Patching component oracle.javavm.server.core, 19.0.0.0.0... Patching component oracle.rdbms.dbscripts, 19.0.0.0.0... Patching component oracle.rdbms, 19.0.0.0.0... Patching component oracle.javavm.client, 19.0.0.0.0... Patch 32399816 successfully applied. Log file location: /u01/app/oracle/product/19.11/cfgtoollogs/opatch/opatch2021-04-22_16-55-58PM_1.log OPatch succeeded.
Bug number for this incident is: Bug 32816171 – 19.11.0 INSTALLATION WITH “-APPLYRU -APPYONEOFFS” FAILS WHEN OJVM 19.11.0 INCLUDED: MAKEFILEEXCEPTION WITH JOX_EUJS_NOWAIT.
UPDATE:
As of now, Nov 26, 2021, there is still no solution for the bug. So this issue will persist at least until 19.14.0, if not even longer.
Further Links and Information
- How to install and patch in one single action
- Patching all my environments with the April 2021 Patch Bundles
- MOS Note: 2725756.1 – Critical Patch Update (CPU) Program Jan 2021 Patch Availability Document (PAD)
- MOS Note: 555.1 (Oracle Database 19c Important Recommended One-off Patches)
- MOS Note: 412160.1 – Updated DST Transitions and New Time Zones in Oracle RDBMS and OJVM Time Zone File Patches
–Mike
I have been installing the quarterly RU and OJVM patches at the same time for a while now in a data guard configuration using the standby-first method. Our developers do not use OJVM, but we just have never explicitly avoided it during install or performed the removal steps. Performing the OJVM patching following the standby-first method has (as of yet) never caused other issues and no reported errors, maybe because no one uses OJVM.
I would prefer to keep doing what I’m doing, as it avoids downtime, however, I’m wondering if there are circumstances where I really should perform the OJVM patches the correct way?
Alternatively, should I just consider using the mitigation patch moving forward and avoid OJVM patches altogether?
Hi Matthew,
check the readme please – OJVM is not supported to be “standby first” 😉
But anyhow, the order of keeping OJVM separate in my example is caused by a bug and shouldn’t be the norm.
But honestly, if you ask me, the mitigation patch is a good thing in case you can’t deinstall OJVM. Just check please what I blogged about the MITIGATION patch here as there are a few things to take care on as well.
Cheers,
Mike
Thanks for posting – I opened an SR yesterday with a very similar observation on the OJVM patch. Would you mind sharing the bug number so I may attach it to my SR?
My SR # is 3-25737396181 (feel free to remove the SR# before posting, if it is against your guidelines).
Thanks!!!
Hi Pete,
I updated the blog post already:
Bug 32816171 – 19.11.0 INSTALLATION WITH “-APPLYRU -APPYONEOFFS” FAILS WHEN OJVM 19.11.0 INCLUDED: MAKEFILEEXCEPTION WITH JOX_EUJS_NOWAIT
is the bug number. And as far as I see, it hasn’t received any attention yet while I write this comment.
Cheers, and sorry for the inconvenience!
Mike
It’s funny because MoS site has no clue about this issue. It does not appear in the search results at least. Is the bug made not visible to public? Any info on when this is going to be fixed?
Hi Roman,
the bug is non-public as I don’t have privileges to set a bug “public”.
And there is no information available when this is going to be fixed.
Cheers,
Mike
Mike, I’ve been an Oracle DBA for 20+ years and this is quickly becoming my new favorite DBA blog. We were performing the first run of RU11 in our Lab environment today in prep for our main patching cycle and came across this very issue. AFter drawing a complete blank on a MOS search, this page is the only one I found in the general interwebs that referenced this error at all, and you even provide a workaround.
Your ideas and solutions are very well put together and I find more and more the most useful links I find for various issues come from your site. Just wanted to thank you for your efforts and your expertise!
Thanks for your feedback, Mike – and I’m happy that it helped!
Cheers,
Mike
Hi Mike,
thank you for providing that great upgrade blog for years (decades?) now.
I just walked through all steps but the OJVM Patch is “not needed” (at least Opatch tells me that .. -> https://pastebin.com/0D2yG6cj )
Am i missing something?
Hi Basti,
is JOX linked off potentially? This is the situation when I’d expect the patch to be not needed?
Or has it been applied already? I would need to see the inventory.
Check JOX:
https://mikedietrichde.com/2015/12/02/what-or-who-is-jox-and-what-happens-if/
Cheers,
Mike
Hi Mike,
i’ve had the old ORACLE_HOME (from the 18c i wanted to upgrade) set .. with the correct ORACLE_HOME the patch applies successfully .. sorry for the hassle.
No worries – glad if it worked now 🙂
Cheers,
Mike
Hi Mike,
I came to know about your blog when i attended the Oracle 19c Upgrade Series. Eversince I always look at your blog for scope for improvements and suggestions etc . Your blog is full of resources. I have just persuaded my team to apply the RU and oneoffs as part of the new installation/upgrade as part of 19c upgrade on our estate. It is really useful topic and just typing this after sucessfully did GI & DB home installations with 19.10.0. All works like a treat.
It is really helpful for many Senior DBA’s like us to explore new options and improve various mundane DBA tasks.
Thanks a lot.
Arun
Thanks a lot, Arun, for the kind feedback 🙂
Thanks so much, and take care!
Mike
Hi Mike,
what about OCW – remains at 19.3.0 Level (with 32545013, or better use: 32545008 ?)
Question 1: Is it recommended to use GI RU Patches – see ?
https://www.fernandosimon.com/blog/patching-ocw-for-oh/
Question 2: Seems there are troubles with Snapshot Standby Patching – maybe not supported (Unsupported named object type for bind parameter at $ORACLE_HOME/sqlpatch/sqlpatch.pm line 5645) ?
Hi Jo,
if you refer to the standard database home and its OCW partion:
I never applied an OCW patch to the DB home.
And regarding the Snapshot Standby datapatch – we have used this for testing several times. Maybe this is caused by a known issue within the patch bundle?
Cheers,
Mike
Hi,
sqlpatch.pm line 5645 was in my case caused by a tempfile in a PDB which could not be extended. Adding a tempfile to the PDB resolved the error.
Hope it helps…
Dieter
Thanks Dieter!
Cheers,
Mike
Hi Mike,
can we apply the corresponding OCW as well along with the DB RU , say for ex 32579761 (OCW) along with 19.11 RU(32545013) ..?
Hi Chan,
I didn’t apply it since I don’t use a component of the GI RU in my DB Homes.
But you could if you needed.
Thanks,
Mike
By the way bug 32356628 (Significant increase in library cache: mutex x wait time after upgrading database to 19c) is included in the 19.12 RU.
Thanks Martin – and yes, we pushed for it 🙂
Cheers,
Mike
Mike – this appears to still be an issue with 19.12 RU and 19.12 OJVM (32876380)
Hi Pete,
I know – and the bug is still open. So this will stay certainly for 19.13.0 as well for sure.
Cheers,
Mike
I am trying to install on zlinux redhat 8.4 the oracle 19c 19.11 and i receive the following
Exception String: Error in invoking target ‘libasmclntsh19.ohso libasmperl19.ohso client_sharedlib’ of makefile ‘/oracle/app/oracle/product/19c/dbhome_1/rdbms/lib/ins_rdbms.mk’. See ‘/tmp/InstallActions2021-10-08_12-49-33PM/installActions2021-10-08_12-49-33PM.log’ for details.
Exception Severity: 1
anyone know anything about the error ?
Please open an SR – please do so!
Thanks,
Mike
Servus Mike,
The bug you described still exists even in 19.13 RU + 19.13 OJVM:)
Cheers, Nariman
Hi Nariman,
yes, unfortunately it does – and there is still no fix in sight despite the fact that this is a Sev.1 bug.
thanks,
Mike
Hi Mike thanks for the throughtly detailed post. Helped me to mind some steps whiel doing Oracle updates
Question is, i have upgraded my grid 19 / ora 19 to RU13 sucessfully. do i really need to upgrade OJVM to 13. its a brand new install andi see no issues to upgrade it, but i would like to know if OJVM is a requrement for the engine itself. Its not clear to me
Hi Alexandre,
unless you have an application which puts java code into the database and uses the OJVM, there is no need for it.
Life is easier without it.
Cheers,
Mike
Hi Mike,
A silly question: as the JDK is included in each RU (“JDK patching happens with every RU since January 2020”), why you’re upgrading it anyway using 32490416?
regards,
Fabrizio
Hi Fabrizio,
because it is not the most recent one but the n-1 version. And not in the 11.2.0.4, 12.1.0.2, 12.2.0.1 homes but only for 19c.
Cheers,
Mike
It appears that this is still an issue with Linux x84-64 19.14 (Jan 2022)
Oh yes, unfortunately.
The bug is still not fixed.
Mike
Mike,
I have a unique situation with a database running on 19.8 without any Java components installed nor OJVM patches. If I move the database to a 19.14 plus OJVM new Oracle Home, would datapatch fail or attempt to install java?
Thank you,
-Pedro
Hi Pedro,
it won’t raise an error – if OJVM is in your home, that has no effect since the component is not in your database.
So neither an install nor a patch error will happen.
Thanks
Mike
This still seems to be around for 19.15 as well. Just tried to do a fresh install calling “-applyRU /oracle/patch/33859214/33803476/33806152 -applyOneOffs /oracle/patch/33859214/33803476/33815596,/oracle/patch/33859214/33808367” and It threw the exact same error.
If I don’t include the patch then it doesn’t seem to install OJVM at all. How can I do the OJVM install on its own? If I try to apply the patch it says that it doesn’t exist.
If I hit continue and the install finished and I can see the patches for OJVM have all been installed. But I assume there are probably going to be some underlying issues would bite me later with that install.
Hi Matt,
please see:
https://mikedietrichde.com/2022/05/17/simple-database-installation-with-applyru-and-applyoneoffs/
There is a one-off available, and the issue is finally supposed to be cured by default (without an extra one.off) in 19.16. (July 2022).
Cheers
Mike
Hello,
Thanks for your note about OJVM bug it helped a lot, the issue still exists on 19.15.
Thank you, Ismail!
Cheers,
Mike