Some of you may remember my blog post from April 22, 2021 about issues I saw with the unattended installation using applyRU and applyOneOffs switches. The installation failed with a strange jox error when I passed on the OJVM bundle to my installation queue. But there is relief – finally, after almost one year. So let me give you some background, and of course show you the workaround for the Simple database installation with applyRU and applyOneOffs.
Photo by Xan Griffin on Unsplash
What has happened?
A longer while ago, we introduced the very convenient ability to install patches right away in one action with runInstaller using the switches -applyRU and -applyOneOffs. You could have the most recent (or even an older) RU applied but also a list of one off patches, merges and bundles such as the OJVM patch.
But unfortunately, as soon as I’ve been thrilled by this simple installation technique, it started failing. Exactly in the case you wanted to apply the OJVM bundle, you received:
Error in invoking target 'irman ioracle idrdactl idrdalsnr idrdaproc' of makefile '/u02/patch/rdbms/lib/ins_rdbms.mk'. See '/u01/app/oraInventory/logs/InstallActions2022-05-16_12-29-19PM/installActions2022-05-16_12-29-19PM.log' for details.
from Oracle Database 19.11.0 on.
And this was flagged in the install.log:
INFO: /u02/patch/lib//libserver19.a(joxwtp.o): In function `jox_eujs_nowait': joxwtp.c:(.text+0xf7b): undefined reference to `jox_eujs_nowait_' INFO: make: *** [/u02/patch/rdbms/lib/oracle] Error 1 INFO: End output from spawned process. INFO: ---------------------------------- INFO: Exception thrown from action: make Exception Name: MakefileException Exception String: Error in invoking target 'irman ioracle idrdactl idrdalsnr idrdaproc' of makefile '/u02/patch/rdbms/lib/ins_rdbms.mk'. See '/u01/app/oraInventory/logs/InstallActions2022-05-16_12-29-19PM/installActions2022-05-16_12-29-19PM.log' for details. Exception Severity: 1 INFO: [May 16, 2022 12:39:35 PM] Linking RMAN Utility INFO: [May 16, 2022 12:39:35 PM] Linking RMAN Utility INFO: The output of this make operation is also available at: '/u02/patch/install/make.log' INFO:
I described it in this blog post:
And of course, I did open a bug for this issue and discussed this topic quite a few times with the development team:
- Bug 32816171 – 19.11.0 INSTALLATION WITH “-APPLYRU -APPYONEOFFS” FAILS WHEN OJVM 19.11.0 INCLUDED: MAKEFILEEXCEPTION WITH JOX_EUJS_NOWAIT
Also a MOS note got written – which basically told you to install OJVM afterwards separately.
This of course is not a fancy workaround as it leads the concept ad absurdum.
Finally, more customers hit this trap in the following months.
There is relief – with a fix
It took a little while – and then apparently after a lot of back and forth, my bug got closed as a duplicate of another issue which had been raised later.
- Bug 33865116 – REPLACE OBJ FILES IN LIBKNLOPT.A WITH FILES IN RDBMS/LIB WHILE LINKING ORACLE
Well, I don’t really care which bug is supposed to be a duplicate of another one. But I was curious whether this works. When you download the fix for Bug 33865116 and unpack it, you will recognize that it only exchanged the make file for the rdbms. Nothing more and nothing less than that.
So I did a quick test and kicked off an installation of 19.15.0 with the matching OJVM 19.15.0 on my Linux environment:
- RU: p33806152_190000_Linux-x86-64.zip
- OJVM: p33808367_190000_Linux-x86-64.zip
And as expected, it failed.
Then I added the fix for Bug 33865116 to my install queue – and now, magically, it worked fine without issues.
Hurray – this issue is now finally fixed. And as far as I can see, the fix should be included in the 19.16.0 July 2022 RU.
A quick step-by-step guidance
In short, this is what you need to do.
- Create a new Oracle Home
Create your new Oracle Home directory, for instance /u01/app/oracle/product/19_15_0 - Download 19.3.0 Base Release
Download the 19c base software and unzip it in this newly created Oracle Home - Download newest OPatch
Download the most recent OPatch and unzip it into the same directory with zip -o (overwrites the existing – too old – OPatch directory with the newer one) - Create a patch directory
Create a subdirectory where you unpack the patches into, for instance /u01/app/oracle/product/19_15_0/patch - Download the newest RU
Download the most recent RU, for instance 19.15.0. You can find it the easiest with MOS Note: 2118136.2 – Download Assistant for Patch Bundles. Unpack it into a separate directory, for instance into /u01/app/oracle/product/19_15_0/RU19150 - Download the newest OJVM
Download the most recent OJVM, for instance 19.15.0. You can find it the easiest with MOS Note: 2118136.2 – Download Assistant for Patch Bundles. Unpack it into a separate directory, for instance into /u01/app/oracle/product/19_15_0/OJVM19150 - Download the Patch for the install issue
Download the fix for Bug 33865116 and unpack into another separate directory, for instance into /u01/app/oracle/product/19_15_0/OUIpatch
Now you have the following structure:
$ tree -L 2
.
├── RU19150
│ └── 33806152
│ └── PatchSearch.xml
├── OJVM19150
│ └── 33808367
│ └── PatchSearch.xml
├── OUIpatch
├── 33865116
└── PatchSearch.xml
It is important to keep patches in separate subdirectories as otherwise the PatchSearch.xml will be overwritten by every other patch, and the installation will fail.
And here is the solution – it is time to install:
./runInstaller -applyRU RU19150/33806152 -applyOneOffs OUIpatch/33865116,OJVM19159/33808367
Of course, you can pass on more one offs such as the DST patches or the JDK. Just separate them with a comma from each other in the above call.
And finally, don’t forget to cleanup:
- Remove the patch download zip archives
- Remove the subdirectories where you unpacked the patches into
Further Links and Information
- MOS Note: 2788203.1 – 19c Installation with 19.11 OJVM RU Fails with Error “undefined reference to jox_eujs_nowait_”
- MOS Note: 2118136.2 – Download Assistant for Patch Bundles
- Oracle 19c Installation with 19.11.0 RU, OJVM and some other fixes (April 22, 2021)
- Opatch download link
- Download the fix for Bug 33865116
–Mike
Hello, Mike,
What do you think about to remove the SQL Developer from Oracle Home? To minimize OH size.
SQL Developer and SQLcl are good separate products, which we download, install and use independently from RDBMS. SD in OH is very old and occupy the disk space. If you remove it then you’ll not be able to apply patches to this OH. It is the old-fashion paradigm to include SD into OH, i think.
Ask, please, the development to remove SD from DB binaries?
Thank you !
Hi Yury,
I don’t see any issues when you remove them (except for Jeff Smith being angry with you 🙂 ).
Thanks
Mike
Thank you a lot for this blog and showing off so many tutorials.
I have a question, when you apply a patch you usually run various other steps (datapart, optimizaition, recompile, etc) Do you need to do any of these steps after you datapump import from a older database (19.9 to 19.15)?
Hi Matt,
no, you don’t.
Cheers,
Mike
Hi MIke ,
Is it possible to directly upgrade oracle 12c DB to 19c (with 19.15.0 RU) without using gold image.
e.g. if i unzip 19.15 RU in new 19c home and start running DBUA .will it work?
Hi Rajkumar,
you need 19.3.0, 19.15.0 – and then you should use AutoUpgrade, and not the fully outdated and limited DBUA.
Cheers
Mike
Do you know if there are any plans to include applyRU and applyoneoff functionality into opatch[auto]?
I’m trying to decide if its going to be simpler/quicker to unpack 19c and use runInstaller as above to build new home for patching and moved db’s to it or use the traditional opatch[auto] tools given your recent description of the perfromance issues opatch has currently. Obviously legacy 11/12 (which should have been upgraded to 19 by now :)) wont be able to make use of this.
This functionality almost looks like a move to build opatch functionality into runInstaller in future releases?
Hi James,
I doubt that this will be included anytime soon into opatch.
But maybe we can do something in our area. Watch out for some news in 2023.
Cheers
Mike
Hi Mike
Just want to make sure, this is for a new install, right? If we already have a 19c install patched to 19.14 RU for instance and want to patch to 19.16 or 19.18 RU if they become available, we just use opatch as usual, is that correct?
Hi Ed,
yes, a new install – brand-new home with all the software in it.
Cheers
Mike
D:\Oracle\db193>setup -applyRU RU19160/34110685
[INS-32825] Applying PSU is not supported in this platform.
…what Windows team doing? 🙂
Hi Bjoern,
I think this feature is not supported on Windows – unfortunately 🙁
Cheers
Mike
Hi Mike,
I have question about time zone upgrade. D
I did some testing on a virtual machine(Oracle Linux 8.6) and created two homes 19.3 (default timezone file v32) and 19.16 (I applied v39 timezone file). created a dummy database on 19.3 and then I used autoupgrade to move it to 19.16. even if timezone_upg is set to yes in config file, it didn’t upgrade time zone of of database.
but when I install 18c and do all steps all over (new dummy db on 18c), autoupgrade upgraded database time zone to 39.
so does autoupgrade, upgrades time zone only in upgrade operation (like 18c to 19c) and doesn’t it work on patching (like 19.3 to 19.16) ?
thanks.
Hi Mustafa,
sorry to say that but
a) yes, AU should do this
b) you please need to open an SR and
c) upload the AU logs generated with “java -jar autoupgrade.jar -config yourconf.cfg -zip”
Then please either send me the SR number or add it as a comment – nobody outside of Oracle or your company can see it.
Cheers,
Mike
Hi Mike,
thanks for fast response. here is the SR number: 3-31234860721
Hi Mike,
Support created a bug record about this problem. https://support.oracle.com/epmos/faces/BugDisplay?id=34802929&parent=SrDetailText&sourceId=3-31234860721
I just want you to inform.
have a nice day.
it should be “I want to inform you” in my previous message 🙂
Thanks Mustafa!
Cheers,
Mike
Thanks Mustafa – but I see that the Support engineer opened the bug without doing any sort of analysis. The developers won’t be amused and push back. He should have at least go down to the logs and find the task, and check why it had failed. Nothing you can influence – you did everything right. But when you have a second, ask the engineer via the SR why the TZ upgrade has not happened.
Cheers,
Mike
Hi Mike, I asked as you said and support said it must be something with how TZ check is done. by the way this is my suspicion too. Maybe it checks default time zone versions (18c has 31 and 19c has 32) but when au decides to do an upgrade then it checks actual TZ file etc. just a thought 🙂
Mustafa,
now I have a bit more clear picture. You did not UPGRADE but you did PATCH instead. Sorry if I had not recognized this earlier but there are so many parallel streams going on.
So if this is the case, when you patch, AutoUpgrade does not adjust the time zone – which makes no sense to me since you explicitly set upg1.timezone_upg=yes.
Then you as a user would assume that certainly AU will adjust the time zone when you patch as well.
Let me discuss this with the team.
I’ll get back to you.
Cheers,
Mike
Hi Mustafa,
now I have more clarification after discussion this with the team.
AU checks for the “default” time zone file – and this is 32 for 19.3.0 and 32 for 19.16.0. This is why AU assumes not to do a time zone upgrade as part of the patching. The team discusses this tonight in the AU call. And there is a bug already open – I will keep you posted.
Cheers
Mike
Mike,
I got your last comment with email but cannot see it in here. I tried to emphasize about “patching” and “upgrade” issue but clearly I failed 🙂 and of course you are so busy and probably getting many messages.
I am glad that reason is found and also my guess was correct (default timezone is checked) 🙂 thanks for your close attention and fast response.
Hi Mustafa,
the team is trying to deliver a fix overnight. But we’ve had a more general discussion yesterday, and not everything is sorted out to the final bit.
We agreed on the EXPLICITLY having timezone_upg=yes in the patching case should lead to the patching run changing TZ. But since timezone_upg DEFAULTS to YES, it is a bit tricky to do this the right way. Please stay tuned, we’ll get back to you.
Cheers
Mike
Hello Mike,
Base Information: 19.3 Base & 19.17 RU & OJVM & Datapump Patch & Latest Patch
I discovered maybe two problems with this installation method (try to script this gui & silent).
1) The installation continues also if no X11 found (no silent installation) …. should exit with error code 1 is better than completion but problem with inventory and wrong root scripts…. ?
2) If you create Response File before via GUI Installation and then use -silent -responseFile (installation seems to complete but after inventory check: /u01/app19/oracle/product/19.17.0.0/oraInst.loc => File could not be found and last line of install logfile could not find this file: /u01/app19/oracle/product/19.17.0.0/install/files.lst)
ERROR: Unable to verify the graphical display setup. This application requires X display. Make sure that xdpyinfo exist under PATH variable.
Are there any special actions/flags needed – or is silent installation not supported with this method ?
Current workaround: Use GUI Installation or manual Setup (maybe current the best method).
Thank you
Hi Joe,
I’m not an installer expert. You may please need to open an SR and check with Support.
Thanks
Mike
Mike, let’s assume the base release is installed and the latest opatch and latest ru and the previous version had 10 oneoff patches, that are no longer needed with the new version. When we run datapatch from the new ORACLE_HOME, does datapatch find the undo sql’s of the old oneoffs from the old oracle_home?
Yes, it does.
Please feel free to watch our Virtual Classroom Seminar #16 from past week where we explained the mechanism 🙂
Cheers
Mike