This is a Friday morning blog post. And actually I had no intention to write it. I don’t mean it offensive. But yesterday I received an interesting email from Frits Hoogland from Enkitec/Accenture. Frits asked me about the July RUR and some unlogic things on MOS. Once I had answered his email, the next question came in. Hence, I thought I need to write something about our July 2018 Update, Revision, BPs, PSUs – delays and other issues.
From being on stage at last year’s DOAG Conference for a “Patching Forum” I know that things aren’t as obvious as we often think they are. And especially not when it gets into patching. This starts usually with acronyms which are truly familiar when you work in the patching area at Oracle. But they aren’t self-explanatory to customers who patch only once or twice a year. But this blog post is not meant to explain acronyms.
One important thing upfront: I’m not the Product Manager for Patching. I just patch and blog about it.
July 2018 Update, Revision, BPs, PSUs – delays and other issues
The July 2018 patch bundles are out for less than 48 hours, but I think I received 10 comments or emails already. So I’d like to explain and summarize some things I learned in the past 48 hours about our July patch bundles.
Frits Hoogland asking about logic
Frits Hoogland, who is a great guy and a very well known standout Oracle database internals expert asked me about an issue with the July 2018 Revision (RUR):
“[..] the link in the PSU note reports back that the patch can’t be found.
Is there any way this is logical?”
It took me a few minutes to retrace the “logic” Frits struggled with.
Frits wanted to download Patch 28346593 – Database Jul 2018 Release Update Revision (DB RUR) 18.2.1.0.180717, in brief the July 2018 Oracle 18c RUR 18.2.1. It is not important that I and others state they you should keep your hands away from RURs unless Support advises you to install them. We produce them and hence you can install them. In theory.
The above bug/patch note has an interesting section:
I marked the interesting part in BLUE. It points to a fix which supposingly superseeds the RUR from July 2018.
But how can this be when the patch is (in theory) out just for a day? And how can the patch number be significantly lower (27676517) than the one for the RUR (28346593)? It starts getting even more interesting when you click on the link for 27676517 as you’ll get another fix which superseeds this one:
But this is the April 2018 Release Update (RU). How can the April 2018 RU superseed the July 2018 RUR?
And wait a bit? Isn’t this the one Frits originally tried to access: 28346593 ? Yes it is. We are in a loop now.
I don’t know the logic behind this. And I can’t speak for the person who updated the notes. It could be even an automation process. But the solution is here and I mentioned it already in one of my previous blog posts about the July 2018 patch bundles:
Section 2.2 of Patch Availability for the Oracle Database (MOS Note: 2394520.1) has the details about delayed patch bundles. And here it says that the Oracle 18c RUR 18.2.1 is delay to an ETA of July 24, 2018:
Why didn’t the above Patch 28346593 – Database Jul 2018 Release Update Revision (DB RUR) 18.2.1.0.180717 mention it but instead has this sentence:
This is a marker bug for the 18.2.1.0.180717 (Jul 2018) Database Release Update Revision(DB RUR). This Release Update can be downloaded here: Patch:28346593
I don’t know.
Therefore, sorry Frits for the inconvenience and the wasted searching time 🙁 And yes, it’s not logic.
The wrong README and the wrong OPatch
Just 15 minutes after Frits’ email there was an internal email from my colleague Peter:
“Trying to patch my single instance 12.1.0.2.180417 with 12.1.0.2.180717, but this time it fails. “
Peter had done some of the usual things already. But even downloading it again led to no other result:
$ opatch lspatches 27475603;Database PSU 12.1.0.2.180417, Oracle JavaVM Component (APR2018) 27338041;Database Patch Set Update : 12.1.0.2.180417 (27338041) $ opatch apply Oracle Interim Patch Installer version 12.2.0.1.9 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /u01/opt/oracle/product/12.1/db_1 Central Inventory : /u01/opt/oraInventory from : /u01/opt/oracle/product/12.1/db_1/oraInst.loc OPatch version : 12.2.0.1.9 OUI version : 12.1.0.2.0 Log file location : /u01/opt/oracle/product/12.1/db_1/cfgtoollogs/opatch/opatch2018-07-18_19-06-58PM_1.log Verifying environment and performing prerequisite checks... NApply could not load patch from location '/media/sf_stage/28317232/27547329/27547329' UtilSession failed: java.lang.RuntimeException: /media/sf_stage/28317232/27547329/27547329/etc/config/actions.xml with Version field of the component "delete" in actions file cannot be or empty. Please check patch metadata. Log file location: /u01/opt/oracle/product/12.1/db_1/cfgtoollogs/opatch/opatch2018-07-18_19-06-58PM_1.log OPatch failed with error code 73
Nothing wrong here except for two things:
- It’s better to use the Bundle Patch instead of the Patch Set Update for Oracle 12.1.0.2
- OPatch fails with Error 73 – and it looks like as if the shared folder couldn’t be accessed
The fact which caught my attention: Oracle Interim Patch Installer version 12.2.0.1.9
Didn’t I learn the other day:
- opatch 12.2.0.1.12 is the minimum for Oracle 12.1.0.2 July 2018 patch bundles?
- opatch 12.2.0.1.14 (the one I used) needs to be present now in the home I’m patching?
Hence I quickly dropped Peter a message, he refreshed his opatch – and all went fine. Not a directory permission issue but simply an too old opatch.
But this is not the end of the story.
When you use the Patch Download Assistant to access the 12.1.0.2 July Patch Bundles:
you’ll see the PSU and the BP.
But actually there are different README which is not an issue and expected. But in this case the opatch requirements are different as well which is VERY unlikely to be the case:
See the difference? Using the most recent opatch 12.2.0.1.14 instead of 12.2.0.1.9 (and of course 12.1.0.1.4 as mentioned in the readme) fixed the issue.
In order to prevent similar issues, please read: Why should you use the most recent version of OPatch?
Addition on July 23, 2018 – thanks to Martin Berger:
"In 28317214/27967747/27547329/27547329/etc/config/inventory.xml there is a line: <minimum_opatch_version>12.2.0.1.13</minimum_opatch_version> I’m curious why OPatch 12.2.0.1.9 didn’t simply told Peter it’s not capable to do what he asked? Do you have any idea?"
No, I don’t know. But thanks for debugging this, Martin!
Delays …
And even though I wrote about this above already, please have a look into the corresponding MOS notes when patch bundles get released. I had several questions in the past two days about a patch not there. So please have a close look at the Section 2.2 of Patch Availability for the Oracle Database (MOS Note: 2394520.1). It shows you specific bundles not available yet and the expected release date:
There are multiple reasons for a delay. In some cases a late regression got found during testing (yes, we run a lot of regression tests with the patch bundles) and needs to be fixed.
–Mike
Agree,
Sometimes the notes are confusing, mainly with opatch versions and round robin links.
I prefer to follow the Master Note for Database Proactive Patch Program (Doc ID 756671.1), appears to be better (and less noising than note that you pointed).
By the way, in my internal test machine:
SQL> select action_time, action, comments from sys.registry$history;
ACTION_TIME ACTION COMMENTS
—————————— —————————— ————————————————–
BOOTSTRAP RDBMS_18.3.0.0.0DBRU_LINUX.X64_180627
20-JUL-18 07.06.18.791228 AM RU_APPLY Patch applied from 18.1.0.0.0 to 18.3.0.0.0
SQL>
Best regards.
Fernando Simon
when a bug starts, logic stops. Thanks for sorting this out
🙂
Mike,
regarding Peters issue,
in 28317214/27967747/27547329/27547329/etc/config/inventory.xml
there is a line
12.2.0.1.13
I’m curious why OPatch 12.2.0.1.9 didn’t simply told Peter it’s not capable to do what he asked?
Do you have any idea?
Martin
Martin,
thanks – and awesome work!!!
But I have no idea why opatch didn’t tell Peter this.
If you don’t mind I will update the blog post!
Cheers – and THANKS!!!
Mike
Hi Mike,
I downloaded the July 2018 patch and the newest OPatch (v14) to AIX . The OPatch is not really working there (comparing to my v8 version) on AIX.
[itsopstest2:12EE8:/oracle/product/12.1.0/dbee_8/jdk/bin] $ORACLE_HOME/OPatch/opatch version
Error: This Java instance does not support a 64-bit JVM.
Please install the desired version.
OPatch failed with error code 1
Jennifer
Jennifer,
please please please … log an SR for such things.
As much as I’m willing to help and ease the OPatch pain, Support (and the people who are taking care on OPatch) need to get your feedback as a customer. If you don’t get a solution quickly, escalate the SR. Make sure the severity is at least 2, if not 1 (not 24×7) as the July patch contains important fixes you need to apply. Ask for a management callback when you don’t get a rapid solution.
Unfortunately I have no AIX system to test or verify anything. But Support has.
Does:
./java -version -d64
lead to the same error?
Then you may miss the 64 bit JDK:
https://developer.ibm.com/javasdk/support/aix-download-service/
But I’m neither a JDK nor OPatch expert …
Thanks,
Mike