July 2018 Update, Revision, BPs, PSUs – delays and other issues

This is a Friday morning blog post. And actually I had no intention to write it. I don’t mean it offensive. But yesterday I received an interesting email from Frits Hoogland from Enkitec/Accenture. Frits asked me about the July RUR and some unlogic things on MOS. Once I had answered his email, the next question came in. Hence, I thought I need to write something about our July 2018 Update, Revision, BPs, PSUs – delays and other issues.

From being on stage at last year’s DOAG Conference for a “Patching Forum” I know that things aren’t as obvious as we often think they are. And especially not when it gets into patching. This starts usually with acronyms which are truly familiar when you work in the patching area at Oracle. But they aren’t self-explanatory to customers who patch only once or twice a year. But this blog post is not meant to explain acronyms.

One important thing upfront: I’m not the Product Manager for Patching. I just patch and blog about it.

July 2018 Update, Revision, BPs, PSUs – delays and other issues

The July 2018 patch bundles are out for less than 48 hours, but I think I received 10 comments or emails already. So I’d like to explain and summarize some things I learned in the past 48 hours about our July patch bundles.

Frits Hoogland asking about logic

Frits Hoogland, who is a great guy and a very well known standout Oracle database internals expert asked me about an issue with the July 2018 Revision (RUR):

“[..] the link in the PSU note reports back that the patch can’t be found.
Is there any way this is logical?”

It took me a few minutes to retrace the “logic” Frits struggled with.

Frits wanted to download Patch 28346593 – Database Jul 2018 Release Update Revision (DB RUR) 18.2.1.0.180717, in brief the July 2018 Oracle 18c RUR 18.2.1. It is not important that I and others state they you should keep your hands away from RURs unless Support advises you to install them. We produce them and hence you can install them. In theory.

The above bug/patch note has an interesting section:

I marked the interesting part in BLUE. It points to a fix which supposingly superseeds the RUR from July 2018.

But how can this be when the patch is (in theory) out just for a day? And how can the patch number be significantly lower (27676517) than the one for the RUR (28346593)? It starts getting even more interesting when you click on the link for 27676517 as you’ll get another fix which superseeds this one:

But this is the April 2018 Release Update (RU). How can the April 2018 RU superseed the July 2018 RUR?

And wait a bit? Isn’t this the one Frits originally tried to access: 28346593 ? Yes it is. We are in a loop now.

I don’t know the logic behind this. And I can’t speak for the person who updated the notes. It could be even an automation process. But the solution is here and I mentioned it already in one of my previous blog posts about the July 2018 patch bundles:

Section 2.2 of Patch Availability for the Oracle Database (MOS Note: 2394520.1) has the details about delayed patch bundles. And here it says that the Oracle 18c RUR 18.2.1 is delay to an ETA of July 24, 2018:

Why didn’t the above Patch 28346593 – Database Jul 2018 Release Update Revision (DB RUR) 18.2.1.0.180717 mention it but instead has this sentence:

This is a marker bug for the 18.2.1.0.180717 (Jul 2018) Database Release Update Revision(DB RUR).
This Release Update can be downloaded here: Patch:28346593

I don’t know.

Therefore, sorry Frits for the inconvenience and the wasted searching time 🙁 And yes, it’s not logic.

The wrong README and the wrong OPatch

Just 15 minutes after Frits’ email there was an internal email from my colleague Peter:

“Trying to patch my single instance 12.1.0.2.180417 with 12.1.0.2.180717, but this time it fails. “

Peter had done some of the usual things already. But even downloading it again led to no other result:

$ opatch lspatches
27475603;Database PSU 12.1.0.2.180417, Oracle JavaVM Component (APR2018)
27338041;Database Patch Set Update : 12.1.0.2.180417 (27338041)


$ opatch apply
Oracle Interim Patch Installer version 12.2.0.1.9
Copyright (c) 2018, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/opt/oracle/product/12.1/db_1
Central Inventory : /u01/opt/oraInventory
   from           : /u01/opt/oracle/product/12.1/db_1/oraInst.loc
OPatch version    : 12.2.0.1.9
OUI version       : 12.1.0.2.0
Log file location : /u01/opt/oracle/product/12.1/db_1/cfgtoollogs/opatch/opatch2018-07-18_19-06-58PM_1.log

Verifying environment and performing prerequisite checks...
NApply could not load patch from location '/media/sf_stage/28317232/27547329/27547329'
UtilSession failed: java.lang.RuntimeException: /media/sf_stage/28317232/27547329/27547329/etc/config/actions.xml with Version field of the component "delete" in actions file cannot be  or empty. Please check patch metadata.
Log file location: /u01/opt/oracle/product/12.1/db_1/cfgtoollogs/opatch/opatch2018-07-18_19-06-58PM_1.log

OPatch failed with error code 73

Nothing wrong here except for two things:

1. It’s better to use the Bundle Patch instead of the Patch Set Update for Oracle 12.1.0.2
2. OPatch fails with Error 73 – and it looks like as if the shared folder couldn’t be accessed

The fact which caught my attention: Oracle Interim Patch Installer version 12.2.0.1.9

Didn’t I learn the other day:

1. opatch 12.2.0.1.12 is the minimum for Oracle 12.1.0.2 July 2018 patch bundles?
2. opatch 12.2.0.1.14 (the one I used) needs to be present now in the home I’m patching?

Hence I quickly dropped Peter a message, he refreshed his opatch – and all went fine. Not a directory permission issue but simply an too old opatch.

But this is not the end of the story.

When you use the Patch Download Assistant to access the 12.1.0.2 July Patch Bundles:

you’ll see the PSU and the BP.

But actually there are different README which is not an issue and expected. But in this case the opatch requirements are different as well which is VERY unlikely to be the case:

• PSU July 2018 – 12.1.0.2 – README
  
• BP July 2018 – 12.1.0.2 – README
  

See the difference? Using the most recent opatch 12.2.0.1.14 instead of 12.2.0.1.9 (and of course 12.1.0.1.4 as mentioned in the readme) fixed the issue.

In order to prevent similar issues, please read: Why should you use the most recent version of OPatch? 

Addition on July 23, 2018 – thanks to Martin Berger:

"In 28317214/27967747/27547329/27547329/etc/config/inventory.xml there is a line:

 <minimum_opatch_version>12.2.0.1.13</minimum_opatch_version>

I’m curious why OPatch 12.2.0.1.9 didn’t simply told Peter it’s not capable to do what he asked?
Do you have any idea?"

No, I don’t know. But thanks for debugging this, Martin!

Delays …

And even though I wrote about this above already, please have a look into the corresponding MOS notes when patch bundles get released. I had several questions in the past two days about a patch not there. So please have a close look at the Section 2.2 of Patch Availability for the Oracle Database (MOS Note: 2394520.1). It shows you specific bundles not available yet and the expected release date:

There are multiple reasons for a delay. In some cases a late regression got found during testing (yes, we run a lot of regression tests with the patch bundles) and needs to be fixed.

–Mike