Well, there is no Product Manager for patching. And I’m glad that I’m not in this position. But thanks to all of you, internally and externally, who share your findings with me. In this blog post I’d like to inform you about the newest patching surprises – and how to fix some of them.
Patch apply without confirmation?
Peter Lehmann informed me about this a few weeks ago. Peter told me that he patched systems with the July 2019 RU for Oracle 188.8.131.52 and recognized that the patch installed with opatch without waiting for confirmation. It looked as somebody has silently switched it to silent install.
In between, Peter pointed me to this discussion on the My Support Community Forum:
- https://community.oracle.com/message/15413552#15413552With the Oracle 18.7.0 RU the same thing seems to happen:
“During apply of this patch, I did notice:
$ opatch apply Oracle Interim Patch Installer version 184.108.40.206.17 (...)
Do you want to proceed? [y|n] Y (auto-answered by -silent)
User Responded with: Y
All checks passed.
I did not enter the -silent option”
On behalf of Peter, I filed
Bug# 30171495 - 220.127.116.11 JULY 2019 DB BP PATCH 29496791 INSTALLS DIRECTLY WITHOUT WAITING FOR USER CONFIRMATION not knowing at this point that this has nothing to do with the RU itself.
It took a while until a solution (or to be more precise: explanation) was given – but I was a bit surprised as this hasn’t been documented. It will be documented soon I guess.
If OPatch detects the OOP mode (patching busy files), then it comes to silent mode. This is a new feature from OPatch. Before this, OPatch requires all processes to be down before it could continue. That was the reason why OPatch did always ask for confirmation first.
Now with this new feature, OPatch can handle the patch apply even when the database processes are up and running. OPatch can patch busy files. From now on, users don’t need to worry about shutting down the processes before patching – this prompt is no longer needed.
Hence, actually this is a cool new feature. And it turns out that this feature is enabled already in opatch versions 16 and 17.
And of course, nothing needs to be changed or fixed.
Incorrect Patch Level Metadata?
The next topic got sent to me by Thomas Gorsitzke, one of our very experienced Principal Solution Engineers in Oracle Germany (thanks for sharing, Thomas!). And this issue can happen to everybody who applied one of the following patch bundles:
- Oracle Database 18.104.22.168.0 DBRU
- Oracle Database 22.214.171.124.0 DBRUR
- Oracle Database 126.96.36.199.0 DBRU
19.3.0 and 19.3.1 should potentially happen only rarely in some Exadata environments. But I guess, many applied 19.4.0 already. As the patch apply could cause incorrect metadata, either further on-off patch or rollback attempts may fail.
You may see this message when you try to apply another patch on top – or rollback it:
"Skip patch <bug#> from list of patches to apply: This patch is not needed.”
What you need to do in case you have installed 19.3.0, 19.3.1 and/or 19.4.0:
You must download the script
check_patch_level.sh from this MOS Note:
- MOS Note: 2575299.1 – Potential Impact to Database 19c One-off patches Due to Incorrect Patch Level Metadata
If it identifies any faulty patch, you must rollback them, download them again from MyOracle Support and reapply them.
And please, don’t shoot the messenger!
We’ had an issue with July 2019 patch that I don’t think I’ve ever noticed before– for our databases that had invalid objects (like a package that couldn’t be compiled or an invalid DB Link), the datapatch returned errors until all objects were valid. Was this something new with this patch? Or was I lucky with previous DBs only having valid objects (which seems unlikely!)
can you share more information with me? Which objects especially – and the datapatch log would be ideal.
PS: Yes, I think there was something added and removed now again – but I need to check with my datapatch mates.
thank you for the quick reply!
This is the error we got for the DB with an invalid DB Link- once I updated the TNS entry, the link was valid and the datapatch completed successfully.:
Patch 29757449 apply: WITH ERRORS
logfile: /opt/oracle/cfgtoollogs/sqlpatch/29757449/23013437/29757449_apply_MSAQAS_2019Sep04_13_43_40.log (errors)
Error at line 12700: ORA-04045: errors during recompilation/revalidation of
Error at line 12702: ORA-04052: error occurred when looking up remote object
Error at line 12704: ORA-00604: error occurred at recursive SQL level 2
Error at line 12705: ORA-12541: TNS:no listener
On another database, we had an invalid package (one of our own) that couldn’t compile because the table definition had changed (missing column). We dropped the bad package and datapatch completed without errors:
Patch 29757449 apply: WITH ERRORS
logfile: /opt/oracle/cfgtoollogs/sqlpatch/29757449/23013437/29757449_apply_APEXTST_2019Sep04_23_55_57.log (errors)
Error at line 515: Warning: Package Body created with compilation errors.
If you think it’s worthwhile, I can open a ticket with Oracle support for them to look it…we’ve worked around all of the issues, but it just seems odd that the patch would care about *our* objects being invalid. 🙂
at least the bad package issue should be fixed – I read about this internally. And datapatch had an issue there.
See this note here:
Oracle Support Document 2569427.1 (ORA-04045 ORA-04042 ORA-00604 Error After Applying July 2019 RU Patches) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=2569427.1
I think this may fix the first issue as well.
I had noticed this auto confirmation in opatch even last year. It was kind of surprising as I was patching multiple RAC nodes, one at a time. I would shutdown all Oracle processes on the node to be patched. On the first node, opatch would always ask for conformation. On the second node, it would patch without confirmation. I did notice the OOP message, but could not find much on why it was happening, so I filed it under the Oracle unsolved mysteries. It was not worth wasting time with support. But, I still don’t understand the explanation. I bring down all Oracle processes on the node to be patched. So, why does opatch asks for confirmation on first node, but not on the second one? Hopefully, there is an explanation somewhere.
I don’t know – and I wait for some explanations, too. The documentation is not very helpful in this case.
Sorry to say that, but only an SR may shed some light.
Regarding the 1st issue, is the database considered patch after this or only after bouncing it?
Also, I remember that some patches in the past allowed that (don’t remember the name of the feature) but it was only temporary until you could stop the database and properly install the patch. This is not the case here, right?
I have exactly the same question you have.
And so far, nobody wanted to reply with a clear explanation.
I’ll keep you updated because I’m very curios, too.
Thanks for sharing. Latest Opatch can apply patches online only to the 12cR1 and 12cR2 version or it’s applicable to 11gR2 also. If there any recommendation to restart the database after a certain time as we got earlier when we apply some online patches.
Also, if the post install of JVM can be done online we can do patching without any downtime.
I think this should be obvious but I haven’t pointed this out explicitly.
Only the 16 and 17 versions of opatch can do this – and the versioning and release for 11g is different according to the MOS download.
Re Patch apply with out confirmation… is this Oracle moving to zero downtime patching?
I have (as others) some open questions here. I will update as soon as possible …
Thanks for these warnings, Mike! Minor note, the script check_patch_level.sh needs dos2unix run on it (has windows line endings). As of today when downloaded from MOS and never opened in an editor on my end.
Oh lovely – thanks for letting me know.
I will alert the MOS note owner. No idea why people do still such simple mistakes 🙁
Thanks and sorry for the inconvenience!
I’m currently testing the new online Patching feature of OPatch.
But it does’nt work for us. I tried to apply October Patchset on a 12.1 database using opatch version 188.8.131.52.18 but no success.
Prerequisite check “CheckActiveFilesAndExecutables” failed.
The details are:
Following active executables are not used by opatch process :
the same issue on 18c tried opatch version 17 and 18
Is this realy a new feature or a bug ?
thanks a lot
thanks a lot for this feedback. Actually the information I received from the owners of opatch was:
“The problem seems to be an issue with opatch 184.108.40.206.17 which does not happen anymore with opatch 220.127.116.11.18/19.”
But at first, my question would be:
Does opatch fallback into the original working path – or does the patch apply fail with the error you pasted above?
thank’s for the reply.
I’ve testet Opatch 17 and 18 same issue with both versions.
Opatch is failing with error code 73. To continue patching I’ve to stop all processes and start opatch again.
After restarting Opatch I’ve two different behaviors:
On some nodes opatch is waiting for confirmation and on some other Opatch is running with auto-confirmation.
Meanwhile I’ve opened a Service Request and I got the answer that auto-confirmation is the normal behavior from version 18.104.22.168.17 onwards.
I’m awaiting some more Informations why the behavior is different on some Installations.
Hello, for your Information here’s the answer from Oracle Support.
If the incoming Patch touches one of the following components OPatch,OUI and JDK , then it will be in OOP mode.
OPatch scans the incoming patch content and verifies if it touches above components and accordingly it will run in “-silent –restart”(which is OOP) .
It does not really answer my question about the different behavior at the same db versions.
But OK. It is as it is.
Thanks for the update, Christian!
And unfortunately I have not more details on this.
Plus I think the answer is not really correct as for instance in GI there are specific checks on certain files.