Actually this is not an alerting blog post but just a hint. At DOAG conference in Nürnberg several customers came by after our patching talk on Thursday in Tokio and complained that the Patch list in MOS Note: 555.1 is not conflict-free, and thus hard to use. Let me see if I can help here.
What does MOS Note: 555.1 provide?
I have a love-and-hate relationship with this MOS Note: 555.1 – Oracle Database 19c Important Recommended One-off Patches. It lists the issues on-top of a given Release Update (RU). This is why I love it, it helps you and me to find out what important issues on-top of an RU I should be aware of. And of course, nobody wants to learn about issues just when you moved to a new RU. Hence, please check MOS Note: 555.1 beforehand.
Now, when you look closer, the authors provide a super-helpful feature in this MOS note. They list the patches as a comma-separated list as well making life easier:
Example taken for Oracle 19.24.0 from MOS Note: 555.1
Is this list conflict-free?
You can guess the answer already from the title of this blog post. This list isn’t conflict-free. I used the Conflict Checker Tool on MOS: MOS Note: 1091294.1 – How to Use the My Oracle Support Conflict Checker Tool for Patches Installed with OPatch and received this result when I passed on my 19.24.0 inventory to it:
See MOS Note: 1091294.1 on how to use the Conflict Checker Tool in MOS
You see 3 pairs of patches conflicting to each other.
How did I come across this?
Christian Pfundtner, an Oracle ACE Director from Austria gave a training and praised the capabilities of AutoUpgrade patching to his audience. He used AU to download and apply 19.24.0 patch bundles, and passed on the recommended patches list from MOS into AU.
global.global_log_dir=/u01/app/oracle/cfgtoollogs/autoupgrade global.keystore=/u01/app/oracle/cfgtoollogs/autoupgrade_keystore upg1.sid=ORCL upg1.source_home=/u01/app/oracle/product/19c/ee_19_18_0 upg1.target_home=/u01/app/oracle/product/19c/ee_19_24_0 upg1.folder=/u01/app/stage/download upg1.patch=RU:19.24,OPATCH,OJVM:19.24,DPBP,36322321,37166484,35197819,35016258,36006910,36908826,35398148,36916250,35663882,34672698,34774667,29213893 upg1.download=YES
But unfortunately, AutoUpgrade will fail to install since it can’t foresee (right now) whether the list of patches conflict. It would only find out if you run the -mode analyze which downloads and analyzes then but doesn’t install anything beforehand.
So, learning no.1: Run java -jar autoupgrade.jar -config yourconf.cfg -mode analyze at first. And then kick off AutoUpgrade with -mode deploy afterwards but disable the download then with upg1.download=no. I know, this does not sound ideal. But as long as we can’t check the download beforehand, I don’t see a better workaround.
In Christian’s case AutoUpgrade got stuck with the first conflicting set:
[Dec 19, 2024 10:28:23 AM] [INFO] The following warnings have occurred during OPatch execution:
[Dec 19, 2024 10:28:23 AM] [INFO] 1) OUI-67301:
Following patches have conflicts: [ 36006910 36322321 ]
Use the MOS Patch Conflict Checker https://support.oracle.com/epmos/faces/PatchConflictCheck to resolve.
See MOS documents 1941934.1 and 1299688.1 for additional information and resolution methods.
[Dec 19, 2024 10:28:23 AM] [INFO] --------------------------------------------------------------------------------
But as you could see from the above checker tool output, there would have been two additional conflicts.
As alternative, you could also trigger a conflict check with opatch before invoking AU:
opatch prereq CheckConflictAgainstOHWithDetail -ph ./
What is the answer to this problem?
Of course, I passed this on to the team in charge of MOS Note: 555.1 . Ideally, the MOS note would already provide a conflict free list with the required merges.
But wait a second, doesn’t this exist already?
What are MRPs (Monthly Recommended Patches) meant for? You can easily and quickly access the most recent MRP via MOS Note: 888.1 – Primary Note for Database Quarterly Release Updates.
Let me check whether all of the above fixes are included in the most recent MRP for 19.24.0 in December 2024 (Patch 37331707). The README of the MRP tells me that the following fixes are included:
- December 2024 MRP for 19.24.0
- DB MLR 37350603 (37166484, 37104910, 36916250), DB MLR 37360545 (37084037, 35398148), 37234122, 37197141, 37149719, 36736231, 36877696
- November 2024 MRP for 19.24.0
- 35197819, 36006910, DB MLR 37242064 (36982817, 37010561), DB MLR 37244817 (37156413, 37127615)
- October 2024 MRP for 19.24.0
- 36927520, 37048709, 37010561, DB MLR 36961299(36908826, 35663882), 36943142, 36988585, 35398148, 36126983, 36611700, 37081891, 36745665
- September 2024 MRP for 19.24.0
- 35016258, 36916250, 36544236
- August 2024 MRP for 19.24.0
- 35663882
Well, this is not as simple to digest as I’ve had hoped for. But it obviously seems to include MLRs (Merge Label Request – in short: Merges of existing fixes which would conflict) already. And you can use MOS Note: 555.1 to check quickly which of those fixes are part of the MRP.
MOS 555.1 | MRP Dec for 19.24 ================================== 36322321 | not included 37166484 | included with MLR 37350603 Dec MRP 35197819 | included with Nov MRP 35016258 | included with Sep MRP 36006910 | included with Nov MRP 36908826 | included with Oct MRP 35398148 | included with MLR 37360545 Dec MRP 36916250 | included with MLR 37350603 Dec MRP 35663882 | included with MLR 36961299 Oct MRP 34672698 | not included 34774667 | not included 29213893 | not included
Well, this really doesn’t solve my problem at first sight. 4 of the 12 fixes listed on-top of RU 19.24.0 are not included in an MRP. But at least 8 of them are since MRPs are cumulative within one RU stream.
Those are the 4 fixes not included for various reasons, for instance the last two ones for not being RAC-rolling: 36322321, 34672698, 34774667, 29213893.
Unfortunately, a quick check with the Conflict Checker Tool output from above tells me that – even if I’d use the MRP from December 2024 – I would have one conflict: 36006910 and 36322321 conflict with each other (the middle-pair in the above screenshot).
In addition, MRPs are available only on Linux but not on other platforms such as Windows, AIX or Solaris.
Therefore, your only option is to run your patch list together with the inventory through the My Oracle Support Conflict Checker Tool for Patches and request the required merge patches upfront. This way, you can then install safely, with or without AutoUpgrade. But let me emphasize here that this isn’t an AutoUpgrade issue but rather an underlying root cause.
Further Links and Information
- MOS Note: 555.1 – Oracle Database 19c Important Recommended One-off Patches
- MOS Note: 888.1 – Primary Note for Database Quarterly Release Updates
- MOS Note: 1091294.1 – How to Use the My Oracle Support Conflict Checker Tool for Patches Installed with OPatch
- AutoUpgrade Patching – The Feature You Waited For
- Christian Pfundtner – Oracle ACE Director
–Mike
It is true doc id 555.1 is not patch conflict free, the 555.1 is vitally important.
I think may lack the warning which one-offs not included in the alternative MRP may conflict among the one-offs or the MRP subpatches. For me that would better than not having some one off not listed.
Than their is the RU readme recommended patches just to help with the conflicting issue and beyond 555.1.
I am a user of this 555.1 and the RU readmes a lot. My advice is
a) MRP over One-offs for starters.
b) Test for conflicts is always required as long as multiple patches are listed and this does not change.
c) When conflict is detected, we have to think, can we wait another month for next MRP another list? or open a SR by which a merge may came late or sooner? Can we go with only MRP or do we only apply MRP subpatches and some one-offs ?
These are things the only way to be solved is not having 555.1 with none tested one-offs with MRP
I prefer having the one-offs+MRP and the info which may conflict and identity those tested with MRP.
Like your suggestions, Luis!
Cheers,
Mike
“opatch prereq CheckConflictAgainstOHWithDetail -ph ./” has – in the Oracle 11.2ish/12.1ish past, not always been been reliable in my perception.
Our approach ist to build our goldimages with qualified input from 555.1 as well the support hot alerts: that way we get to find any (conflict) issues easily.
Also – since HP/UX-Itanium and AIX/Power are still our main platforms – we are usually at least one RU behind since for those platforms the latest RU is often only available 1-2 months after the release date of the Linux-x86-64 release … that goes for the 555.1 patches quite often, too.
And then we have to (let the app teams) test through the deployment pipelines … which takes up some time to.
All being said, 555.1 is great but we see a lot of potential for improvement 😜
Hi Christian,
I am in discussion with the team owning 555.1 about improvements.
Cheers,
Mike
Hi Mike,
I can’t find the conflict checker tool anymore in the new Oracle Support portal. Do you know if it does still exists and if so, where?
BR
Jörg
Hi Joerg,
I wish I’d know. I found KB135057 which has links – but for me, those go into the login page into an endless loop.
Did you open an SR to check whether Support knows??
Cheers,
MIke
Hi Mike
Same with we. But in the meantime I have found it. You have to go to the patches and certification matrix, search for a patch, select a patch and then you can see a button “Analyze with OPatch” in the upper part of the page.
Tricky…
Greetings
Jörg
Thanks Joerg!!
Cheers,
Mike