One and a half weeks ago we released the Database and Grid Infrastructure Release Updates (RU) 19.18.0. You could read on the blog how to apply it, and you could read also about an alert. So far, all was more or less transparent but apparently sort of an Hide and Seek with RU 19.18.0 has happened. So let me shed some light.
I am just the messenger trying to bring some light into the topic. Neither is our team responsible nor do we have any sort of control on this process. I write this upfront since my team has received some really nasty and angry comments from customers, ACEs and also colleagues. Don’t shot the messenger please!
Upload, Alert, Removal …
When Database and Grid Infrastructure 19.18.0 Release Update (RU) got uploaded, it all looked fine. We made it available on Linux at first, so everybody being on another platform is not affected by this anyway. Interestingly enough, there was an incredibly high download count of this RU within the first days. This makes me think very positive since it tells me that you download the RUs right when we release them. Great news.
But it turned out that the RUs have a flaw which affects RAC customers only. And since the RU got released on Linux only, it were RAC-Linux customers only. Still, this number is not small. The issue is described in Important Alert with RU 19.18.0. As a consequence, and to fix the issue, both 19.18.0 RUs for the Database and for Grid Infrastructure got removed.
Bad side effect:
The MyOracle Support portal (MOS) is not very user-friendly and only told you “Patch does not exist“. No further explanation, no information, nothing.
This is when the first emails and comments reached me asking why the patch bundles vanished. Even though I just had explained how to apply the January 2023 patch bundles.
Thanks to all of you being faster than I, Peter Lehmann in this case was the first to point me to the alert. It dawned me that this was the reason for the RUs being removed which they obviously were.
Some hours later I understood the issue and reason why this has happened. And I did update my original blog post with some additional information and the tentative re-upload date of Jan 31, 2023.
Upload, Removal, Upload again …
And so it happened.
The RUs appeared again, the first customers commented that the RUs are available again.
Until they disappeared. Again.
And certainly without any information or comment.
Of course, they did not disappear completely at first, but the option to download disappeared. So the button marked in red had no function. And since you could not choose the platform, there was no download possible.
The bad situation:
Some people had used the first drop of the RU, some had downloaded the 2nd drop – and some were still waiting since it had vanished again. At this stage, the first people commented about wrong inventory information with the 2nd drop.
- Previous one:
34765931;Database Release Update : 22.214.171.124.230117 (34765931)
- New (intermediate) one:
34765931;DATABASE RELEASE UPDATE : 126.96.36.199.0 (REL-JAN230131) (34765931)
The zero in the 5th number of the release looked strange. And I’m always amazed how many tech experts we have out there. Several customers noted this right away. And complained to me.
Well, it turned out that you weren’t the only ones wondering and complaining.
This 5th number broke the provisioning of 19.18.0 in our cloud tooling as well since it expects the date as above “230117” instead of a plain “0“. Making a long story short, the RU had to be removed and re-uploaded again, now with a new entry into the inventory which most likely is:
- 34765931;DATABASE RELEASE UPDATE : 188.8.131.52.230117 (REL-JAN230131) (34765931)
I haven’t verified it but this is what I’d been told.
An issue with the 19.18.0 RUs affecting only RAC customers on Linux made it necessary to repacked both, the Database and the Grid Infrastructure Release Updates. But unfortunately, and issue with the inventory identification of the reuploaded RUs happened, so another removal and upload was necessary.
Again, I am just the messenger here trying to explain what has happened.
And sorry for all the inconvenience and lack of transparency.
By now, the COMBO patches should have been reuploaded again, too.
Further Links and Information
“Ordre, contreordre, desordre.” Napoleon Bonaparte
Yep – was true a long time ago already, may still be true today.
Thanks, so I guess I downloaded again when it first reappeared. Is the “version” (.0) the only thing that was fixed? I installed everything with “opatchauto” and it looks like everything is patched and works. Of course, “lsinventory” shows the “.0”. That is just cosmetic, yes? Do I need to download and patch again?
Patch 34765931 : applied on Wed Feb 01 07:15:26 CST 2023
Unique Patch ID: 25093776
Patch description: “DATABASE RELEASE UPDATE : 184.108.40.206.0 (REL-JAN230131) (34765931)”
Created on 27 Jan 2023, 11:25:14 hrs UTC
35001966, 34958012, 32509606, 32790012, 31326998, 31466433, 28873575
32298004, 30374739, 29249289, 29920804, 31559415, 33050626, 31750581
33598840, 28709063, 31200057, 30045242, 32319410, 30045983, 29487189
31760738, 31163498, 31254929, 25979242, 33425046, 31895670, 29997553
29436522, 27907898, 32587994, 33181881, 30081055, 31661586, 31952046
exactly – this is that you easily can live with but what caused trouble in the cloud provisioning.
No need to change anything.
I deployed the intermediate release to an on-prem database. I won’t lose any sleep over this as long as the next (April) RU will be able to deploy over it despite the bad patch description.
Good choice, Doug!!
And I just wanted to explain for those who were quite disappointed.
OK. We downloaded and applied the first version. Now how to upgrade to the latest and greatest 19.18? 🙂
see in the MOS alert – you need the one off patch, then it is supposed to be the exact same.
Please push for the feature of having _all_ released files in the ARU JSON, once withdrawn obviously without the download link which wouldn’t work anyway but rather with explanation that “this patch has been withdrawn on DATE”, or yet better with a JSON field with the timestamp.
since we work in the same company, feel free to mail the owners directly 🙂
I downloaded and applied the original to non-RAC. ok to use or should I download latest.
Absolutely ok to use – you won’t see any issues.
Is this also affecting the availability of 19.18RU bundle for AIX? It’s February and still I have been able to find only the OJVM bundle.
MOS Note: 2906899.1 – Critical Patch Update (CPU) Program Jan 2023 Patch Availability Document (DB-only
Goto to section 2.2 (Post Release Patches) and check what AIX tells you. I have unfortunately no insider information nor more than what’s mentioned in this note.
Thanks Mike, the AIX db patch was released yesterday.
I need help.
Last patching 19.18 via autoupgrade fails on all databases – UPG-1400:
Patch 34765931 apply: WITH ERRORS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/34765931/25098466/34765931_apply_ABINAUTH_2023Feb08_20_25_15.log (errors)
-> Error at line 47025: script rdbms/admin/backport_files/bug_33527739_apply.sql
– ORA-13516: AWR Operation failed: CATPROC not valid
– ORA-06512: at “SYS.DBMS_WORKLOAD_REPOSITORY”, line 328
– ORA-06512: at “SYS.DBMS_WORKLOAD_REPOSITORY”, line 355
– ORA-06512: at line 12
Patch 34786990 apply: WITH ERRORS (PREV PATCH)
I restarted database in normal mode and resume job.
But there is a lot of manual work.
There is other solution for this ?
this is a problem with an AWR change in 19.18.0 which magically expects the database to be NOT in upgrade/restricted mode (which it was in this case).
Problem is that it breaks the upgrade. We are chasing the AWR team already.
Would you please mind to open an SR and have Support analyze and tag this? Please share the SR number with me via the blog then.
just for your and other DBA info:
after I patched our RAC-testenv with RU18 last week I saw this in MOS dashboard:
Bug 34885986 – Flashback log file was not reused even if db_flashback_retention_target is passed (Doc ID 34885986.8)
which is affected the 19.18.
Just checked – As of now I doesn’t see any growup of occupied space in recovery area.
The instances runs since 6 days. Hopefully this remains. But I’m a little bit concerned to patch the produktive env. RU18 doesnt’ seem to make us happy.
thanks for the hint. And I was not aware of this issue. It is supposed to be fixed in the coming RU. It happens only in a RAC environment, and the workaround 1 would be to turn FB off and again ON afterwards. But this will only clean out the consumed space and won’t prevent the problem. Workaround 2 would be to delete those FB logs not needed anymore (this is a bit more complex to find out).
I hope that there will be a MOS note soon explaining this in more detail.
Cheers, and again, thanks for the hint!
Is everything already fixed with 19.18 RU for RAC systems as of today? Is it safe to go with it now?
Of course, everything is fixed by now 🙂
There may be only a few tiny issues here and there … and maybe a few others there and here too.
yesterday I opened a SR with the following question:
It’s now 4 weeks since the release date of january 2023 cpu and Patch 34765931 (DATABASE RELEASE UPDATE 220.127.116.11.0) and Patch 34777391 (Latest JDK Patch 19c) for 32bit Linux are still missing.
Doc ID 2584628.1 (JDK and PERL Patches for Oracle Database Home and Grid Home) says for Patch 34777391 (Latest JDK Patch 19c) a release date 31-Jan-2023.
When will these 2 patches finally be available ?
Until now I have not received any reply. Do you have any information about this?
sorry, this took a bit longer.
That is done on a per-request basis as you can see below.
It is listed on the same CPU Patch Availability Document Section 1.13 (Also posted below) that you posted to explain what “On Request Platforms” means.
18.104.22.168 Oracle Database 19
32-bit client-only platforms
1.3 On-Request Patches
Oracle does not proactively release patches for historically inactive platforms. However, Oracle will deliver these patches when requested.
The following guidelines describe how to initiate an on-request (OR) patch.
A request may be made:
At any time. However, a patch for a specific quarterly release, such as CPUOct2019, cannot be requested. Depending on when the request is received and processed, either the patch for the current quarterly release or the next quarterly release will be provided. Your Service Request (SR) will provide you the planned availability date for the patch.
As long as the version is in either Premier Support or Extended Support and error correction support has not expired. For example, if a product release is under Extended Support through the release of CPUJan2020 on January 15, 2020, then you can file a request for the product release through January 29, 2020. For more information, see Oracle Lifetime Support Policies at http://www.oracle.com/us/support/lifetime-support/index.html, and Note 209768.1, Database, FMW, Enterprise Manager, TimesTen In-Memory Database, and OCS Software Error Correction Support Policy.
For a platform-version combination when a major release or patch set is released on a platform after a quarterly release date. Oracle will provide the next patch for that platform-version combination, however you may request the current patch by following the on-request process. For example, if a patch is released for a platform on August 1, 2020, Oracle will provide the CPUOct2020 patch for that platform. You may request a CPUOct2020 patch for the platform, and Oracle will review the request and determine whether to provide CPUJul2020 or CPUOct2020.
A patch that is marked as on-request (OR) may already have been requested by another customer and be available on My Oracle Support. Before you file a Service Request (SR), check on My Oracle Support to see if the patch is already available for your platform.
hi, mike, how to get the GI patch of Windows platform 19c RAC? I tried the Bundle Patch of 19.14 and 19.18, and found that there is no GI patch in it, so how to patch the Grid?
See here, Thomas:
MOS Note: 2906899.1 – Critical Patch Update (CPU) Program Jan 2023 Patch Availability Document (DB-only).
see section “Post Release Patches”.
To me it looks as if it is available.
Problems with dbms_metadata.get_ddl after 19.18 DB-RU patch. Before 19.18 I could extract out the create user syntax including the encrypted password. This is used to reset passwords after database refreshes from Prod to lower environments.
After the 19.18 patch I still get the create user statement, but it no longer displays the encrypted password.
Please delete this comment if it isn’t acceptable here. Thanks!
Please disregard my previous post. It isn’t the DB-RU that caused the issue but with the Datapump Bundle Patch 34972375 that is changing the dbms_metadata.get_ddl behavior. I will research this further as this is the first time I’m applying the DPBP patch. Sorry about that.
this is correct – and I have a blog post in the making since you are not the first customer being trapped by this.
It is a security fix which seemed to strike too far. We have escalated this within the company to our security architect but we wait for a reply now since 3 weeks.
If you don’t mind, I would find it very helpful if you’d open an SR and send me the SR number. This allows us to put some food into our internal communication.
And regarding your first message, there is absolutely nothing wrong with asking such topics here on the blog. I’m just a bit “behind” with answering the comments.
when you open an SR, you can reference this bug number: 35018026
BUG 35018026 – DBMS_METADATA.GET_DDL(‘USER’,’USERNAME’) DOES NOT RETURN HASHED PASSWORD
SR opened and email sent to you with the number. Thanks for the helpful reference on the bug.
I still did not receive an email with your SR.
Can you please paste it here (if you want, I can delete the comment afterwards).
Here’s the SR: SR 3-32263011xxx
Our customer has 19c SEHA (Oracle Standard Edition High Availability) with RU 19.16 on Linux.
Is it really required to apply the latest GI RU 19.18?
Does GI RU 19.18 has the previous issue resolved?
GI 19.18.0 has potentially 200 fixes on to of 19.16.0 as far as I am aware.
Ok and Thanks a lot Mike,