One and a half weeks ago we released the Database and Grid Infrastructure Release Updates (RU) 19.18.0. You could read on the blog how to apply it, and you could read also about an alert. So far, all was more or less transparent but apparently sort of an Hide and Seek with RU 19.18.0 has happened. So let me shed some light.

Photo by Braydon Anderson on Unsplash
Heads Up!
I am just the messenger trying to bring some light into the topic. Neither is our team responsible nor do we have any sort of control on this process. I write this upfront since my team has received some really nasty and angry comments from customers, ACEs and also colleagues. Don’t shot the messenger please!
Upload, Alert, Removal …
When Database and Grid Infrastructure 19.18.0 Release Update (RU) got uploaded, it all looked fine. We made it available on Linux at first, so everybody being on another platform is not affected by this anyway. Interestingly enough, there was an incredibly high download count of this RU within the first days. This makes me think very positive since it tells me that you download the RUs right when we release them. Great news.
But it turned out that the RUs have a flaw which affects RAC customers only. And since the RU got released on Linux only, it were RAC-Linux customers only. Still, this number is not small. The issue is described in Important Alert with RU 19.18.0. As a consequence, and to fix the issue, both 19.18.0 RUs for the Database and for Grid Infrastructure got removed.
Bad side effect:
The MyOracle Support portal (MOS) is not very user-friendly and only told you “Patch does not exist“. No further explanation, no information, nothing.
This is when the first emails and comments reached me asking why the patch bundles vanished. Even though I just had explained how to apply the January 2023 patch bundles.
Thanks to all of you being faster than I, Peter Lehmann in this case was the first to point me to the alert. It dawned me that this was the reason for the RUs being removed which they obviously were.
Some hours later I understood the issue and reason why this has happened. And I did update my original blog post with some additional information and the tentative re-upload date of Jan 31, 2023.
Upload, Removal, Upload again …
And so it happened.
The RUs appeared again, the first customers commented that the RUs are available again.
Until they disappeared. Again.
And certainly without any information or comment.
Of course, they did not disappear completely at first, but the option to download disappeared. So the button marked in red had no function. And since you could not choose the platform, there was no download possible.
The bad situation:
Some people had used the first drop of the RU, some had downloaded the 2nd drop – and some were still waiting since it had vanished again. At this stage, the first people commented about wrong inventory information with the 2nd drop.
- Previous one:
34765931;Database Release Update : 19.18.0.0.230117 (34765931) - New (intermediate) one:
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.0 (REL-JAN230131) (34765931)
The zero in the 5th number of the release looked strange. And I’m always amazed how many tech experts we have out there. Several customers noted this right away. And complained to me.
Well, it turned out that you weren’t the only ones wondering and complaining.
This 5th number broke the provisioning of 19.18.0 in our cloud tooling as well since it expects the date as above “230117” instead of a plain “0“. Making a long story short, the RU had to be removed and re-uploaded again, now with a new entry into the inventory which most likely is:
- 34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)
I haven’t verified it but this is what I’d been told.
Summary
An issue with the 19.18.0 RUs affecting only RAC customers on Linux made it necessary to repacked both, the Database and the Grid Infrastructure Release Updates. But unfortunately, and issue with the inventory identification of the reuploaded RUs happened, so another removal and upload was necessary.
Again, I am just the messenger here trying to explain what has happened.
And sorry for all the inconvenience and lack of transparency.
By now, the COMBO patches should have been reuploaded again, too.
Further Links and Information
–Mike
“Ordre, contreordre, desordre.” Napoleon Bonaparte
Yep – was true a long time ago already, may still be true today.
Mike
Thanks, so I guess I downloaded again when it first reappeared. Is the “version” (.0) the only thing that was fixed? I installed everything with “opatchauto” and it looks like everything is patched and works. Of course, “lsinventory” shows the “.0”. That is just cosmetic, yes? Do I need to download and patch again?
——-
Patch 34765931 : applied on Wed Feb 01 07:15:26 CST 2023
Unique Patch ID: 25093776
Patch description: “DATABASE RELEASE UPDATE : 19.18.0.0.0 (REL-JAN230131) (34765931)”
Created on 27 Jan 2023, 11:25:14 hrs UTC
Bugs fixed:
35001966, 34958012, 32509606, 32790012, 31326998, 31466433, 28873575
32298004, 30374739, 29249289, 29920804, 31559415, 33050626, 31750581
33598840, 28709063, 31200057, 30045242, 32319410, 30045983, 29487189
31760738, 31163498, 31254929, 25979242, 33425046, 31895670, 29997553
29436522, 27907898, 32587994, 33181881, 30081055, 31661586, 31952046
Hi Bob,
exactly – this is that you easily can live with but what caused trouble in the cloud provisioning.
No need to change anything.
Cheers
Mike
I deployed the intermediate release to an on-prem database. I won’t lose any sleep over this as long as the next (April) RU will be able to deploy over it despite the bad patch description.
Good choice, Doug!!
And I just wanted to explain for those who were quite disappointed.
Cheers
Mike
OK. We downloaded and applied the first version. Now how to upgrade to the latest and greatest 19.18? 🙂
Hi Andy,
see in the MOS alert – you need the one off patch, then it is supposed to be the exact same.
Cheers
Mike
Please push for the feature of having _all_ released files in the ARU JSON, once withdrawn obviously without the download link which wouldn’t work anyway but rather with explanation that “this patch has been withdrawn on DATE”, or yet better with a JSON field with the timestamp.
Hi Juraj,
since we work in the same company, feel free to mail the owners directly 🙂
Cheers
Mike
I downloaded and applied the original to non-RAC. ok to use or should I download latest.
Absolutely ok to use – you won’t see any issues.
Thanks,
Mike
Is this also affecting the availability of 19.18RU bundle for AIX? It’s February and still I have been able to find only the OJVM bundle.
Hi Bill,
please see:
https://support.oracle.com/rs?type=doc&id=2906899.1
MOS Note: 2906899.1 – Critical Patch Update (CPU) Program Jan 2023 Patch Availability Document (DB-only
Goto to section 2.2 (Post Release Patches) and check what AIX tells you. I have unfortunately no insider information nor more than what’s mentioned in this note.
CHeers
Mike
Thx Mike 🙂
SR# 3-32112853251
Sorry for the inconvenience, Rafal, but I think this is solved now.
Thanks
Mike
Thanks Mike, the AIX db patch was released yesterday.
Hi Mike,
I need help.
Last patching 19.18 via autoupgrade fails on all databases – UPG-1400:
Patch 34765931 apply: WITH ERRORS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/34765931/25098466/34765931_apply_ABINAUTH_2023Feb08_20_25_15.log (errors)
-> Error at line 47025: script rdbms/admin/backport_files/bug_33527739_apply.sql
– ORA-13516: AWR Operation failed: CATPROC not valid
– ORA-06512: at “SYS.DBMS_WORKLOAD_REPOSITORY”, line 328
– ORA-06512: at “SYS.DBMS_WORKLOAD_REPOSITORY”, line 355
– ORA-06512: at line 12
Patch 34786990 apply: WITH ERRORS (PREV PATCH)
I restarted database in normal mode and resume job.
But there is a lot of manual work.
There is other solution for this ?
Hi Rafal,
this is a problem with an AWR change in 19.18.0 which magically expects the database to be NOT in upgrade/restricted mode (which it was in this case).
Problem is that it breaks the upgrade. We are chasing the AWR team already.
Would you please mind to open an SR and have Support analyze and tag this? Please share the SR number with me via the blog then.
Cheers,
Mike
Hi Mike,
just for your and other DBA info:
after I patched our RAC-testenv with RU18 last week I saw this in MOS dashboard:
Bug 34885986 – Flashback log file was not reused even if db_flashback_retention_target is passed (Doc ID 34885986.8)
which is affected the 19.18.
Just checked – As of now I doesn’t see any growup of occupied space in recovery area.
The instances runs since 6 days. Hopefully this remains. But I’m a little bit concerned to patch the produktive env. RU18 doesnt’ seem to make us happy.
Cheers Peter
Hi Peter,
thanks for the hint. And I was not aware of this issue. It is supposed to be fixed in the coming RU. It happens only in a RAC environment, and the workaround 1 would be to turn FB off and again ON afterwards. But this will only clean out the consumed space and won’t prevent the problem. Workaround 2 would be to delete those FB logs not needed anymore (this is a bit more complex to find out).
I hope that there will be a MOS note soon explaining this in more detail.
Cheers, and again, thanks for the hint!
Mike
Hi Mike,
Is everything already fixed with 19.18 RU for RAC systems as of today? Is it safe to go with it now?
Thanks,
Franky
Hahaha 🙂
Of course, everything is fixed by now 🙂
There may be only a few tiny issues here and there … and maybe a few others there and here too.
Mike
Hi Mike,
yesterday I opened a SR with the following question:
It’s now 4 weeks since the release date of january 2023 cpu and Patch 34765931 (DATABASE RELEASE UPDATE 19.18.0.0.0) and Patch 34777391 (Latest JDK Patch 19c) for 32bit Linux are still missing.
Doc ID 2584628.1 (JDK and PERL Patches for Oracle Database Home and Grid Home) says for Patch 34777391 (Latest JDK Patch 19c) a release date 31-Jan-2023.
When will these 2 patches finally be available ?
Until now I have not received any reply. Do you have any information about this?
Cheers, Hans-Martin
Hello Hans-Martin,
sorry, this took a bit longer.
That is done on a per-request basis as you can see below.
It is listed on the same CPU Patch Availability Document Section 1.13 (Also posted below) that you posted to explain what “On Request Platforms” means.
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2906899.1
3.1.7.3 Oracle Database 19
On-Request platforms
32-bit client-only platforms
1.3 On-Request Patches
Oracle does not proactively release patches for historically inactive platforms. However, Oracle will deliver these patches when requested.
The following guidelines describe how to initiate an on-request (OR) patch.
A request may be made:
At any time. However, a patch for a specific quarterly release, such as CPUOct2019, cannot be requested. Depending on when the request is received and processed, either the patch for the current quarterly release or the next quarterly release will be provided. Your Service Request (SR) will provide you the planned availability date for the patch.
As long as the version is in either Premier Support or Extended Support and error correction support has not expired. For example, if a product release is under Extended Support through the release of CPUJan2020 on January 15, 2020, then you can file a request for the product release through January 29, 2020. For more information, see Oracle Lifetime Support Policies at http://www.oracle.com/us/support/lifetime-support/index.html, and Note 209768.1, Database, FMW, Enterprise Manager, TimesTen In-Memory Database, and OCS Software Error Correction Support Policy.
For a platform-version combination when a major release or patch set is released on a platform after a quarterly release date. Oracle will provide the next patch for that platform-version combination, however you may request the current patch by following the on-request process. For example, if a patch is released for a platform on August 1, 2020, Oracle will provide the CPUOct2020 patch for that platform. You may request a CPUOct2020 patch for the platform, and Oracle will review the request and determine whether to provide CPUJul2020 or CPUOct2020.
A patch that is marked as on-request (OR) may already have been requested by another customer and be available on My Oracle Support. Before you file a Service Request (SR), check on My Oracle Support to see if the patch is already available for your platform.
hi, mike, how to get the GI patch of Windows platform 19c RAC? I tried the Bundle Patch of 19.14 and 19.18, and found that there is no GI patch in it, so how to patch the Grid?
See here, Thomas:
https://mikedietrichde.com/2020/11/09/why-is-the-19-9-0-release-update-not-available-yet-for-my-platform/
According to:
MOS Note: 2906899.1 – Critical Patch Update (CPU) Program Jan 2023 Patch Availability Document (DB-only).
see section “Post Release Patches”.
To me it looks as if it is available.
Cheers
Mike
Problems with dbms_metadata.get_ddl after 19.18 DB-RU patch. Before 19.18 I could extract out the create user syntax including the encrypted password. This is used to reset passwords after database refreshes from Prod to lower environments.
After the 19.18 patch I still get the create user statement, but it no longer displays the encrypted password.
Please delete this comment if it isn’t acceptable here. Thanks!
Please disregard my previous post. It isn’t the DB-RU that caused the issue but with the Datapump Bundle Patch 34972375 that is changing the dbms_metadata.get_ddl behavior. I will research this further as this is the first time I’m applying the DPBP patch. Sorry about that.
Hi Devin,
this is correct – and I have a blog post in the making since you are not the first customer being trapped by this.
It is a security fix which seemed to strike too far. We have escalated this within the company to our security architect but we wait for a reply now since 3 weeks.
If you don’t mind, I would find it very helpful if you’d open an SR and send me the SR number. This allows us to put some food into our internal communication.
And regarding your first message, there is absolutely nothing wrong with asking such topics here on the blog. I’m just a bit “behind” with answering the comments.
Cheers
Mike
Hi Devin,
when you open an SR, you can reference this bug number: 35018026
BUG 35018026 – DBMS_METADATA.GET_DDL(‘USER’,’USERNAME’) DOES NOT RETURN HASHED PASSWORD
Mike
SR opened and email sent to you with the number. Thanks for the helpful reference on the bug.
Hi Devin,
I still did not receive an email with your SR.
Can you please paste it here (if you want, I can delete the comment afterwards).
Cheers,
Mike
Here’s the SR: SR 3-32263011xxx
Thanks Devin!
Cheers
Mike
Hi,
Our customer has 19c SEHA (Oracle Standard Edition High Availability) with RU 19.16 on Linux.
Is it really required to apply the latest GI RU 19.18?
Does GI RU 19.18 has the previous issue resolved?
Thanks,
Thiru
Hi Thiru,
GI 19.18.0 has potentially 200 fixes on to of 19.16.0 as far as I am aware.
Cheers
Mike
Ok and Thanks a lot Mike,
Regards,
Thiru
Hello Mike,
Is there any specific instructions for applying 19.18 in an Oracle 19c SEHA environment.
In the patch readme I could not find any specific instructions for this case.
I am not sure under which case it falls under.
I followed Note: Doc ID 2246888.1 and selected this case:
Case 1.1.2B: Patching the GI Home and the Database Home Together, the GI Home Is Not Shared, the Database Home Is Not Shared, ACFS May Be Used
I applied in the Node 1 with the following command:
# $ORACLE_HOME/OPatch/opatchauto apply /_install/orasoft/grid_patch/34762026
It worked perfectly fine in Node1. GI patch was applied. RDBMS patch was also applied including data patch.
But when I executed the same command (# $ORACLE_HOME/OPatch/opatchauto apply /_install/orasoft/grid_patch/34762026) in Node2, it completed successfully but only GI patch was applied. It didn’t apply the RDBMS patch.
Any suggestion, advice is highly appreciated.
Thanks
Thiru
Hi Thiru,
with SEHA you please need to log an SR. I am absolutely blank on SEHA so far.
Cheers,
Mike
Hello Mike,
Thanks for your response. I opened SR but the support engineer was not able to help.
Finally, I myself found a way to apply the 19.18 successfully in 19c SEHA Production environment.
Regards,
Thiru
Thanks Thiru!
Cheers,
Mike
Hello,
maybe a stupid question, but how to build a goldImage including MRP patch ?
In the past, we’ve downloaed OEDA gold images as a base to build our own GoldImages with additional oneoffs required in our Company.
The (raw) steps are:
– Extract OEDA gold image
– run gridSetup.sh -silent -responseFile -waitForCompletion – applyOneOffs /u01/app/grid/oneoff_stage/19.18.0.0/34810252/34810252,/u01/app/grid/oneoff_stage/19.18.0.0/34932268/34932268
we are performing a sw_only installation (the contains “oracle.install.option=CRS_SWONLY”)
– root.sh is executed
– as next step run opatch auto for 19.18.0.0.230321 MRP with the -analyze -online option.
this also succeeds.
– i assumed the next step now is to run opatchauto apply, but this step failes because it assumes that the target GI_HOME is already active (-switchGridHome already executed, with is not done in our case) and of course rootcrs.pl -prepatch fails.
Long story short, this is my question: do i need to switch to the new 19.18 GI home before i could run opatchauto apply for MRP, or is there another option ?
The final aim is to build a GoldImagefor 19.18.0.0 including MRP 230321 + additional oneoffs.
Thanks in advance for your response.
Hi Manfred,
not a stupid question at all. In fact, it is us being a bit … how should I say … not finding the right word.
It turns out that you can’t apply the MRP with ./runInstaller -applyOneOffs but only with -napply.
So let me circle your question to the right people. I think you should be able to apply the MRP once you unzipped it with “-nappy” option pointing to every subdirectory.
Cheers,
Mike
Greetings Mike,
Having a issue(s) with a 12.1.0.2 non-CDB DB to 19c CDB/PDB (19.18.0.0) autoupgrade.
Although the the Analyze phase came back clean, it failed after 92% complete.
Any help or guidance would be much appreciated as always, because we are always under a time crunch.
Cheers, George
Config file:
db-axiumt-2:caxiumd: $ cat upg1_axiumd.cfg
global.autoupg_log_dir=/oradata/admin/autoupgrade/logs
upg1.dbname=axiumd
upg1.start_time=now
upg1.source_home=/orahome/app/oracle/product/12102/db_1
upg1.target_base=/orahome/app/oracle
upg1.target_home=/orahome/app/oracle/product/19c/db_1
upg1.sid=axiumd
upg1.log_dir=/oradata/admin/autoupgrade/logs
upg1.upgrade_node=db-axiumt-2
upg1.target_version=19
upg1.restoration=no
upg1.target_cdb=caxiumd
upg1.run_utlrp=yes
upg1.timezone_upg=yes
db-axiumt-2:caxiumd: $
*** WARNING: ERRORS FOUND DURING UPGRADE ***
1. Evaluate the errors found in the upgrade logs
and determine the proper action.
2. Rerun the upgrade when the problem is resolved
REASON:
ERRORS FOUND: During Upgrade
FILENAME: /oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620113308axiumd0.log AT LINE NUMBER: 12971
——————————————————
Identifier DATAPATCH_ 23-06-20 11:33:10
SCRIPT = [/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620113308axiumd_datapatch_upgrade.log]
ERROR = [Error: prereq checks failed!
]
STATEMENT = []
——————————————————
LOG FILES: (/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620113308axiumd*.log)
Upgrade Summary Report Located in:
/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/upg_summary.log
End of Input Commands
——————————————————
Start of DataPatch Logs
**************************
Start of DataPatch Logs
——————————————————
stdout from running datapatch to install upgrade SQL patches and PSUs:
SQL Patching tool version 19.18.0.0.0 Production on Tue Jun 20 11:33:09 2023
Copyright (c) 2012, 2023, Oracle. All rights reserved.
Log file for this invocation: /orahome/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_1771733_2023_06_20_11_33_09/sqlpatch_invocation.log
Connecting to database…OK
Gathering database info…done
Bootstrapping registry and package to current versions…done
Error: prereq checks failed!
verify_queryable_inventory returned ORA-20001: Latest xml inventory is not loaded into table
Prereq check failed, exiting without installing any patches.
Please refer to MOS Note 1609718.1 and/or the invocation log
/orahome/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_1771733_2023_06_20_11_33_09/sqlpatch_invocation.log
for information on how to resolve the above errors.
SQL Patching tool complete on Tue Jun 20 11:33:10 2023
stderr from running datapatch to install upgrade SQL patches and PSUs:
From your post changed the following init.ora parameters on CDB and bounced:
From > https://mikedietrichde.com/2020/07/31/upgrade-fails-with-ora-20001-during-datapatch-run/
SQL> EXEC dbms_qopatch.clean_metadata();
PL/SQL procedure successfully completed.
SQL> show parameter _xt_preproc_timeout
NAME TYPE
———————————— ———————————
VALUE
——————————
_xt_preproc_timeout integer
500
Then got this:
build.version:23.1.230224
build.date:2023/02/24 14:53:24 -0500
build.max_target_version:21
build.supported_target_versions:12.2,18,19,21
build.type:production
build.hash_date:2023/02/24 14:44:39 -0500
build.label:(HEAD, tag: v23.1, origin/stable_devel, stable_devel)
2023-06-20 11:50:44.453 INFO Restoration inactive based on current configuration for database axiumd
2023-06-20 11:50:44.584 INFO Total Number of phases is 108
2023-06-20 11:50:44.590 INFO Executing ALTER SYSTEM SET PLSQL_WARNINGS=’DISABLE:ALL’;AXIUMD
2023-06-20 11:50:44.619 INFO Begin Upgrade on Database [AXIUMD]
2023-06-20 11:50:49.655 INFO 92%Upgraded
2023-06-20 11:53:42.778 ERROR
DATABASE NAME: axiumd
CAUSE: ERROR at Line 14858 in [/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620115044axiumd0.log]
REASON: Error: prereq checks failed!
ACTION: [MANUAL]
DETAILS:
2023-06-20 11:53:42.785 ERROR Database Error in File [/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620115044axiumd0.log] on Database [/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620115044axiumd0.log]
2023-06-20 11:53:42.792 ERROR UPGRADE FAILED [AXIUMD]
2023-06-20 11:53:42.792 ERROR Exception Error in Database [UPG-1400#UPGRADE FAILED [AXIUMD]]
2023-06-20 11:53:42.793 INFO End Upgrade on Database [AXIUMD]
2023-06-20 11:53:42.825 INFO Executing ALTER SYSTEM SET PLSQL_WARNINGS=’DISABLE:ALL’;AXIUMD
2023-06-20 11:53:45.112 ERROR UPGRADE FAILED [AXIUMD]
2023-06-20 11:53:45.113 ERROR Exception Error in Database [AXIUMD]
2023-06-20 11:53:45.135 ERROR axiumd Return status is ERROR
2023-06-20 11:53:45.141 ERROR Dispatcher failed: AutoUpgException [Database job failed with errors]
2023-06-20 11:53:45.143 ERROR Error running dispatcher for job 105
Cause: Database job failed with errors
2023-06-20 11:53:45.143 ERROR Dispatcher failed:
Error: UPG-1400
UPGRADE FAILED [AXIUMD]
Hi George,
it has finished but failed in the datapatch execution.
Just run datapatch again afterwards (make sure the PDB is open at least restricted).
Otherwise, check my blog for the ORA-20001 error – there you’ll find an underscore. The problem usually is the time out for reading the inventory.
The other reason could be your setting for PLSQL_WARNINGS.
Can you check this please and see whether it is DISABLED?
Cheers
Mike