Hide and Seek with RU 19.18.0

Posted on February 2, 2023 by Flaws and Pitfalls Patch Recommendation 46

One and a half weeks ago we released the Database and Grid Infrastructure Release Updates (RU) 19.18.0. You could read on the blog how to apply it, and you could read also about an alert. So far, all was more or less transparent but apparently sort of an Hide and Seek with RU 19.18.0 has happened. So let me shed some light.

Heads Up!

I am just the messenger trying to bring some light into the topic. Neither is our team responsible nor do we have any sort of control on this process. I write this upfront since my team has received some really nasty and angry comments from customers, ACEs and also colleagues. Don’t shot the messenger please!

 

Upload, Alert, Removal …

When Database and Grid Infrastructure 19.18.0 Release Update (RU) got uploaded, it all looked fine. We made it available on Linux at first, so everybody being on another platform is not affected by this anyway. Interestingly enough, there was an incredibly high download count of this RU within the first days. This makes me think very positive since it tells me that you download the RUs right when we release them. Great news.

But it turned out that the RUs have a flaw which affects RAC customers only. And since the RU got released on Linux only, it were RAC-Linux customers only. Still, this number is not small. The issue is described in Important Alert with RU 19.18.0. As a consequence, and to fix the issue, both 19.18.0 RUs for the Database and for Grid Infrastructure got removed.

Bad side effect:
The MyOracle Support portal (MOS) is not very user-friendly and only told you “Patch does not exist“. No further explanation, no information, nothing.

This is when the first emails and comments reached me asking why the patch bundles vanished. Even though I just had explained how to apply the January 2023 patch bundles.

Thanks to all of you being faster than I, Peter Lehmann in this case was the first to point me to the alert. It dawned me that this was the reason for the RUs being removed which they obviously were.

Some hours later I understood the issue and reason why this has happened. And I did update my original blog post with some additional information and the tentative re-upload date of Jan 31, 2023.

 

Upload, Removal, Upload again …

And so it happened.

The RUs appeared again, the first customers commented that the RUs are available again.

Until they disappeared. Again.
And certainly without any information or comment.

Of course, they did not disappear completely at first, but the option to download disappeared. So the button marked in red had no function. And since you could not choose the platform, there was no download possible.

The bad situation:
Some people had used the first drop of the RU, some had downloaded the 2nd drop – and some were still waiting since it had vanished again. At this stage, the first people commented about wrong inventory information with the 2nd drop.

  • Previous one:
    34765931;Database Release Update : 19.18.0.0.230117 (34765931)
  • New (intermediate) one:
    34765931;DATABASE RELEASE UPDATE : 19.18.0.0.0 (REL-JAN230131) (34765931)

The zero in the 5th number of the release looked strange. And I’m always amazed how many tech experts we have out there. Several customers noted this right away. And complained to me.

Well, it turned out that you weren’t the only ones wondering and complaining.

This 5th number broke the provisioning of 19.18.0 in our cloud tooling as well since it expects the date as above “230117” instead of a plain “0“. Making a long story short, the RU had to be removed and re-uploaded again, now with a new entry into the inventory which most likely is:

  • 34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)

I haven’t verified it but this is what I’d been told.

 

Summary

An issue with the 19.18.0 RUs affecting only RAC customers on Linux made it necessary to repacked both, the Database and the Grid Infrastructure Release Updates. But unfortunately, and issue with the inventory identification of the reuploaded RUs happened, so another removal and upload was necessary.

Again, I am just the messenger here trying to explain what has happened.

And sorry for all the inconvenience and lack of transparency.

By now, the COMBO patches should have been reuploaded again, too.

 

Further Links and Information

–Mike

 

Share this:

Tags:

You may also like...

46 Responses

  1. Christian Klammer says:
    February 2, 2023 at 17:04

    “Ordre, contreordre, desordre.” Napoleon Bonaparte

    Reply
  2. Bob Bryla says:
    February 2, 2023 at 17:53

    Thanks, so I guess I downloaded again when it first reappeared. Is the “version” (.0) the only thing that was fixed? I installed everything with “opatchauto” and it looks like everything is patched and works. Of course, “lsinventory” shows the “.0”. That is just cosmetic, yes? Do I need to download and patch again?
    ——-
    Patch 34765931 : applied on Wed Feb 01 07:15:26 CST 2023
    Unique Patch ID: 25093776
    Patch description: “DATABASE RELEASE UPDATE : 19.18.0.0.0 (REL-JAN230131) (34765931)”
    Created on 27 Jan 2023, 11:25:14 hrs UTC
    Bugs fixed:
    35001966, 34958012, 32509606, 32790012, 31326998, 31466433, 28873575
    32298004, 30374739, 29249289, 29920804, 31559415, 33050626, 31750581
    33598840, 28709063, 31200057, 30045242, 32319410, 30045983, 29487189
    31760738, 31163498, 31254929, 25979242, 33425046, 31895670, 29997553
    29436522, 27907898, 32587994, 33181881, 30081055, 31661586, 31952046

    Reply
    • Mike.Dietrich says:
      February 2, 2023 at 19:22

      Hi Bob,

      exactly – this is that you easily can live with but what caused trouble in the cloud provisioning.
      No need to change anything.

      Cheers
      Mike

      Reply
      • Doug Kushner says:
        February 2, 2023 at 20:08

        I deployed the intermediate release to an on-prem database. I won’t lose any sleep over this as long as the next (April) RU will be able to deploy over it despite the bad patch description.

        Reply
  3. Andy says:
    February 2, 2023 at 19:00

    OK. We downloaded and applied the first version. Now how to upgrade to the latest and greatest 19.18? 🙂

    Reply
  4. Juraj says:
    February 2, 2023 at 21:22

    Please push for the feature of having _all_ released files in the ARU JSON, once withdrawn obviously without the download link which wouldn’t work anyway but rather with explanation that “this patch has been withdrawn on DATE”, or yet better with a JSON field with the timestamp.

    Reply
  5. SteveG says:
    February 3, 2023 at 09:36

    I downloaded and applied the original to non-RAC. ok to use or should I download latest.

    Reply
  6. Bill says:
    February 6, 2023 at 21:39

    Is this also affecting the availability of 19.18RU bundle for AIX? It’s February and still I have been able to find only the OJVM bundle.

    Reply
  7. Bill says:
    February 8, 2023 at 18:00

    Thanks Mike, the AIX db patch was released yesterday.

    Reply
  8. rafal says:
    February 9, 2023 at 08:28

    Hi Mike,
    I need help.

    Last patching 19.18 via autoupgrade fails on all databases – UPG-1400:
    Patch 34765931 apply: WITH ERRORS
    logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/34765931/25098466/34765931_apply_ABINAUTH_2023Feb08_20_25_15.log (errors)
    -> Error at line 47025: script rdbms/admin/backport_files/bug_33527739_apply.sql
    – ORA-13516: AWR Operation failed: CATPROC not valid
    – ORA-06512: at “SYS.DBMS_WORKLOAD_REPOSITORY”, line 328
    – ORA-06512: at “SYS.DBMS_WORKLOAD_REPOSITORY”, line 355
    – ORA-06512: at line 12
    Patch 34786990 apply: WITH ERRORS (PREV PATCH)

    I restarted database in normal mode and resume job.
    But there is a lot of manual work.

    There is other solution for this ?

    Reply
    • Mike.Dietrich says:
      February 9, 2023 at 10:55

      Hi Rafal,

      this is a problem with an AWR change in 19.18.0 which magically expects the database to be NOT in upgrade/restricted mode (which it was in this case).
      Problem is that it breaks the upgrade. We are chasing the AWR team already.

      Would you please mind to open an SR and have Support analyze and tag this? Please share the SR number with me via the blog then.

      Cheers,
      Mike

      Reply
  9. Peter says:
    February 13, 2023 at 08:25

    Hi Mike,
    just for your and other DBA info:
    after I patched our RAC-testenv with RU18 last week I saw this in MOS dashboard:
    Bug 34885986 – Flashback log file was not reused even if db_flashback_retention_target is passed (Doc ID 34885986.8)
    which is affected the 19.18.
    Just checked – As of now I doesn’t see any growup of occupied space in recovery area.
    The instances runs since 6 days. Hopefully this remains. But I’m a little bit concerned to patch the produktive env. RU18 doesnt’ seem to make us happy.
    Cheers Peter

    Reply
    • Mike.Dietrich says:
      February 28, 2023 at 19:00

      Hi Peter,

      thanks for the hint. And I was not aware of this issue. It is supposed to be fixed in the coming RU. It happens only in a RAC environment, and the workaround 1 would be to turn FB off and again ON afterwards. But this will only clean out the consumed space and won’t prevent the problem. Workaround 2 would be to delete those FB logs not needed anymore (this is a bit more complex to find out).

      I hope that there will be a MOS note soon explaining this in more detail.

      Cheers, and again, thanks for the hint!
      Mike

      Reply
  10. Franky Weber Faust says:
    February 13, 2023 at 23:32

    Hi Mike,

    Is everything already fixed with 19.18 RU for RAC systems as of today? Is it safe to go with it now?

    Thanks,

    Franky

    Reply
  11. Hans-Martin Ruff says:
    February 15, 2023 at 16:47

    Hi Mike,
    yesterday I opened a SR with the following question:
    It’s now 4 weeks since the release date of january 2023 cpu and Patch 34765931 (DATABASE RELEASE UPDATE 19.18.0.0.0) and Patch 34777391 (Latest JDK Patch 19c) for 32bit Linux are still missing.
    Doc ID 2584628.1 (JDK and PERL Patches for Oracle Database Home and Grid Home) says for Patch 34777391 (Latest JDK Patch 19c) a release date 31-Jan-2023.
    When will these 2 patches finally be available ?
    Until now I have not received any reply. Do you have any information about this?
    Cheers, Hans-Martin

    Reply
    • Mike.Dietrich says:
      February 27, 2023 at 11:39

      Hello Hans-Martin,

      sorry, this took a bit longer.

      That is done on a per-request basis as you can see below.

      It is listed on the same CPU Patch Availability Document Section 1.13 (Also posted below) that you posted to explain what “On Request Platforms” means.
      https://support.oracle.com/epmos/faces/DocumentDisplay?id=2906899.1

      3.1.7.3 Oracle Database 19

      On-Request platforms

      32-bit client-only platforms

      1.3 On-Request Patches

      Oracle does not proactively release patches for historically inactive platforms. However, Oracle will deliver these patches when requested.

      The following guidelines describe how to initiate an on-request (OR) patch.

      A request may be made:

      At any time. However, a patch for a specific quarterly release, such as CPUOct2019, cannot be requested. Depending on when the request is received and processed, either the patch for the current quarterly release or the next quarterly release will be provided. Your Service Request (SR) will provide you the planned availability date for the patch.
      As long as the version is in either Premier Support or Extended Support and error correction support has not expired. For example, if a product release is under Extended Support through the release of CPUJan2020 on January 15, 2020, then you can file a request for the product release through January 29, 2020. For more information, see Oracle Lifetime Support Policies at http://www.oracle.com/us/support/lifetime-support/index.html, and Note 209768.1, Database, FMW, Enterprise Manager, TimesTen In-Memory Database, and OCS Software Error Correction Support Policy.
      For a platform-version combination when a major release or patch set is released on a platform after a quarterly release date. Oracle will provide the next patch for that platform-version combination, however you may request the current patch by following the on-request process. For example, if a patch is released for a platform on August 1, 2020, Oracle will provide the CPUOct2020 patch for that platform. You may request a CPUOct2020 patch for the platform, and Oracle will review the request and determine whether to provide CPUJul2020 or CPUOct2020.

      A patch that is marked as on-request (OR) may already have been requested by another customer and be available on My Oracle Support. Before you file a Service Request (SR), check on My Oracle Support to see if the patch is already available for your platform.

      Reply
  12. thomas says:
    February 17, 2023 at 16:05

    hi, mike, how to get the GI patch of Windows platform 19c RAC? I tried the Bundle Patch of 19.14 and 19.18, and found that there is no GI patch in it, so how to patch the Grid?

    Reply
  13. Devin says:
    February 23, 2023 at 23:22

    Problems with dbms_metadata.get_ddl after 19.18 DB-RU patch. Before 19.18 I could extract out the create user syntax including the encrypted password. This is used to reset passwords after database refreshes from Prod to lower environments.
    After the 19.18 patch I still get the create user statement, but it no longer displays the encrypted password.
    Please delete this comment if it isn’t acceptable here. Thanks!

    Reply
  14. Devin Conner says:
    February 24, 2023 at 00:26

    Please disregard my previous post. It isn’t the DB-RU that caused the issue but with the Datapump Bundle Patch 34972375 that is changing the dbms_metadata.get_ddl behavior. I will research this further as this is the first time I’m applying the DPBP patch. Sorry about that.

    Reply
    • Mike.Dietrich says:
      February 24, 2023 at 09:06

      Hi Devin,

      this is correct – and I have a blog post in the making since you are not the first customer being trapped by this.
      It is a security fix which seemed to strike too far. We have escalated this within the company to our security architect but we wait for a reply now since 3 weeks.

      If you don’t mind, I would find it very helpful if you’d open an SR and send me the SR number. This allows us to put some food into our internal communication.

      And regarding your first message, there is absolutely nothing wrong with asking such topics here on the blog. I’m just a bit “behind” with answering the comments.

      Cheers
      Mike

      Reply
  15. Thiru says:
    February 24, 2023 at 10:42

    Hi,

    Our customer has 19c SEHA (Oracle Standard Edition High Availability) with RU 19.16 on Linux.

    Is it really required to apply the latest GI RU 19.18?

    Does GI RU 19.18 has the previous issue resolved?

    Thanks,
    Thiru

    Reply
  16. Thiru says:
    February 28, 2023 at 06:16

    Ok and Thanks a lot Mike,

    Regards,
    Thiru

    Reply
  17. Thiru says:
    March 5, 2023 at 14:14

    Hello Mike,

    Is there any specific instructions for applying 19.18 in an Oracle 19c SEHA environment.

    In the patch readme I could not find any specific instructions for this case.

    I am not sure under which case it falls under.

    I followed Note: Doc ID 2246888.1 and selected this case:

    Case 1.1.2B: Patching the GI Home and the Database Home Together, the GI Home Is Not Shared, the Database Home Is Not Shared, ACFS May Be Used

    I applied in the Node 1 with the following command:

    # $ORACLE_HOME/OPatch/opatchauto apply /_install/orasoft/grid_patch/34762026

    It worked perfectly fine in Node1. GI patch was applied. RDBMS patch was also applied including data patch.

    But when I executed the same command (# $ORACLE_HOME/OPatch/opatchauto apply /_install/orasoft/grid_patch/34762026) in Node2, it completed successfully but only GI patch was applied. It didn’t apply the RDBMS patch.

    Any suggestion, advice is highly appreciated.

    Thanks
    Thiru

    Reply
  18. Thiru says:
    April 1, 2023 at 15:31

    Hello Mike,

    Thanks for your response. I opened SR but the support engineer was not able to help.

    Finally, I myself found a way to apply the 19.18 successfully in 19c SEHA Production environment.

    Regards,
    Thiru

    Reply
  19. Sickinger Manfred says:
    April 17, 2023 at 13:44

    Hello,

    maybe a stupid question, but how to build a goldImage including MRP patch ?
    In the past, we’ve downloaed OEDA gold images as a base to build our own GoldImages with additional oneoffs required in our Company.
    The (raw) steps are:
    – Extract OEDA gold image
    – run gridSetup.sh -silent -responseFile -waitForCompletion – applyOneOffs /u01/app/grid/oneoff_stage/19.18.0.0/34810252/34810252,/u01/app/grid/oneoff_stage/19.18.0.0/34932268/34932268
    we are performing a sw_only installation (the contains “oracle.install.option=CRS_SWONLY”)
    – root.sh is executed
    – as next step run opatch auto for 19.18.0.0.230321 MRP with the -analyze -online option.
    this also succeeds.
    – i assumed the next step now is to run opatchauto apply, but this step failes because it assumes that the target GI_HOME is already active (-switchGridHome already executed, with is not done in our case) and of course rootcrs.pl -prepatch fails.

    Long story short, this is my question: do i need to switch to the new 19.18 GI home before i could run opatchauto apply for MRP, or is there another option ?

    The final aim is to build a GoldImagefor 19.18.0.0 including MRP 230321 + additional oneoffs.

    Thanks in advance for your response.

    Reply
    • Mike.Dietrich says:
      May 16, 2023 at 15:52

      Hi Manfred,

      not a stupid question at all. In fact, it is us being a bit … how should I say … not finding the right word.
      It turns out that you can’t apply the MRP with ./runInstaller -applyOneOffs but only with -napply.

      So let me circle your question to the right people. I think you should be able to apply the MRP once you unzipped it with “-nappy” option pointing to every subdirectory.

      Cheers,
      Mike

      Reply
  20. GEORGE M LAVAN says:
    June 20, 2023 at 19:39

    Greetings Mike,

    Having a issue(s) with a 12.1.0.2 non-CDB DB to 19c CDB/PDB (19.18.0.0) autoupgrade.

    Although the the Analyze phase came back clean, it failed after 92% complete.

    Any help or guidance would be much appreciated as always, because we are always under a time crunch.

    Cheers, George

    Config file:

    db-axiumt-2:caxiumd: $ cat upg1_axiumd.cfg
    global.autoupg_log_dir=/oradata/admin/autoupgrade/logs
    upg1.dbname=axiumd
    upg1.start_time=now
    upg1.source_home=/orahome/app/oracle/product/12102/db_1
    upg1.target_base=/orahome/app/oracle
    upg1.target_home=/orahome/app/oracle/product/19c/db_1
    upg1.sid=axiumd
    upg1.log_dir=/oradata/admin/autoupgrade/logs
    upg1.upgrade_node=db-axiumt-2
    upg1.target_version=19
    upg1.restoration=no
    upg1.target_cdb=caxiumd
    upg1.run_utlrp=yes
    upg1.timezone_upg=yes
    db-axiumt-2:caxiumd: $

    *** WARNING: ERRORS FOUND DURING UPGRADE ***

    1. Evaluate the errors found in the upgrade logs
    and determine the proper action.
    2. Rerun the upgrade when the problem is resolved

    REASON:
    ERRORS FOUND: During Upgrade
    FILENAME: /oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620113308axiumd0.log AT LINE NUMBER: 12971
    ——————————————————
    Identifier DATAPATCH_ 23-06-20 11:33:10
    SCRIPT = [/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620113308axiumd_datapatch_upgrade.log]
    ERROR = [Error: prereq checks failed!
    ]
    STATEMENT = []
    ——————————————————

    LOG FILES: (/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620113308axiumd*.log)

    Upgrade Summary Report Located in:
    /oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/upg_summary.log

    End of Input Commands
    ——————————————————

    Start of DataPatch Logs

    **************************

    Start of DataPatch Logs
    ——————————————————
    stdout from running datapatch to install upgrade SQL patches and PSUs:
    SQL Patching tool version 19.18.0.0.0 Production on Tue Jun 20 11:33:09 2023
    Copyright (c) 2012, 2023, Oracle. All rights reserved.

    Log file for this invocation: /orahome/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_1771733_2023_06_20_11_33_09/sqlpatch_invocation.log

    Connecting to database…OK
    Gathering database info…done
    Bootstrapping registry and package to current versions…done
    Error: prereq checks failed!
    verify_queryable_inventory returned ORA-20001: Latest xml inventory is not loaded into table
    Prereq check failed, exiting without installing any patches.

    Please refer to MOS Note 1609718.1 and/or the invocation log
    /orahome/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_1771733_2023_06_20_11_33_09/sqlpatch_invocation.log
    for information on how to resolve the above errors.

    SQL Patching tool complete on Tue Jun 20 11:33:10 2023
    stderr from running datapatch to install upgrade SQL patches and PSUs:

    From your post changed the following init.ora parameters on CDB and bounced:

    From > https://mikedietrichde.com/2020/07/31/upgrade-fails-with-ora-20001-during-datapatch-run/

    SQL> EXEC dbms_qopatch.clean_metadata();

    PL/SQL procedure successfully completed.

    SQL> show parameter _xt_preproc_timeout

    NAME TYPE
    ———————————— ———————————
    VALUE
    ——————————
    _xt_preproc_timeout integer
    500

    Then got this:

    build.version:23.1.230224
    build.date:2023/02/24 14:53:24 -0500
    build.max_target_version:21
    build.supported_target_versions:12.2,18,19,21
    build.type:production
    build.hash_date:2023/02/24 14:44:39 -0500
    build.label:(HEAD, tag: v23.1, origin/stable_devel, stable_devel)

    2023-06-20 11:50:44.453 INFO Restoration inactive based on current configuration for database axiumd
    2023-06-20 11:50:44.584 INFO Total Number of phases is 108
    2023-06-20 11:50:44.590 INFO Executing ALTER SYSTEM SET PLSQL_WARNINGS=’DISABLE:ALL’;AXIUMD
    2023-06-20 11:50:44.619 INFO Begin Upgrade on Database [AXIUMD]
    2023-06-20 11:50:49.655 INFO 92%Upgraded
    2023-06-20 11:53:42.778 ERROR
    DATABASE NAME: axiumd
    CAUSE: ERROR at Line 14858 in [/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620115044axiumd0.log]
    REASON: Error: prereq checks failed!
    ACTION: [MANUAL]
    DETAILS:
    2023-06-20 11:53:42.785 ERROR Database Error in File [/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620115044axiumd0.log] on Database [/oradata/admin/autoupgrade/logs/axiumd/105/dbupgrade/catupgrd20230620115044axiumd0.log]
    2023-06-20 11:53:42.792 ERROR UPGRADE FAILED [AXIUMD]
    2023-06-20 11:53:42.792 ERROR Exception Error in Database [UPG-1400#UPGRADE FAILED [AXIUMD]]
    2023-06-20 11:53:42.793 INFO End Upgrade on Database [AXIUMD]
    2023-06-20 11:53:42.825 INFO Executing ALTER SYSTEM SET PLSQL_WARNINGS=’DISABLE:ALL’;AXIUMD
    2023-06-20 11:53:45.112 ERROR UPGRADE FAILED [AXIUMD]
    2023-06-20 11:53:45.113 ERROR Exception Error in Database [AXIUMD]
    2023-06-20 11:53:45.135 ERROR axiumd Return status is ERROR
    2023-06-20 11:53:45.141 ERROR Dispatcher failed: AutoUpgException [Database job failed with errors]
    2023-06-20 11:53:45.143 ERROR Error running dispatcher for job 105
    Cause: Database job failed with errors
    2023-06-20 11:53:45.143 ERROR Dispatcher failed:
    Error: UPG-1400
    UPGRADE FAILED [AXIUMD]

    Reply
    • Mike.Dietrich says:
      June 20, 2023 at 20:13

      Hi George,

      it has finished but failed in the datapatch execution.
      Just run datapatch again afterwards (make sure the PDB is open at least restricted).

      Otherwise, check my blog for the ORA-20001 error – there you’ll find an underscore. The problem usually is the time out for reading the inventory.

      The other reason could be your setting for PLSQL_WARNINGS.
      Can you check this please and see whether it is DISABLED?

      Cheers
      Mike

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Mike Dietrich
Oracle Senior Director of Product Manager for Database Upgrade, Migrations and Patching writing just another blog.

Opinions certainly are my own as is this page.

Archives

Categories

Upgrade @YouTube

Subscribe via Email

Join 7,849 other subscribers

Follow me on Twitter