Some additional information regarding the Oracle 19.14.0 RUs

Posted on January 27, 2022 by Patch Recommendation 27

You may have read my previous blog post about patching all my environments to the January 2022 patch bundles. But since I do this in a very simple non-RAC, non-GI environment, I may not see issues you may have encountered. Some people commented – and I would like to share Some additional information regarding the Oracle 19.14.0 RUs . Still, please, this is NOT a collection of all open issues. I just see this blog post as a summary of the issues I have seen and/or heard about so far.

Some additional information regarding the Oracle 19.14.0 RUs

Photo by Josh Hild on Unsplash

555.1 is one of the most important MOS notes

At first, before you read the about the topics below, make sure you have checked one of THE most important MOS notes:

And please, add it to your favorites as well (the upper left star in the MOS note). It lists the most important issues for each of the patch bundles and the fixes you should consider to put on top of any of them.

ACFS RU issue

Important information:
The Development team tried to reproduce the below case but didn’t succeed. In addition, please be aware that the initial setup was a supported combination (kernel version <=> ACFS version) while the result at the end isn’t a supported mix of kernel and ACFS. So this isn’t a solution you should attempt.

Nariman reported this issue to me via the blog (and thanks for sharing it with me including the solution). He had a significant problem with applying 19.14.0 to his environment. At first, his apply of 19.14.0 failed in the CRS part already. Then he tried to install each of the sub-components of the patch bundle separately and found that the fail is caused by the ACFSRU, patch 33534448.

The key error message was:

2022/01/20 12:46:28 CLSRSC-400: A system reboot is required to continue installing.

Argument “” isn’t numeric in numeric ne (!=) at /u01/app/grid/19/grid/lib/acfslib.pm line 347.

But the system reboot did not solve the problem.

The solution finally as Nariman told me was:

As soon as I updated kernel, it was upgraded from “4.18.0-305.19.1.el8_4.x86_64” to “4.18.0-348.12.2.el8_5.x86_64” and the patching went perfect!

I wouldn’t have been considered a kernel update as a potential solution – but very happy that it fixed it.

GI Patching Alert: CRS-5037

This is something you can find already in the comments section of Patching all my environments with the January 2022 Patch Bundles. And at the same time, my dear ACS colleague Thomas alerted me on slack as well about this issue.

In this note you can read that with 19.14 GI, user-defined database services may fail to start with error CRS-5037: Database does not have the required fix for bug ‘31143870’. Not everybody is going to see this issue. The MOS note explains that only configurations satisfying all 3 conditions listed below may experience the issue:

  • Grid Infrastructure Release “19.14”
  • Contains database(s) running on “RDBMS Releases <= 18c” and/or “RDBMS Releases 19.3 to 19.10 without Patch 31143870”
  • Has one or more user-defined services NOT created on PDB. i.e. “crsctl stat res ora.db_name.service_name.svc -f | grep PLUGGABLE_DATABSE” will show null value for PLUGGABLE_DATABSE key such as: 
    $ crsctl stat res ora.database_name.service_name.svc -f | grep PLUGGABLE_DATABASE
PLUGGABLE_DATABASE=
$

     

And a solution is to apply patch 33733953 beforehand.

Meanwhile, this issue got added to MOS Note: 555.1 as well:

Some additional information regarding the Oracle 19.14.0 RUs

Gridsetup SwitchGridHome fails

This is actually an issue which came up with 19.13.0 already. You may encounter it only when you do out-of-place GI patching. When you patch in-place you won’t see this bug. Still, this isn’t fixed in 19.14.0 yet but only from 19.15.0 onward.

It is labeled as:

  • Bug 33601195 : GRIDSETUP -SWITCHGRIDHOME OOP – ROOT.SH GENERATED INCORRECTLY

And there are two good news for you. At first, you can download a patch 33601195 on top of 19.13.0 if you plan to patch GI out-of-place to 19.13.0. But as of now I can’t see any patch on top of 19.14.0 yet.

What’s the other good news? Well, you can use this proven workaround:

  1. Node_1: Execute $ORACLE_HOME/root.sh
  2. Node_1: scp $ORACLE_HOME/crs/config/rootconfig.sh Node_2:$ORACLE_HOME/crs/config/rootconfig.sh
  3. Node_1: scp $ORACLE_HOME/crs/install/crsconfig_params Node_2:$ORACLE_HOME/ crs/install/crsconfig_params
  4. Node_2: Execute $ORACLE_HOME/root.sh

And of course, if you have more than 2 nodes, you must repeat the copy operation to the other nodes as well and execute root.sh there, too.

Patching Duration

Several people over the past months complained about patching duration. If this applies to the opatch or opatchauto part of the patching, please open an SR. As I commented already in Patching all my environments with the January 2022 Patch Bundles, it takes a significant long time in my 19c environment while the 21c or the 12.2.0.1 environments get patches fairly quick. I see minute-long waits during the opatch process but I haven’t found either the root cause or a solution yet. Opatch 28 (6880880) is supposed to fix one of these issues – but since I used opatch 28 for my environments, I haven’t seen it completing faster.

For the datapatch part of the patching, I will have another blog post within the coming days.

Further Links and Information

–Mike

 

Share this:

Tags:

You may also like...

27 Responses

  1. Andrew says:
    January 27, 2022 at 11:20

    Hi Mike,

    Interesting Narimans issue with ACFS and updating kernel to “4.18.0-348.12.2.el8_5.x86_64”. I had just been looking at ACFS certification yesterday and noticed that ACFS is only certified up to kernel 4.18.0-339 at the moment with patch 33535435 and only got a patch for 19.13 at the moment 🙁

    Reply
    • Nariman Narimanov says:
      January 31, 2022 at 19:53

      Hi Andrew,

      You are absolutely right, but the thing is I’m not using ACFS in my env. That’s why the patching went flaweless. If I configured ACFS, it would 100% have been failed because of the unsupported kernel 🙂

      1369107.1 – really good doc

      Reply
  2. Nenad Noveljic says:
    January 27, 2022 at 12:17

    Hi Mike,

    datapatch failed after migrating from Solaris to Linux because datapatch tried to create dbms_hadoop* packages, which depend on dba_hive* views. dba_hive* views aren’t created on Solaris x86. The following article contains more details: https://nenadnoveljic.com/blog/compilation-error-in-19-14-datapatch/

    Reply
  3. PeT says:
    January 28, 2022 at 15:21

    HI Mike,

    If 555.1 is important , so is 1369107.1… Could you ask them to update it with the latest info?

    Reply
  4. Srini says:
    January 29, 2022 at 03:42

    ok

    Reply
  5. Paul says:
    January 31, 2022 at 01:35

    How confusing. I dont see mention of the “Nariman” bug in MOS note 555.1? uname -a shows “4.1.12-124.53.5.1.el7uek.x86_64” on our system – how do I know if our patching will succeed? I am wanting to upgrade from 12.2 to 19c and use -applyRU to apply the 19.14RU.

    Bug 33733953 says it is only applicable to RAC systems – so as we are using Oracle restart I assume I can ignore this bug/patch.

    Bug 33601195 – we are installing a new 19c grid home and applying 19.14 RU using -applyRU. Do we need to worry about this bug?

    Reply
    • Mike.Dietrich says:
      February 11, 2022 at 22:07

      Hi Paul,

      actually I added an important comment to make this more clear. You should be in a supported kernel/acfs configuration. Dev couldn’t reproduce the issue in this case, and they tried really hard.

      Cheers,
      Mike

      PS: Into 555.1 only very critical issues which may affect the majority of our customers will be added usually.

      Reply
  6. Bhupendra says:
    February 2, 2022 at 12:04

    Hi Mike,

    Thanks for posting the issue. I’m also facing similar issue, Have applied Jan 2022 19.14 RU and now not able to bring up the services for the databases with 12.2.0.1 version.

    Oracle has posted to apply 33733953 patch on top of GRID 19.14 RU.

    ##Apply one-off Patch 33733953 on top of 19.14 Grid Infrastructure Home.

    But I’m not getting a direct download link.

    Reply
  7. Alejandra says:
    February 3, 2022 at 14:18

    Hi Mike,
    I found the same issue you commented on Patching Duration.
    I have already opened an SR on Mysupport several months ago but they really make it difficult….I have nothing done yet….
    My SR is SR 3-27508151421 : GRID Infrastructure – ASM – Opatch bad performance

    Reply
    • Mike.Dietrich says:
      February 4, 2022 at 16:30

      Hi Alejandra,

      thanks a lot for sharing your SR.
      At first, I reviewed your SR – and I did ask the engineer to explain why he thinks that GOOD/BAD runs were equal in timing.
      To me they seem to be different. Furthermore I told him the bug numbers we have opened for other customers – I guess he will add the bug number 33425636 to the SR as well.
      And in addition, I brought your issue to the attention to the DEV team since it looks very similar than what others see.

      I hope that we’ll get some progress soon.

      Thanks,
      Mike

      Reply
  8. Alejandra says:
    February 4, 2022 at 19:26

    Thanks a lot Mike!!!!

    Reply
  9. Paul says:
    February 12, 2022 at 00:46

    555.1 doesn’t differentiate between GI RAC and GI restart? If I look at bug note 33733953 it says it only impacts RAC but it would be nice if 555.1 also mentioned this. Bug 33550827 doesn’t say it only impacts RAC but as it concerns Ora-1093 and DLM (distributed lock manager) I assume it also is only impacts RAC systems?

    And then known issues for 19.14 note 19202201.9, in contrast to 555.1, says there are no issues.

    Reply
    • Mike.Dietrich says:
      February 27, 2022 at 20:20

      Hi Paul,

      it doesn’t – and I guess the emphasis is on GI RAC but not on RESTART.
      The other note you are quoting is more regarding know issues during apply and such.

      Cheers,
      Mike

      Reply
  10. William says:
    April 6, 2022 at 14:36

    Hi. I’m passing by as usual on this awesome blog. I just want to say. Last week a made a upgrade from 12.1.0.2 to 19.14 Grid and two RAC databases on Solaris sparc 5.11. GI ok. However using the last autoupgrade tool on my journey it wasn’t the pleasure I have. 2.5 for each database and the compilation time took 1 hour more and the end I just compile myself in another terminal after that AU follow the next step. Both db at the end left with invalid object package body call sys.dbms_swrf_report_internal. Yes I have open SR 3-29144649911 waiting and looking for response. I try compile the package nothing happens. The best part is everything else works as expected. And yes client is happy and working with both database.

    Reply
    • Mike.Dietrich says:
      April 7, 2022 at 19:43

      Hi William,

      I’m checking your SR right now.

      At first, I’m happy that you’ve received a response so quickly since you did open it with Sev.4.
      Second, this has nothing to do with AutoUpgrade. AutoUpgrade is the vehicle to upgrade your databases – but it doesn’t own the underlying scripts and procedures upgrading your database. These are two different pair of shoes.

      I see that the problem in your case is a compilation issue with the internal package:
      DBMS_SWRF_REPORT_INTERNAL

      What I miss in the SR is the upload of the upgrade logs,
      Can you please zip them together and add them to the SR:

      java -jar autoupgrade.jar -config yourconfig -zip

      AutoUpgrade can’t do miracles.

      But it is important to understand WHERE and WHEN this happens (during the regular rebuild of AWR objects or during the 19.14. datapatch run.
      Did you migrate from non-CDB to PDB as well?

      Thanks,
      Mike

      Reply
      • William says:
        April 8, 2022 at 19:49

        Hi Mike, it was only upgrade without migrate or converted to CDB/PDB. Due some errors during autoupgrade zip file

        oracle@paloma00(SATPRE1):/export/home/oracle>$ /software/app/oracle/product/19.0.0/SATPRE/jdk/bin/java -jar /software/app/oracle/product/19.0.0/SATPRE/rdbms/admin/autoupgrade.jar -config SATPRECO_config.cfg
        Internal Error has occurred. Please contact Oracle support.
        java.lang.NullPointerException
        at oracle.upgrade.autoupgrade.config.Settings.findUpgradeConfigFromSID(Settings.java:161)
        at oracle.upgrade.autoupgrade.config.links.ziplinks.ZipAlertLogs.getAlertLogPath(ZipAlertLogs.java:78)
        at oracle.upgrade.autoupgrade.config.links.ziplinks.ZipAlertLogs.process(ZipAlertLogs.java:44)

        I just uploaded to SR the AU database upgrade folder: SATPRECO1.tar

        Yes, I agree with you AU is not a miracle one, anyway.

        I keep doing my research on the subject and I Realize this two parameters init.ora plsql_code_type, plsql_optimize_level I changed to default values and run compile again and That’s was beatifull the units got compile and the registry entry to catproc was already validated too.

        SQL> alter session set plsql_code_type=’INTERPRETED’;
        Sesion modificada.
        SQL> alter session set plsql_optimize_level=2;
        Sesion modificada.
        SQL> alter package sys.dbms_swrf_report_internal compile body;
        Cuerpo del paquete modificado.
        SQL> exec dbms_registry_sys.validate_catproc;
        SQL> select comp_name,status,version from dba_registry;
        Oracle Database Packages and Types VALID 19.0.0.0.0

        Dont know why or when my client changes plsql compilation mode to Native from default INTERPRETED.

        I think the compilation time was a matter of thing during upgrade because of that.

        Best Regards

        Reply
  11. Pedro says:
    June 6, 2022 at 20:05

    I am observing a higher number of Wnnn (KTSJ) background processes after upgrading from 12.1 to 19.14.
    The Wnnn count was about ~10 to 20 in 12c, now in 19.14 is up to ~80 to 90. As a result we need to increase the PROCESSES parameter. I see same situation in databases running in 19.8 as well.
    I have not found a reference to this change in behavior yet; I wondering if anyone has and whether we need to plan for an increase in the PROCESSES parameter in the next upgrade.
    Any input is appreciated.

    Reply
    • Mike.Dietrich says:
      July 3, 2022 at 22:13

      Hi Pedro,

      that is an interesting one. At first, KTSJ background processes are, as far as I know: space management slave pool processes.

      Can you check whether you see more IO, too?

      SELECt substr(sample_time, 1,9),
      round(sum(delta_read_io_bytes)/1024/1024/1024,1) from DBA_HIST_ACTIVE_SESS_HISTORY
      where module = 'KTSJ' and sample_time > sysdate -29 group by substr(sample_time, 1,9) order by 1;

      To lower the number, you can set _max_spacebg_slaves=20 for instance.

      I found some issues but you may check via an SR please, too.

      Cheers
      Mike

      Reply
  12. Marcelo Fação says:
    August 26, 2022 at 12:40

    Hello Mike,

    this bug 33733953, oracle asks to apply the patch to GRID_HOME 19c, but they are asking me to apply this same patch to 12.2, but when downloading it, there is no option for 12.2, does that make sense? A version 19c patch being applied to 12c?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Mike Dietrich
Oracle Senior Director of Product Manager for Database Upgrade, Migrations and Patching writing just another blog.

Opinions certainly are my own as is this page.

Archives

Categories

Upgrade @YouTube

Subscribe via Email

Join 7,331 other subscribers

Follow me on Twitter