You may have read my previous blog post about patching all my environments to the January 2022 patch bundles. But since I do this in a very simple non-RAC, non-GI environment, I may not see issues you may have encountered. Some people commented – and I would like to share Some additional information regarding the Oracle 19.14.0 RUs . Still, please, this is NOT a collection of all open issues. I just see this blog post as a summary of the issues I have seen and/or heard about so far.
555.1 is one of the most important MOS notes
At first, before you read the about the topics below, make sure you have checked one of THE most important MOS notes:
And please, add it to your favorites as well (the upper left star in the MOS note). It lists the most important issues for each of the patch bundles and the fixes you should consider to put on top of any of them.
ACFS RU issue
Important information:
The Development team tried to reproduce the below case but didn’t succeed. In addition, please be aware that the initial setup was a supported combination (kernel version <=> ACFS version) while the result at the end isn’t a supported mix of kernel and ACFS. So this isn’t a solution you should attempt.
Nariman reported this issue to me via the blog (and thanks for sharing it with me including the solution). He had a significant problem with applying 19.14.0 to his environment. At first, his apply of 19.14.0 failed in the CRS part already. Then he tried to install each of the sub-components of the patch bundle separately and found that the fail is caused by the ACFSRU, patch 33534448.
- 33515361- ok
- This is the Database 19.14.0 RU
- 33529556- ok
- 33534448- fails
- This is the ACFS 19.14.0 RU
- 33239955- ok
- This is the Tomcat 19.0.0.0.0 RU
- 33575402- ok
- This is the WLM / QoS 19.14.0 RU
The key error message was:
2022/01/20 12:46:28 CLSRSC-400: A system reboot is required to continue installing. Argument “” isn’t numeric in numeric ne (!=) at /u01/app/grid/19/grid/lib/acfslib.pm line 347.
But the system reboot did not solve the problem.
The solution finally as Nariman told me was:
As soon as I updated kernel, it was upgraded from “4.18.0-305.19.1.el8_4.x86_64” to “4.18.0-348.12.2.el8_5.x86_64” and the patching went perfect!
I wouldn’t have been considered a kernel update as a potential solution – but very happy that it fixed it.
GI Patching Alert: CRS-5037
This is something you can find already in the comments section of Patching all my environments with the January 2022 Patch Bundles. And at the same time, my dear ACS colleague Thomas alerted me on slack as well about this issue.
In this note you can read that with 19.14 GI, user-defined database services may fail to start with error CRS-5037: Database does not have the required fix for bug ‘31143870’. Not everybody is going to see this issue. The MOS note explains that only configurations satisfying all 3 conditions listed below may experience the issue:
- Grid Infrastructure Release “19.14”
- Contains database(s) running on “RDBMS Releases <= 18c” and/or “RDBMS Releases 19.3 to 19.10 without Patch 31143870”
- Has one or more user-defined services NOT created on PDB. i.e. “crsctl stat res ora.db_name.service_name.svc -f | grep PLUGGABLE_DATABSE” will show null value for PLUGGABLE_DATABSE key such as:
$ crsctl stat res ora.database_name.service_name.svc -f | grep PLUGGABLE_DATABASE PLUGGABLE_DATABASE= $
And a solution is to apply patch 33733953 beforehand.
Meanwhile, this issue got added to MOS Note: 555.1 as well:
Gridsetup SwitchGridHome fails
This is actually an issue which came up with 19.13.0 already. You may encounter it only when you do out-of-place GI patching. When you patch in-place you won’t see this bug. Still, this isn’t fixed in 19.14.0 yet but only from 19.15.0 onward.
It is labeled as:
- Bug 33601195 : GRIDSETUP -SWITCHGRIDHOME OOP – ROOT.SH GENERATED INCORRECTLY
And there are two good news for you. At first, you can download a patch 33601195 on top of 19.13.0 if you plan to patch GI out-of-place to 19.13.0. But as of now I can’t see any patch on top of 19.14.0 yet.
What’s the other good news? Well, you can use this proven workaround:
- Node_1: Execute $ORACLE_HOME/root.sh
- Node_1: scp $ORACLE_HOME/crs/config/rootconfig.sh Node_2:$ORACLE_HOME/crs/config/rootconfig.sh
- Node_1: scp $ORACLE_HOME/crs/install/crsconfig_params Node_2:$ORACLE_HOME/ crs/install/crsconfig_params
- Node_2: Execute $ORACLE_HOME/root.sh
And of course, if you have more than 2 nodes, you must repeat the copy operation to the other nodes as well and execute root.sh there, too.
Patching Duration
Several people over the past months complained about patching duration. If this applies to the opatch or opatchauto part of the patching, please open an SR. As I commented already in Patching all my environments with the January 2022 Patch Bundles, it takes a significant long time in my 19c environment while the 21c or the 12.2.0.1 environments get patches fairly quick. I see minute-long waits during the opatch process but I haven’t found either the root cause or a solution yet. Opatch 28 (6880880) is supposed to fix one of these issues – but since I used opatch 28 for my environments, I haven’t seen it completing faster.
For the datapatch part of the patching, I will have another blog post within the coming days.
Further Links and Information
- Patching all my environments with the January 2022 Patch Bundles
- MOS Note: 2835152.1 – ALERT: Database Service Fails to Start With “CRS-5037: Database does not have the required fix for bug ‘31143870’” AFTER Grid Infrastructure Patching or Upgrade to 19.14 RU
- MOS Note:555.1
–Mike
Hi Mike,
Interesting Narimans issue with ACFS and updating kernel to “4.18.0-348.12.2.el8_5.x86_64”. I had just been looking at ACFS certification yesterday and noticed that ACFS is only certified up to kernel 4.18.0-339 at the moment with patch 33535435 and only got a patch for 19.13 at the moment 🙁
Hi Andrew,
You are absolutely right, but the thing is I’m not using ACFS in my env. That’s why the patching went flaweless. If I configured ACFS, it would 100% have been failed because of the unsupported kernel 🙂
1369107.1 – really good doc
Hi Mike,
datapatch failed after migrating from Solaris to Linux because datapatch tried to create dbms_hadoop* packages, which depend on dba_hive* views. dba_hive* views aren’t created on Solaris x86. The following article contains more details: https://nenadnoveljic.com/blog/compilation-error-in-19-14-datapatch/
Hi Nenad.
thanks for sharing – just read your blog post 🙁
Did you receive a bug number or in case you haven’t, would you please mind to share the SR with me (via email mike.dietrich — at — oracke.com or whatever works fine for you)
Thanks,
Mike
Hi Mike,
no bug was opened. I shared SR vie e-mail.
Best regards,
Nenad
Thanks Nenad – I will get back to you via email.
Cheers,
Mike
HI Mike,
If 555.1 is important , so is 1369107.1… Could you ask them to update it with the latest info?
Hi PeT,
unfortunately I have no control on the ACFS note owners:
Oracle Support Document 1369107.1 (ACFS Support On OS Platforms (Certification Matrix).) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1369107.1
But please you can always place a comment for a note.
It got updated on Jan 28, the day you placed your comment here.
Can you check whether the information you were looking for is there now?
Thx,
Mike
ok
How confusing. I dont see mention of the “Nariman” bug in MOS note 555.1? uname -a shows “4.1.12-124.53.5.1.el7uek.x86_64” on our system – how do I know if our patching will succeed? I am wanting to upgrade from 12.2 to 19c and use -applyRU to apply the 19.14RU.
Bug 33733953 says it is only applicable to RAC systems – so as we are using Oracle restart I assume I can ignore this bug/patch.
Bug 33601195 – we are installing a new 19c grid home and applying 19.14 RU using -applyRU. Do we need to worry about this bug?
Hi Paul,
actually I added an important comment to make this more clear. You should be in a supported kernel/acfs configuration. Dev couldn’t reproduce the issue in this case, and they tried really hard.
Cheers,
Mike
PS: Into 555.1 only very critical issues which may affect the majority of our customers will be added usually.
Hi Mike,
Thanks for posting the issue. I’m also facing similar issue, Have applied Jan 2022 19.14 RU and now not able to bring up the services for the databases with 12.2.0.1 version.
Oracle has posted to apply 33733953 patch on top of GRID 19.14 RU.
##Apply one-off Patch 33733953 on top of 19.14 Grid Infrastructure Home.
But I’m not getting a direct download link.
Hi Bhupendra,
the fix is there since a week – I’m just not updating and answering the blog quickly enough.
Cheers,
Mike
Hi Mike,
I found the same issue you commented on Patching Duration.
I have already opened an SR on Mysupport several months ago but they really make it difficult….I have nothing done yet….
My SR is SR 3-27508151421 : GRID Infrastructure – ASM – Opatch bad performance
Hi Alejandra,
thanks a lot for sharing your SR.
At first, I reviewed your SR – and I did ask the engineer to explain why he thinks that GOOD/BAD runs were equal in timing.
To me they seem to be different. Furthermore I told him the bug numbers we have opened for other customers – I guess he will add the bug number 33425636 to the SR as well.
And in addition, I brought your issue to the attention to the DEV team since it looks very similar than what others see.
I hope that we’ll get some progress soon.
Thanks,
Mike
Thanks a lot Mike!!!!
You are welcome – and I hope that we’ll get some progress.
Thanks,
Mike
555.1 doesn’t differentiate between GI RAC and GI restart? If I look at bug note 33733953 it says it only impacts RAC but it would be nice if 555.1 also mentioned this. Bug 33550827 doesn’t say it only impacts RAC but as it concerns Ora-1093 and DLM (distributed lock manager) I assume it also is only impacts RAC systems?
And then known issues for 19.14 note 19202201.9, in contrast to 555.1, says there are no issues.
Hi Paul,
it doesn’t – and I guess the emphasis is on GI RAC but not on RESTART.
The other note you are quoting is more regarding know issues during apply and such.
Cheers,
Mike
Hi. I’m passing by as usual on this awesome blog. I just want to say. Last week a made a upgrade from 12.1.0.2 to 19.14 Grid and two RAC databases on Solaris sparc 5.11. GI ok. However using the last autoupgrade tool on my journey it wasn’t the pleasure I have. 2.5 for each database and the compilation time took 1 hour more and the end I just compile myself in another terminal after that AU follow the next step. Both db at the end left with invalid object package body call sys.dbms_swrf_report_internal. Yes I have open SR 3-29144649911 waiting and looking for response. I try compile the package nothing happens. The best part is everything else works as expected. And yes client is happy and working with both database.
Hi William,
I’m checking your SR right now.
At first, I’m happy that you’ve received a response so quickly since you did open it with Sev.4.
Second, this has nothing to do with AutoUpgrade. AutoUpgrade is the vehicle to upgrade your databases – but it doesn’t own the underlying scripts and procedures upgrading your database. These are two different pair of shoes.
I see that the problem in your case is a compilation issue with the internal package:
DBMS_SWRF_REPORT_INTERNAL
What I miss in the SR is the upload of the upgrade logs,
Can you please zip them together and add them to the SR:
java -jar autoupgrade.jar -config yourconfig -zip
AutoUpgrade can’t do miracles.
But it is important to understand WHERE and WHEN this happens (during the regular rebuild of AWR objects or during the 19.14. datapatch run.
Did you migrate from non-CDB to PDB as well?
Thanks,
Mike
Hi Mike, it was only upgrade without migrate or converted to CDB/PDB. Due some errors during autoupgrade zip file
oracle@paloma00(SATPRE1):/export/home/oracle>$ /software/app/oracle/product/19.0.0/SATPRE/jdk/bin/java -jar /software/app/oracle/product/19.0.0/SATPRE/rdbms/admin/autoupgrade.jar -config SATPRECO_config.cfg
Internal Error has occurred. Please contact Oracle support.
java.lang.NullPointerException
at oracle.upgrade.autoupgrade.config.Settings.findUpgradeConfigFromSID(Settings.java:161)
at oracle.upgrade.autoupgrade.config.links.ziplinks.ZipAlertLogs.getAlertLogPath(ZipAlertLogs.java:78)
at oracle.upgrade.autoupgrade.config.links.ziplinks.ZipAlertLogs.process(ZipAlertLogs.java:44)
I just uploaded to SR the AU database upgrade folder: SATPRECO1.tar
Yes, I agree with you AU is not a miracle one, anyway.
I keep doing my research on the subject and I Realize this two parameters init.ora plsql_code_type, plsql_optimize_level I changed to default values and run compile again and That’s was beatifull the units got compile and the registry entry to catproc was already validated too.
SQL> alter session set plsql_code_type=’INTERPRETED’;
Sesion modificada.
SQL> alter session set plsql_optimize_level=2;
Sesion modificada.
SQL> alter package sys.dbms_swrf_report_internal compile body;
Cuerpo del paquete modificado.
SQL> exec dbms_registry_sys.validate_catproc;
SQL> select comp_name,status,version from dba_registry;
Oracle Database Packages and Types VALID 19.0.0.0.0
Dont know why or when my client changes plsql compilation mode to Native from default INTERPRETED.
I think the compilation time was a matter of thing during upgrade because of that.
Best Regards
Hi William,
thanks for the hint with the parameters. I told the team about it – we may need a check for this.
Thanks
Mike
I am observing a higher number of Wnnn (KTSJ) background processes after upgrading from 12.1 to 19.14.
The Wnnn count was about ~10 to 20 in 12c, now in 19.14 is up to ~80 to 90. As a result we need to increase the PROCESSES parameter. I see same situation in databases running in 19.8 as well.
I have not found a reference to this change in behavior yet; I wondering if anyone has and whether we need to plan for an increase in the PROCESSES parameter in the next upgrade.
Any input is appreciated.
Hi Pedro,
that is an interesting one. At first, KTSJ background processes are, as far as I know: space management slave pool processes.
Can you check whether you see more IO, too?
SELECt substr(sample_time, 1,9),
round(sum(delta_read_io_bytes)/1024/1024/1024,1) from DBA_HIST_ACTIVE_SESS_HISTORY
where module = 'KTSJ' and sample_time > sysdate -29 group by substr(sample_time, 1,9) order by 1;
To lower the number, you can set _max_spacebg_slaves=20 for instance.
I found some issues but you may check via an SR please, too.
Cheers
Mike
Hello Mike,
this bug 33733953, oracle asks to apply the patch to GRID_HOME 19c, but they are asking me to apply this same patch to 12.2, but when downloading it, there is no option for 12.2, does that make sense? A version 19c patch being applied to 12c?
Hi Marcelo,
no, this does not make any sense to me. I hope you’ve had a chance to clarify this with Oracle Support meanwhile.
Cheers,
Mike