Groundhog day on the upgrade blog. It’s time for my quaterly Patching all my environments with the January 2021 Patch Bundles blog post. And still no RAC and ASM. Sorry for that … too many virtual events and other tasks since months unfortunately.
As usual, an important annotation upfront: I patch in-place due to space issues. But in reality, you please patch always out-of-place with a separate home. Please see this blog post about how to apply the RU directly when you provision a new home with OUI.
Security Alert January 2021
Find the Security Alert for January 2021 here. And don’t forget to take a look at the Oracle Database Server Risk Matrix for January 2021. It is relatively short this time but unfortunately lists a 8.8 score with the RDBMS Scheduler.
Oracle Database Server Risk Matrix January 2021
Still, the list is much shorter than the one from October 2020.
Database Patch Bundles
You will find the links to the individual patch bundles in MOS Note: 2725756.1 – Critical Patch Update (CPU) Program Jan 2021 Patch Availability Document (PAD). And please note that a patch number in the document does not necessarily mean that your patch is available already. Please find a discussion about this recurring topic here: Why is the release update not available on my platform yet? Check especially the Section 2.2 (Post Release Patches).
- Oracle Database 19c
- Database Release Update 19.10.0.0.210119 Patch 32218454 for UNIX (1.4GB on Linux)
- Readme
- List of fixes: MOS Note: 2523220.1 (as usual, it does not include any 19.10.0 fixes while I write this blog post unfortunately)
- Database Release Update 19.10.0.0.210119 Patch 32218454 for UNIX (1.4GB on Linux)
- Oracle Database 12.2.0.1
- Database Jan2021 Release Update 12.2.0.1.210119 Patch 32228578 for UNIX (817MB on Linux)
- Readme
- List of fixes: MOS Note: 2245178.1 (also not updated yet)
- Database Jan2021 Release Update 12.2.0.1.210119 Patch 32228578 for UNIX (817MB on Linux)
- Oracle Database 11.2.0.4
- Ups … yes, there is no patch bundle for 11.2.0.4 anymore.
That is correct. You may be aware that 11.2.0.4 went out of Extended Support on Dec 31, 2020.
If you still need bug fixing support for Oracle 11.2.0.4, then please check out the Market Driven Support offering.
- Ups … yes, there is no patch bundle for 11.2.0.4 anymore.
No news as MOS Note: 2118136.2 – Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases hasn’t been updated yet with the links for the January 2021 bundles by the time I write this.
Do I need a new OPatch?
First check while the download is progressing: Do I need to refresh my OPatch versions?
- 19.10.0 requires opatch 12.2.0.1.23 or later
- 12.2.0.1 requires opatch 12.2.0.1.23 or later
In my case I need to update opatch in both homes. But I realize also that the opatch download has been simplified. So your feedback was heard. Just choose “19.0.0.0.0” version and you get the Nov 2020 Opatch:
Download OPatch via: 6880880. Wipe out your current OPatch directories in your homes. Once you unzip the new OPatch bundles. you can proceed.
Applying RU 19.10.0 to my 19c home
At first, I unzip the patch into a separate directory and place myself into this directory ~/31741641 .
- patch conflict check
$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 12.2.0.1.23 Copyright (c) 2021, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.23 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2021-01-20_00-46-32AM_1.log Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed. OPatch succeeded.
- opatch apply
[CDB2] oracle@hol:~/patch/32218454 $ $ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.23 Copyright (c) 2021, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.23 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2021-01-20_00-50-39AM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 32218454 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/19') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '32218454' to OH '/u01/app/oracle/product/19' ApplySession: Optional component(s) [ oracle.network.gsm, 19.0.0.0.0 ] , [ oracle.rdbms.ic, 19.0.0.0.0 ] , [ oracle.rdbms.tg4db2, 19.0.0.0.0 ] , [ oracle.tfa, 19.0.0.0.0 ] , [ oracle.net.cman, 19.0.0.0.0 ] , [ oracle.network.cman, 19.0.0.0.0 ] , [ oracle.oid.client, 19.0.0.0.0 ] , [ oracle.options.olap.api, 19.0.0.0.0 ] , [ oracle.options.olap, 19.0.0.0.0 ] , [ oracle.xdk.companion, 19.0.0.0.0 ] , [ oracle.jdk, 1.8.0.191.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.rdbms, 19.0.0.0.0... Patching component oracle.rdbms.rsf, 19.0.0.0.0... Patching component oracle.rdbms.util, 19.0.0.0.0... Patching component oracle.assistants.acf, 19.0.0.0.0... Patching component oracle.assistants.deconfig, 19.0.0.0.0... Patching component oracle.assistants.server, 19.0.0.0.0... Patching component oracle.buildtools.rsf, 19.0.0.0.0... Patching component oracle.ctx, 19.0.0.0.0... Patching component oracle.dbjava.ic, 19.0.0.0.0... Patching component oracle.dbjava.jdbc, 19.0.0.0.0... Patching component oracle.dbjava.ucp, 19.0.0.0.0... Patching component oracle.dbtoolslistener, 19.0.0.0.0... Patching component oracle.ldap.owm, 19.0.0.0.0... Patching component oracle.ldap.rsf, 19.0.0.0.0... Patching component oracle.network.rsf, 19.0.0.0.0... Patching component oracle.oracore.rsf, 19.0.0.0.0... Patching component oracle.rdbms.dbscripts, 19.0.0.0.0... Patching component oracle.rdbms.deconfig, 19.0.0.0.0... Patching component oracle.sdo, 19.0.0.0.0... Patching component oracle.sdo.locator.jrf, 19.0.0.0.0... Patching component oracle.sqlplus, 19.0.0.0.0... Patching component oracle.xdk, 19.0.0.0.0... Patching component oracle.marvel, 19.0.0.0.0... Patching component oracle.xdk.rsf, 19.0.0.0.0... Patching component oracle.ctx.atg, 19.0.0.0.0... Patching component oracle.rdbms.scheduler, 19.0.0.0.0... Patching component oracle.rdbms.lbac, 19.0.0.0.0... Patching component oracle.duma, 19.0.0.0.0... Patching component oracle.ldap.rsf.ic, 19.0.0.0.0... Patching component oracle.odbc, 19.0.0.0.0... Patching component oracle.ctx.rsf, 19.0.0.0.0... Patching component oracle.oraolap.api, 19.0.0.0.0... Patching component oracle.xdk.parser.java, 19.0.0.0.0... Patching component oracle.oraolap, 19.0.0.0.0... Patching component oracle.sdo.locator, 19.0.0.0.0... Patching component oracle.sqlplus.ic, 19.0.0.0.0... Patching component oracle.mgw.common, 19.0.0.0.0... Patching component oracle.ons, 19.0.0.0.0... Patching component oracle.dbdev, 19.0.0.0.0... Patching component oracle.network.listener, 19.0.0.0.0... Patching component oracle.nlsrtl.rsf, 19.0.0.0.0... Patching component oracle.ovm, 19.0.0.0.0... Patching component oracle.oraolap.dbscripts, 19.0.0.0.0... Patching component oracle.xdk.xquery, 19.0.0.0.0... Patching component oracle.precomp.rsf, 19.0.0.0.0... Patching component oracle.javavm.client, 19.0.0.0.0... Patching component oracle.precomp.common.core, 19.0.0.0.0... Patching component oracle.ldap.security.osdt, 19.0.0.0.0... Patching component oracle.rdbms.oci, 19.0.0.0.0... Patching component oracle.rdbms.rman, 19.0.0.0.0... Patching component oracle.rdbms.crs, 19.0.0.0.0... Patching component oracle.rdbms.install.common, 19.0.0.0.0... Patching component oracle.javavm.server, 19.0.0.0.0... Patching component oracle.rdbms.drdaas, 19.0.0.0.0... Patching component oracle.rdbms.install.plugins, 19.0.0.0.0... Patching component oracle.rdbms.dv, 19.0.0.0.0... Patching component oracle.ldap.client, 19.0.0.0.0... Patching component oracle.network.client, 19.0.0.0.0... Patching component oracle.rdbms.rsf.ic, 19.0.0.0.0... Patching component oracle.precomp.common, 19.0.0.0.0... Patching component oracle.precomp.lang, 19.0.0.0.0... Patching component oracle.jdk, 1.8.0.201.0... Patch 32218454 successfully applied. Sub-set patch [31771877] has become inactive due to the application of a super-set patch [32218454]. Please refer to Doc ID 2161861.1 for any possible further required actions. Log file location: /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2021-01-20_00-50-39AM_1.log OPatch succeeded.
- datapatch
Don’t forget to startup all your PDBs before invoking datapatch.[CDB2] oracle@hol:~/patch/32218454 $ $ORACLE_HOME/OPatch/datapatch -verbose SQL Patching tool version 19.10.0.0.0 Production on Wed Jan 20 01:00:14 2021 Copyright (c) 2012, 2020, Oracle. All rights reserved. Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_19749_2021_01_20_01_00_14/sqlpatch_invocation.log Connecting to database...OK Gathering database info...done Note: Datapatch will only apply or rollback SQL fixes for PDBs that are in an open state, no patches will be applied to closed PDBs. Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1) Bootstrapping registry and package to current versions...done Determining current state...done Current state of interim SQL patches: No interim patches found Current state of release update SQL patches: Binary registry: 19.10.0.0.0 Release_Update 210108185017: Installed PDB CDB$ROOT: Applied 19.9.0.0.0 Release_Update 200930183249 successfully on 21-OCT-20 11.56.37.379130 AM PDB PDB$SEED: Applied 19.9.0.0.0 Release_Update 200930183249 successfully on 21-OCT-20 11.56.37.581802 AM Adding patches to installation queue and performing prereq checks...done Installation queue: For the following PDBs: CDB$ROOT PDB$SEED No interim patches need to be rolled back Patch 32218454 (Database Release Update : 19.10.0.0.210119 (32218454)): Apply from 19.9.0.0.0 Release_Update 200930183249 to 19.10.0.0.0 Release_Update 210108185017 No interim patches need to be applied Installing patches... Patch installation complete. Total patches installed: 2 Validating logfiles...done Patch 32218454 apply (pdb CDB$ROOT): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/32218454/24018797/32218454_apply_CDB2_CDBROOT_2021Jan20_01_00_45.log (no errors) Patch 32218454 apply (pdb PDB$SEED): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/32218454/24018797/32218454_apply_CDB2_PDBSEED_2021Jan20_01_02_28.log (no errors) SQL Patching tool complete on Wed Jan 20 01:04:03 2021
Everything looks good.
Finally I just check quickly the contents of the CDB_REGISTRY_SQLPATCH view:
CON_ID ACTION_TIME PATCH_ID PATCH_TYPE ACTION DESCRIPTION SOURCE_VERSI TARGET_VERSI ---------- -------------------- ---------- ---------- ---------- ------------------------------------------------------ ------------ ------------ 1 2019-04-28 29517242 RU APPLY Database Release Update : 19.3.0.0.190416 (29517242) 19.1.0.0.0 19.3.0.0.0 1 2019-10-16 30125133 RU APPLY Database Release Update : 19.5.0.0.191015 (30125133) 19.3.0.0.0 19.5.0.0.0 1 2020-01-21 30557433 RU APPLY Database Release Update : 19.6.0.0.200114 (30557433) 19.5.0.0.0 19.6.0.0.0 1 2020-04-15 30869156 RU APPLY Database Release Update : 19.7.0.0.200414 (30869156) 19.6.0.0.0 19.7.0.0.0 1 2020-07-15 31281355 RU APPLY Database Release Update : 19.8.0.0.200714 (31281355) 19.7.0.0.0 19.8.0.0.0 1 2020-10-21 31771877 RU APPLY Database Release Update : 19.9.0.0.201020 (31771877) 19.8.0.0.0 19.9.0.0.0 1 2021-01-20 32218454 RU APPLY Database Release Update : 19.10.0.0.210119 (32218454) 19.9.0.0.0 19.10.0.0.0 2 2019-04-28 29517242 RU APPLY Database Release Update : 19.3.0.0.190416 (29517242) 19.1.0.0.0 19.3.0.0.0 2 2019-10-16 30125133 RU APPLY Database Release Update : 19.5.0.0.191015 (30125133) 19.3.0.0.0 19.5.0.0.0 2 2020-01-21 30557433 RU APPLY Database Release Update : 19.6.0.0.200114 (30557433) 19.5.0.0.0 19.6.0.0.0 2 2020-04-15 30869156 RU APPLY Database Release Update : 19.7.0.0.200414 (30869156) 19.6.0.0.0 19.7.0.0.0 2 2020-07-15 31281355 RU APPLY Database Release Update : 19.8.0.0.200714 (31281355) 19.7.0.0.0 19.8.0.0.0 2 2020-10-21 31771877 RU APPLY Database Release Update : 19.9.0.0.201020 (31771877) 19.8.0.0.0 19.9.0.0.0 2 2021-01-20 32218454 RU APPLY Database Release Update : 19.10.0.0.210119 (32218454) 19.9.0.0.0 19.10.0.0.0 14 rows selected.
Applying RU 12.2.0.1.210119 to my 12.2.0.1 home
I won’t copy/paste all steps as the output is similar to the above.
- opatch conflict check
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./
- opatch apply
$ORACLE_HOME/OPatch/opatch apply
Then I will need to startup my databases and all PDBs.
- datapatch -verbose
Final steps is “datapatch”. It requires the databases and all its PDBs to be up and running.
No issues seen. All worked fine. But there is more to come. My databases have OJVM – and I will patch it tomorrow.
Addition Mar 2, 2021
Frits Hogland raised and issue to Anil Nair, our RAC PM an myself wondering why applying the Jan 2021 Grid Infrastructure (GI) RU adds a strange name to the inventory. As Frits demonstrated:
Before Patching
[oracle ~]$ /u01/app/19.0.0/grid/OPatch/opatch lspatches -oh /u01/app/oracle/product/19.0.0/db_1
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
After Patching
[oracle ~]$ $ORACLE_HOME/OPatch/opatch lspatches -oh /u01/app/oracle/product/19.0.0/db_1
32222571;OCW Interim patch for 32222571
32218454;Database Release Update : 19.10.0.0.210119 (32218454)
Anil investigated and we both learned that this was known already but internally we took a decision to not reupload the entire RU for GI. Hence, no issue with this since the patch number is correct and guides you to the RU and not a one-off or interim patch of course. With 19.11.0 this shouldn’t happen again.
Addition Apr 12, 2021
Several people reported issued when trying to apply this RU on IBM AIX. It leads to a Java Nullpointer Exception. This is an issue which is described here:
Workaround is to use a previous version of opatch which is linked from MOS Note: 2756755.1.
More Information and Links
- Security Alert for January 2021
- Oracle Database Server Risk Matrix for January 2021
- Patching all my environments with the October 2020 Patch Bundles
- MOS Note: 2725756.1 – Critical Patch Update (CPU) Program Jan 2021 Patch Availability Document (PAD)
- Why is the release update not available on my platform yet?
- Market Driven Support for Oracle Database 11.2.0.4
- OPatch Download via patch 6880880
- MOS Note: 2756755.1 – AIX OPatch 12.2.0.1.24 Version Fails with “Opatchcore binary error message=Assertion” and “VMState: 0x0005ff0a”
–Mike
Hi Mike,
I think you meant “19.10”:
“Applying RU 19.9.0 to my 19c home”
Yes, classical copy&paste flaw in the middle of the night 😉
Thanks – and I corrected it.
Cheers,
Mike
Mike, did you use OPatch 12.2.0.1.23 for DB 19.x releases (Nov 2020) for versions 12.1.x and 12.2.x?
For Microsoft Windows x64 64bit, I see a few options:
OPatch 12.2.0.1.23 for Db 12.x, 18.x, 19.x and 20.x releases (Nov 2020)
OPatch 12.2.0.1.23 for Db 12.x releases (Nov 2020)
OPatch 12.2.0.1.23 for DB 19.x releases (Nov 2020)
OPatch 12.2.0.1.23 for DB 20.x releases (Nov 2020)
Regards,
jorge
Hi Jorge,
I used the Nov 20 version 12.2.0.1.23
The versions:
OPatch 12.2.0.1.23 for Db 12.x, 18.x, 19.x and 20.x releases (Nov 2020)
OPatch 12.2.0.1.23 for Db 12.x releases (Nov 2020)
OPatch 12.2.0.1.23 for DB 19.x releases (Nov 2020)
OPatch 12.2.0.1.23 for DB 20.x releases (Nov 2020)
are all exactly the same. Not a zero byte difference 😉
Cheers,
Mike
Mike,
Couple of questions. Have you heard from any customer about:
a) Jan 2021 patch to 12.1.0.2 database in Oracle restart environment failing. This is in lab setup, so I cannot open SR. Sub-patch 22806133 fails to apply. Oct 2020 patch works fine.
b) CDB_REGISTRY_SQLPATCH in 19c RAC database fails to return any rows. I have opened a SR 3-25036363911 where I have identified the root cause at high level, but nothing from support.
Please also take a look at SR 3-24659178261. One-off Oracle Text patch is causing OJVM to show as not applied.
Thanks,
Arun
Hi Arun,
you can open SRs for test installations of course as well. I don’t see a reason why you shouldn’t be able (at least from our side).
Please escalate the SR if you haven’t gotten a solution yet. Raise the Severity to Sev.1, request a callback from a manager if needed.
Cheers,
Mike
Hi Mike,
Oracle recommends to perform manual relinking of Oracle Home Binaries after Patching. What do you practice or recommend?
Interesting. Who’s “Oracle” in this case? Oracle Support? Or the documentation?
Check the readme:
https://updates.oracle.com/Orion/Services/download?type=readme&aru=24018797
It doesn’t say anything about relinking.
I don’t think that you will need to manually relink.
Cheers,
Mike
Hi Mike,
Installing the latest Grid Infrastructure Release Update (Patch 32226491 Grid Infrastructure Release Update 12.2.0.1.210119) and having a question about DB RU under grid_home.
Current configuration is non-RAC, it’s Oracle Restart home.
According ReadMe (https://updates.oracle.com/Orion/Services/download?type=readme&aru=24022381), for Database Jan 2021 Release Update 12.2.0.1.210119 (Patch 32228578) applicable home -> “Only DB home for Oracle Non-RAC setup. Both DB homes and Grid home for Oracle RAC setup.”. So, according these sentence there is no need to install DB RU under grid_home in our case.
Installation performing with the following command, as it said in ReadMe:
# opatchauto apply /32226491 -oh
The same command is also in My Oracle Support Document 2246888.1, point 2.3 Patch Installation, Case 4: Patching Oracle Restart Home:
# opatchauto apply /%BUGNO%
But as a result of such command, DB RU was installed anyway under grid_home. Why it’s happened?
Or does the sentence “Only DB home for Oracle Non-RAC setup. Both DB homes and Grid home for Oracle RAC setup.” mean something else?
Hi Bill,
you struggle with the same wording as I do. Please raise this with Oracle Support. I opted for more “readable” readmes but failed.
Cheers,
Mike
Hi Mike,
Oracle Database 19c Release Update & Release Update Revision January 2021 Critical Issues (Doc ID 2725758.1) mentions to install Patch 31862593 also. It’s not clear for me whether it should be installed only on Database Homes or GI Home also.
Regards,
Kálmán
Hi,
thanks a lot for this hint – I wasn’t aware of this issue.
See here now:
https://mikedietrichde.com/2021/02/01/you-may-need-a-one-off-for-dbms_optim_bundle-in-19-10-0/
And certainly, you WON’T need this patch in GI.
Cheers,
Mike
Did you try patching 19c RAC nodes with Jan 2021 RU? So far not so smooth…
Hi Guy,
as I apologized already at the beginning of my blog post, I’m still not testing in a RAC environment unfortunately.
Cheers,
Mike
Hi,
we had some troubles with the Jan 2021 RU on RAC too.
Maybe Patch 32242453 – ASM FAILED TO START DUE TO HAIP NETWORK CHANGE AFTER APPLYING THE 19.9 RU
helps you too. ASM fails to start or clusterware crashes without this patch on 19.10 for us.
Regards
Sven
Thanks Sven!
Cheers,
Mike
Hi Mike,
32228578
31802727
32231681
26839277
32253903
These are the sub patches in Patch 32226491 – GI Jan 2021 Release Update 12.2.0.1.210119..
Upon installation using opatchauto – 32253903 doesnt get installed.
But for OCT2020 PSU , the Tomcat subpatch did get installed
Log: /oracle/product/grid/12.2.0.1/cfgtoollogs/opatchauto/core/opatch/opatch2021-02-04_22-54-57PM_1.log
Reason: /opt/stage/patches/32226491/32253903 is not required to be applied to oracle home /oracle/product/grid/12.2.0.1
Could you please advise
Hi Midhun,
please raise an SR – I can’t give Support on patch-apply issues unfortunately – at least not if I don’t have the environment in place.
Cheers,
Mike
Hi Mike,
first: thanks for your very interesting todays webcast, very valuable!
But I also have a question concerning 19.10.0 RU: couldnt find anything for AIX on POWER (64-bit) – I know, very exotic – neither for RDBMS nor for GI. Any idea?
Thanks
Axel D.
Hi Axel,
I know that the date has been slipped twice already – but I have no further details.
Cheers,
Mike
Hi Axel,
Are you able to apply latest released 19.10 RU for AIX on POWER (64bit) ?
we are having issues and wondering if this is common across board
Thanks,
Ramanan
Hi Ramanan, indeed I do have a problem just trying the “One-off Patch Conflict Detection and Resolution” of 32226239. The opatch crashes with java core dump. I already opened a SR for this, maybe its a problem of the latest opatch utility 12.2.0.1.24, Im currently investigating this. What kind of problem do you have?
Regards
Axel D.
Hi, Mike,
We did patch 19c RAC environment using RU 19.10. But I noticed OCW is Interim patch for 32222571 when running opatch lspatches command. But the 19.10 release note it is not a Interim Patch. Which one is right?
Thanks,
Weidong
Hi Weidong,
this is a “mistake” in the update of the inventory. I think, it gets or got added to the “Known Issues with 19.10.0 RU” note already.
You can safely ignore this – it is an RU, and not an interim patch.
Sorry for the inconvenience!
Cheers,
Mike
Hi Mike
We were challenged to apply RU 19.10 on AIX power 64bit though it was released after a month of original release date , we are having issues during Opatchauto apply , did you hear this is common? I am wondering it’s not specific to our environment
Thanks
Ramanan
Hi Ramanan,
no, I haven’t heard anything about it yet even though some of my customers asked for it as well. But I haven’t seen feedback.
Please open an SR with Oracle Support.
Cheers,
Mike
Hi Mike,
I have applied Jan 2021 19.10.0.0 patches into my 19c database running on Windows 2012 R2.
However, when I ran datapatch -verbose to push the changes in to my PDBs I have encountered below error –
###############################################################
C:\app\Employee\product\19C\dbhome_1\OPatch>datapatch -verbose
SQL Patching tool version 19.3.0.0.0 Production on Fri Mar 5 19:20:56 2021
Copyright (c) 2012, 2019, Oracle. All rights reserved.
Log file for this invocation: C:\app\Employee\cfgtoollogs\sqlpatch\sqlpatch_7924_2021_03_05_19_20_57\sqlpatch_invocation.log
Connecting to database…OK
Gathering database info…done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions…done
Determining current state…done
Current state of interim SQL patches:
Interim patch 30805684 (OJVM RELEASE UPDATE: 19.7.0.0.200414 (30805684)):
Binary registry: Not installed
PDB CDB$ROOT: Rolled back successfully on 25-NOV-20 12.14.21.917000 AM
PDB HCM92UAT: Rolled back successfully on 25-NOV-20 12.14.30.452000 AM
PDB PDB$SEED: Rolled back successfully on 25-NOV-20 12.14.26.229000 AM
Interim patch 31668882 (OJVM RELEASE UPDATE: 19.9.0.0.201020 (31668882)):
Binary registry: Not installed
PDB CDB$ROOT: Rolled back successfully on 05-MAR-21 01.04.45.000000 PM
PDB HCM92UAT: Rolled back successfully on 05-MAR-21 01.04.50.909000 PM
PDB PDB$SEED: Rolled back successfully on 05-MAR-21 01.04.47.990000 PM
Interim patch 32067171 (OJVM RELEASE UPDATE 19.10.0.0.0 (32067171)):
Binary registry: Installed
PDB CDB$ROOT: Rolled back successfully on 05-MAR-21 04.17.54.389000 PM
PDB HCM92UAT: Rolled back successfully on 05-MAR-21 04.17.54.480000 PM
PDB PDB$SEED: Rolled back successfully on 05-MAR-21 04.17.54.463000 PM
Current state of release update SQL patches:
Binary registry:
19.10.0.0.0 Release_Update 210122180248: Installed
PDB CDB$ROOT:
Rolled back to 19.9.0.0.0 Release_Update 201020093047 successfully on 05-MAR-21 05.05.53.532000 PM
PDB HCM92UAT:
Rolled back to 19.9.0.0.0 Release_Update 201020093047 with errors on 05-MAR-21 05.05.55.186000 PM
PDB PDB$SEED:
Rolled back to 19.9.0.0.0 Release_Update 201020093047 successfully on 05-MAR-21 05.05.54.420000 PM
Adding patches to installation queue and performing prereq checks…done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED
No interim patches need to be rolled back
Patch 32062765 (Windows Database Bundle Patch : 19.10.0.0.210119 (32062765)):
Apply from 19.9.0.0.0 Release_Update 201020093047 to 19.10.0.0.0 Release_Update 210122180248
The following interim patches will be applied:
32067171 (OJVM RELEASE UPDATE 19.10.0.0.0 (32067171))
For the following PDBs: HCM92UAT
No interim patches need to be rolled back
Patch 31719903 (Windows Database Bundle Patch : 19.9.0.0.201020 (31719903)):
Rollback from 19.9.0.0.0 Release_Update 201020093047 to 19.1.0.0.0 Feature Release
Patch 32062765 (Windows Database Bundle Patch : 19.10.0.0.210119 (32062765)):
Apply from 19.1.0.0.0 Feature Release to 19.10.0.0.0 Release_Update 210122180248
The following interim patches will be applied:
32067171 (OJVM RELEASE UPDATE 19.10.0.0.0 (32067171))
Installing patches…
Patch installation complete. Total patches installed: 7
Validating logfiles…done
Patch 32062765 apply (pdb CDB$ROOT): SUCCESS
logfile: C:\app\Employee\cfgtoollogs\sqlpatch\32062765\23986305/32062765_apply_CDBNPRD_CDBROOT_2021Mar05_19_24_06.log (no errors)
Patch 32067171 apply (pdb CDB$ROOT): SUCCESS
logfile: C:\app\Employee\cfgtoollogs\sqlpatch\32067171\24096715/32067171_apply_CDBNPRD_CDBROOT_2021Mar05_19_22_00.log (no errors)
Patch 32062765 apply (pdb PDB$SEED): SUCCESS
logfile: C:\app\Employee\cfgtoollogs\sqlpatch\32062765\23986305/32062765_apply_CDBNPRD_PDBSEED_2021Mar05_19_26_44.log (no errors)
Patch 32067171 apply (pdb PDB$SEED): SUCCESS
logfile: C:\app\Employee\cfgtoollogs\sqlpatch\32067171\24096715/32067171_apply_CDBNPRD_PDBSEED_2021Mar05_19_26_29.log (no errors)
Patch 31719903 rollback (pdb HCM92UAT): WITH ERRORS
logfile: C:\app\Employee\cfgtoollogs\sqlpatch\31719903\23815960/31719903_rollback_CDBNPRD_HCM92UAT_2021Mar05_19_28_12.log (errors)
-> Error at line 36197: script rdbms/admin/catlsby.sql
– SP2-0310: unable to open file “C:\app\Employee\product\19C\dbhome_1\sqlpatch\31719903\23815960\rollback_files\19.1.0.0.0\rdbms\admin\catlsby.sql”
-> Error at line 42974: script rdbms/admin/prvtemxi.plb
– Warning: Package Body created with compilation errors.
-> Error at line 42981: script rdbms/admin/prvtemxi.plb
– 447/12 PLS-00323: subprogram or cursor ‘OMX_LOCALES’ is declared in a
-> Error at line 110400: script rdbms/admin/backport_files/bug_30528547_rollback.sql
– ORA-04068: existing state of packages has been discarded
– ORA-04061: existing state of package body “SYS.DBMS_REGISTRY” has been
– invalidated
– ORA-04065: not executed, altered or dropped package body “SYS.DBMS_REGISTRY”
– ORA-06512: at line 10
-> Error at line 117438: script ctx/admin/drixmd.pkh
– SP2-0310: unable to open file “C:\app\Employee\product\19C\dbhome_1\sqlpatch\31719903\23815960\rollback_files\19.1.0.0.0\ctx\admin\drixmd.pkh”
-> Error at line 117467: script ctx/admin/driparse.plb
– SP2-0310: unable to open file “C:\app\Employee\product\19C\dbhome_1\sqlpatch\31719903\23815960\rollback_files\19.1.0.0.0\ctx\admin\driparse.plb”
-> Error at line 117496: script ctx/admin/drirepm.plb
– SP2-0310: unable to open file “C:\app\Employee\product\19C\dbhome_1\sqlpatch\31719903\23815960\rollback_files\19.1.0.0.0\ctx\admin\drirepm.plb”
-> Error at line 117525: script ctx/admin/drixmd.plb
– SP2-0310: unable to open file “C:\app\Employee\product\19C\dbhome_1\sqlpatch\31719903\23815960\rollback_files\19.1.0.0.0\ctx\admin\drixmd.plb”
-> Error at line 117554: script ctx/admin/drvddl.plb
– SP2-0310: unable to open file “C:\app\Employee\product\19C\dbhome_1\sqlpatch\31719903\23815960\rollback_files\19.1.0.0.0\ctx\admin\drvddl.plb”
-> Error at line 139152: script rdbms/admin/prvtbpw.plb
– Warning: Package Body created with compilation errors.
-> Error at line 139159: script rdbms/admin/prvtbpw.plb
– 27567/13 PL/SQL: Statement ignored
-> Error at line 139160: script rdbms/admin/prvtbpw.plb
– 27567/36 PLS-00302: component ‘SET_KGL_TIME_TO_WAIT_FOR_LOCKS’ must be
-> Error at line 139163: script rdbms/admin/prvtbpw.plb
– 27593/13 PL/SQL: Statement ignored
-> Error at line 139164: script rdbms/admin/prvtbpw.plb
– 27593/36 PLS-00302: component ‘SET_KGL_TIME_TO_WAIT_FOR_LOCKS’ must be
Patch 32062765 apply (pdb HCM92UAT): WITH ERRORS (PRIOR RU)
logfile: C:\app\Employee\cfgtoollogs\sqlpatch\32062765\23986305/32062765_apply_CDBNPRD_HCM92UAT_2021Mar05_19_30_36.log (not checked)
Patch 32067171 apply (pdb HCM92UAT): SUCCESS
logfile: C:\app\Employee\cfgtoollogs\sqlpatch\32067171\24096715/32067171_apply_CDBNPRD_HCM92UAT_2021Mar05_19_27_51.log (no errors)
Please refer to MOS Note 1609718.1 and/or the invocation log
C:\app\Employee\cfgtoollogs\sqlpatch\sqlpatch_7924_2021_03_05_19_20_57\sqlpatch_invocation.log
for information on how to resolve the above errors.
SQL Patching tool complete on Fri Mar 5 19:34:12 2021
##############################################################
Please advise !!
I have also created an SR# as below
SR 3-25368002451 : Error during datapatch -verbose after applying Jan 2021 CPU – Windows DB Bundle Patch 32062765
Thanks,
Dabir
Thanks Dabir/Srini,
I see that both SRs have been resolved with this solution:
Execute below script to recreate package KUPW$WORKER
SQL> conn / as sysdba
SQL> @?/rdbms/admin/prvthpui.plb
SQL> @?/rdbms/admin/prvtbpui.plb
SQL> alter package KUPW$WORKER compile body;
ref:
KUPW$WORKER or KUPU$UTILITIES_INT Invalid after Upgrade (Doc ID 2691905.1)
But one thing wasn’t needed (despite mentioned in the SR):
Containers don’t need to be started in UPGRADE mode when you run datapatch. This is not needed.
Cheers,
Mike
Hi Mike,
Thanks for your response. Yes I was able to resolve with the help from mentioned Oracle docs. But I am still not sure why it failed at the 1st place.
And you are right upgrade mode not required for Bundle patch however seems needed only for OJVM patches.
Lesson learnt : Check for invalid objects before patching the database.
Regards,
Dabir
Hi Dabir,
actually performance-wise it is always good to run an utlrp.sql before invoking datapatch.
But I confess, I’m not doing it on my quarterly patch runs.
Especially in a CDB/PDB environment it can speed up the completion of datapatch drastically.
Cheers, and happy that it got solved so quickly!
Mike
Hi Mike,
Very well noted and thanks to Oracle for maintaining such a good document repository 🙂
Good day !!
Regards,
Dabir
Hi Mike, I know you are always very busy, just would like to get your valuable opinion: it seems that for AIX there is a problem with opatchauto at opatch release 12.2.0.1.24. I already opened a SR (3-25291094041). The colleagues are not able to provide the version 23 of opatch so that I can test it but suggested to perform the update for GI and RDBMS ‘manually’. For me this is not an acceptable workaround. What do you think?
Thanks & regards
Axel D.
Hi Alex,
it could be that this version simply does not exist for external use. I have seen in the past that opatch does 2-version-increments. But please chase Support for a clarification.
Cheers,
Mike
Hi Mike, now your colleagues told me that we and other customers unfortunately hit Bug 32547619 “12.2.0.1.24 OPATCH FOR AIX IS FAILING WITH JAVA.LANG.NULLPOINTEREXCEPTION ERROR”.
Regards
Axel
Thanks Axel – and sorry to hear this 🙁
I see that this MOS note recommends to use a previous version of opatch:
Oracle Support Document 2756755.1 (AIX OPatch 12.2.0.1.24 Version Fails with “Opatchcore binary error message=Assertion” and “VMState: 0x0005ff0a”) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=2756755.1
Cheers,
Mike
Hi Axel,
we had similar issue with Opatch version 12.2.0.1.24
Please find this note , we have to use 12.2.0.1.23 Opatch version to fix the java issue
AIX OPatch 12.2.0.1.24 Version Fails with Java Error “VMState: 0x0005ff0a” ( Doc ID 2756755.1 )
Its password protected ,so please contact Support to get password and download it
Thanks,
Ramanan
Mike, in your webinar you recommend cloning your Oracle Database Home and applying the RU patch to that home, then opening your instance in the new home and running datapatch. Do you have a blog post or white paper which runs through the most efficient way to clone a home on the same server and patch using this method? I ask because the docs say that clone.pl will be deprecated after Oracle 19c and that a Software Only install is the preferred method of cloning an Oracle Home.
Hi Thomas,
please see here:
https://docs.oracle.com/en/database/oracle/oracle-database/19/ssdbi/cloning-an-oracle-home.html#GUID-494E59C3-C381-4A35-8ABE-F6E5DBF29032
Cheers,
Mike
can we do the automation of opatch apply for each quaterly patch?
You can – FPP (Fleet Patching and Provisioning) is the Oracle product for it. If you have a RAC license, it is included already.
Otherwise you can script it with ansible or any other scripting automation.
Enterprise Manager has a Fleet Management Pack, too.
Cheers,
Mike