It’s brutally hot out there. It must be July. And I sit in my cool basement office and start my quarterly patching journey hoping that you can enjoy the sun and the sea somewhere. It is time to do my usual exercise, this time Patching most of my environments with the July 2022 Patch Bundles.
Photo by National Cancer Institute on Unsplash
As usual, an important annotation upfront: I patch in-place due to space issues. But in reality, you please patch always out-of-place with a separate home. Please see this blog post about how to apply the RU directly when you provision a new home with OUI.
Security Alert July 2022
You will find the July 2022 Security Alerts for all products here. And as usual, please pay close attention to the Database Server Products Risk Matrix. You will find the usual suspects such as OJVM but more important, a 9.1 database issue. Hence, you won’t need advice. You must apply this RU to your environments since 10.0 would be the high score. Also keep in mind that there may be already exploits circulating at least in the darker corners of the web.
Oracle Database risk matrix – July 2022 – see: https://www.oracle.com/security-alerts/cpujul2022.html#AppendixDB
Something very important I’d like to mention since I’ve had numerous discussions in the past weeks about Oracle Database 11.2.0.4.
The column “Supported Versions Affected” do not mention Oracle 11g, 12.2 or 18c anymore for a simple reason: None of these releases is under either Premier nor Extended Support. But you can be almost certain assuming that the first issue will affect those releases as well. So you have two choices: Sign up for Market Driven Support (MDS) or upgrade to 19c. No expert anywhere can help you as there is no such thing as “virtual patching”.
Always be aware that the issues noted in the risk matrix may happen in older releases, too, even though they are not mentioned anymore.
Database Patch Bundles
You will find the links to the individual patch bundles in MOS Note: 2867871.1 – Critical Patch Update (CPU) Program Jul 2022 Patch Availability Document (DB-only). But you can use as well MOS Note: 2118136.2 – Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases to download “your” bundle.
- Oracle Database 21c
- Database Release Update 21.7.0.0.220719 Patch 34160444 for UNIX
- README
- List of fixes: MOS Note: 2814015.1
- Database Release Update 21.7.0.0.220719 Patch 34160444 for UNIX
- Oracle Database 19c
- Database Release Update 19.16.0.0.220719 Patch 34133642 for UNIX
- README
- List of fixes: MOS Note: 2523220.1
- OJVM Release Update 19.16.0.0.220719 Patch 34086870 for all platforms
- Database Release Update 19.16.0.0.220719 Patch 34133642 for UNIX
- Oracle Database 12.1.0.2
- Database Proactive Bundle Patch 12.1.0.2.220719 Patch 34204559,
- Patch bundle is not available while I write this blog post
- Database Proactive Bundle Patch 12.1.0.2.220719 Patch 34204559,
- Oracle Database 11.2.0.4
- Not available yet – will update the blog post later
Do I need a new OPatch?
And then it is time again for the opatch check. Without even seeing the number, I can tell you that you MUST refresh your opatch since there are some significant performance improvements in the 30 version on all platforms. But let me double check with the READMEs:
- 21.7.0 requires opatch 12.2.0.1.30 or later
- 19.16.0 requires opatch 12.2.0.1.30 or later
- 12.1.0.2 BP July 2022 requires opatch 12.2.0.1.30 or later
- 11.2.0.4 PSU July 2022 requires opatch 11.2.0.3.33 or later
The most recent opatch version is already 32. And since I know that there may be some improvements included, I will update opatch in all my homes. The 6880880 link from the readmes takes you directly to the correct download. BUT … you will be directed to the 12.2.0.1.30 version of opatch, not the 29 version. And it may not wonder you, all three opatch releases for 12.1 – 21c are identical even though all sail with different display labels on MOS. So it is fairly enough in my case to download it only once.
For the 11g opatch, it will point you to the 11.2.0.3.34 version as well. Which is fine. I just wanted to mention it.
Hence, action no.1 for all my environments before I will start patching:
- Either wipe out the current
OPatchdirectory in each home, then unzip the new OPatch bundles – or use unzip -o option instead
Applying the RU 21.7.0 to my Oracle 21c home
I always do something you shouldn’t do. I apply the patch to my existing home. This has to do with the available space in my lab environment. You please provide a new home and apply the patch(es) right away as I describe here.
Patching my 21c database home to RU 21.7.0 is again pretty straight forward.
- Conflict and space checks
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 12.2.0.1.32 Copyright (c) 2022, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/21 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/21/oraInst.loc OPatch version : 12.2.0.1.32 OUI version : 12.2.0.9.0 Log file location : /u01/app/oracle/product/21/cfgtoollogs/opatch/opatch2022-07-20_17-28-09PM_1.log Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed. OPatch succeeded. $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -ph ./ Oracle Interim Patch Installer version 12.2.0.1.32 Copyright (c) 2022, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/21 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/21/oraInst.loc OPatch version : 12.2.0.1.32 OUI version : 12.2.0.9.0 Log file location : /u01/app/oracle/product/21/cfgtoollogs/opatch/opatch2022-07-20_17-28-53PM_1.log Invoking prereq "checksystemspace" Prereq "checkSystemSpace" passed. OPatch succeeded.
Fine. I’m good to go.
- opatch apply
$ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.32 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/21 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/21/oraInst.loc OPatch version : 12.2.0.1.32 OUI version : 12.2.0.9.0 Log file location : /u01/app/oracle/product/21/cfgtoollogs/opatch/opatch2022-07-20_17-29-52PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 34160444 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/21') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '34160444' to OH '/u01/app/oracle/product/21' ApplySession: Optional component(s) [ oracle.network.gsm, 21.0.0.0.0 ] , [ oracle.python, 21.0.0.0.0 ] , [ oracle.tfa, 21.0.0.0.0 ] , [ oracle.network.cman, 21.0.0.0.0 ] , [ oracle.rdbms.tg4ifmx, 21.0.0.0.0 ] , [ oracle.sdo.companion, 21.0.0.0.0 ] , [ oracle.rdbms.tg4sybs, 21.0.0.0.0 ] , [ oracle.rdbms.tg4msql, 21.0.0.0.0 ] , [ oracle.rdbms.tg4db2, 21.0.0.0.0 ] , [ oracle.duma, 21.0.0.0.0 ] , [ oracle.oraolap.mgmt, 21.0.0.0.0 ] , [ oracle.rdbms.tg4tera, 21.0.0.0.0 ] , [ oracle.ons.eons.bwcompat, 21.0.0.0.0 ] , [ oracle.rdbms.ic, 21.0.0.0.0 ] , [ oracle.rdbms.tg4ifxm, 21.0.0.0.0 ] , [ oracle.net.cman, 21.0.0.0.0 ] , [ oracle.ons.cclient, 21.0.0.0.0 ] , [ oracle.jdk, 1.8.0.271.00 ] not present in the Oracle Home or a higher version is found. Patching component oracle.network.rsf, 21.0.0.0.0... Patching component oracle.rdbms, 21.0.0.0.0... Patching component oracle.rdbms.util, 21.0.0.0.0... Patching component oracle.rdbms.rsf, 21.0.0.0.0... ... Patching component oracle.rdbms.scheduler, 21.0.0.0.0... Patching component oracle.sdo.locator, 21.0.0.0.0... Patching component oracle.xdk.rsf, 21.0.0.0.0... Patching component oracle.precomp.common, 21.0.0.0.0... Patching component oracle.precomp.lang, 21.0.0.0.0... Patching component oracle.jdk, 1.8.0.291.09... Patch 34160444 successfully applied. Sub-set patch [33516412] has become inactive due to the application of a super-set patch [34160444]. Please refer to Doc ID 2161861.1 for any possible further required actions. Log file location: /u01/app/oracle/product/21/cfgtoollogs/opatch/opatch2022-07-20_17-29-52PM_1.log OPatch succeeded.
Looks good – no errors. And again, please note that there is no OJVM bundle patch anymore since Oracle 21c since it is included into the standard RU.
- datapatch
SQL*Plus: Release 21.0.0.0.0 - Production on Wed Jul 20 17:42:28 2022 Version 21.7.0.0.0 Copyright (c) 1982, 2022, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORACLE instance started. Total System Global Area 1845489680 bytes Fixed Size 9687056 bytes Variable Size 520093696 bytes Database Buffers 1308622848 bytes Redo Buffers 7086080 bytes Database mounted. Database opened. SQL> exit Disconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production Version 21.7.0.0.0 [CDB3] oracle@hol:~/34160444 $ $ORACLE_HOME/OPatch/datapatch -verbose SQL Patching tool version 21.7.0.0.0 Production on Wed Jul 20 17:42:59 2022 Copyright (c) 2012, 2022, Oracle. All rights reserved. Log file for this invocation: /u01/app/oracle/homes/OraDB21Home1/cfgtoollogs/sqlpatch/sqlpatch_6860_2022_07_20_17_42_59/sqlpatch_invocation.log Connecting to database...OK Gathering database info...done Note: Datapatch will only apply or rollback SQL fixes for PDBs that are in an open state, no patches will be applied to closed PDBs. Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1) Bootstrapping registry and package to current versions...done Determining current state...done Current state of interim SQL patches: No interim patches found Current state of release update SQL patches: Binary registry: 21.7.0.0.0 Release_Update 220628175910: Installed PDB CDB$ROOT: Applied 21.5.0.0.0 Release_Update 220107185933 successfully on 19-JAN-22 02.27.19.938360 PM PDB PDB$SEED: Applied 21.5.0.0.0 Release_Update 220107185933 successfully on 19-JAN-22 02.27.20.278645 PM Adding patches to installation queue and performing prereq checks...done Installation queue: For the following PDBs: CDB$ROOT No interim patches need to be rolled back Patch 34160444 (Database Release Update : 21.7.0.0.220719 (34160444)): Apply from 21.5.0.0.0 Release_Update 220107185933 to 21.7.0.0.0 Release_Update 220628175910 No interim patches need to be applied For the following PDBs: PDB$SEED No interim patches need to be rolled back Patch 34160444 (Database Release Update : 21.7.0.0.220719 (34160444)): Apply from 21.5.0.0.0 Release_Update 220107185933 to 21.7.0.0.0 Release_Update 220628175910 No interim patches need to be applied Installing patches... Patch installation complete. Total patches installed: 2 Validating logfiles...done Patch 34160444 apply (pdb CDB$ROOT): SUCCESS logfile: /u01/app/oracle/homes/OraDB21Home1/cfgtoollogs/sqlpatch/34160444/24850944/34160444_apply_CDB3_CDBROOT_2022Jul20_17_43_13.log (no errors) Patch 34160444 apply (pdb PDB$SEED): SUCCESS logfile: /u01/app/oracle/homes/OraDB21Home1/cfgtoollogs/sqlpatch/34160444/24850944/34160444_apply_CDB3_PDBSEED_2022Jul20_17_45_14.log (no errors) SQL Patching tool complete on Wed Jul 20 17:46:25 2022Looks flawless as well.
- Enabling new optimizer fixes
SQL*Plus: Release 21.0.0.0.0 - Production on Wed Jul 20 17:47:15 2022 Version 21.7.0.0.0 Copyright (c) 1982, 2022, Oracle. All rights reserved. Connected to: Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production Version 21.7.0.0.0 SQL> set serveroutput on; SQL> execute dbms_optim_bundle.getBugsforBundle; 21.7.0.0.220719DBRU: Bug: 30771009, fix_controls: 30771009 Bug: 29413205, fix_controls: 29413205 Bug: 28044739, fix_controls: 28044739 Bug: 33089096, fix_controls: 31545400 PL/SQL procedure successfully completed. SQL> execute dbms_optim_bundle.enable_optim_fixes('ON','BOTH', 'YES') DBMS_OPTIM command: dbms_optim_bundle.enable_optim_fixes('ON', 'BOTH', 'YES') Bundle's _fix_control setting as per action:ON 31988833:1 32800137:0 32408640:1 29738374:1 33325981:1 32913527:0 32766397:0 31912834:1 33145153:1 31843716:0 32212062:0 31880080:0 32856375:1 33297275:1 33323903:1 32302470:1 32851615:1 32754044:1 30771009:1 29413205:1 28044739:1 31545400:1 Taking current instance CDB3 as base, details on _fix_control setting for CON_ID 1 : 1) Current _fix_control setting for spfile: 31988833:1 32800137:0 32408640:1 29738374:1 33325981:1 32913527:0 32766397:0 31912834:1 33145153:1 31843716:0 32212062:0 31880080:0 2) Final _fix_control setting for spfile considering current_setting_precedence is YES 31988833:1 32800137:0 32408640:1 29738374:1 33325981:1 32913527:0 32766397:0 31912834:1 33145153:1 31843716:0 32212062:0 31880080:0 32856375:1 33297275:1 33323903:1 32302470:1 32851615:1 32754044:1 30771009:1 29413205:1 28044739:1 31545400:1 3) Current _fix_control setting in memory: 31988833:1 32800137:0 32408640:1 29738374:1 33325981:1 32913527:0 32766397:0 31912834:1 33145153:1 31843716:0 32212062:0 31880080:0 32856375:0 33297275:0 33323903:0 32302470:0 32851615:0 32754044:0 30771009:0 29413205:0 28044739:0 31545400:0 4) Final _fix_control setting for memory considering current_setting_precedence is YES 31988833:1 32800137:0 32408640:1 29738374:1 33325981:1 32913527:0 32766397:0 31912834:1 33145153:1 31843716:0 32212062:0 31880080:0 32856375:0 33297275:0 33323903:0 32302470:0 32851615:0 32754044:0 30771009:0 29413205:0 28044739:0 31545400:0 WARNING: final _fix_control setting for memory is not same as final _fix_control setting for spfile. Please look at point 2 and 4 above to see the differences. PL/SQL procedure successfully completed.Let me repeat what I wrote before already several times: New optimizer fixes are there for a reason. But it is your choice whether you’d like to enable them after patching, or keep them disabled. Just make sure when you upgrade or create a new database to always enable them.
Applying RU and OJVM 19.16.0 July 2022 to my 19c home
Next exercise is my 19c home, the one I am using the most often. And here I have also OJVM installed to reproduce several cases crossing my radar from time to time.
- Prechecks
Without massaging my oui-patch.xml as in the previous quarter with this workaround, this part is very slow of course. Since the opatch documentation usually is not very informative about improvements, I can’t tell you whether the fixes they’d like to do are in, or not. I’d guess, at least for this part of the patching process, there aren’t any improvements I could see/feel in my environment.To be very frank, this is the most annoying part of the entire patching experience to me. I know why it takes so long. And since the prechecks will get repeated for no obvious reason again later, it is frustrating at best. I know that the opatch team has committed to deliver a solution. And I keep the faith that this will happen soon. Still, I don’t know any ETA date. Let’s keep our fingers crossed. - opatch apply
$ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.32 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.32 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-07-20_18-16-01PM_1.log Verifying environment and performing prerequisite checks... -------------------------------------------------------------------------------- Start OOP by Prereq process. Launch OOP... Oracle Interim Patch Installer version 12.2.0.1.32 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.32 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-07-20_18-22-41PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 34133642 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/19') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '34133642' to OH '/u01/app/oracle/product/19' ApplySession: Optional component(s) [ oracle.network.gsm, 19.0.0.0.0 ] , [ oracle.rdbms.ic, 19.0.0.0.0 ] , [ oracle.rdbms.tg4db2, 19.0.0.0.0 ] , [ oracle.tfa, 19.0.0.0.0 ] , [ oracle.rdbms.tg4ifmx, 19.0.0.0.0 ] , [ oracle.options.olap, 19.0.0.0.0 ] , [ oracle.ons.cclient, 19.0.0.0.0 ] , [ oracle.rdbms.tg4tera, 19.0.0.0.0 ] , [ oracle.sdo.companion, 19.0.0.0.0 ] , [ oracle.xdk.companion, 19.0.0.0.0 ] , [ oracle.options.olap.api, 19.0.0.0.0 ] , [ oracle.ons.eons.bwcompat, 19.0.0.0.0 ] , [ oracle.rdbms.tg4msql, 19.0.0.0.0 ] , [ oracle.oid.client, 19.0.0.0.0 ] , [ oracle.rdbms.tg4sybs, 19.0.0.0.0 ] , [ oracle.net.cman, 19.0.0.0.0 ] , [ oracle.network.cman, 19.0.0.0.0 ] , [ oracle.jdk, 1.8.0.191.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.bali.jewt, 11.1.1.6.0... Patching component oracle.bali.ewt, 11.1.1.6.0... Patching component oracle.help.ohj, 11.1.1.7.0... Patching component oracle.perlint, 5.28.1.0.0... Patching component oracle.rdbms.locator, 19.0.0.0.0... ... Patching component oracle.sdo.locator, 19.0.0.0.0... Patching component oracle.network.listener, 19.0.0.0.0... Patching component oracle.rdbms.rsf.ic, 19.0.0.0.0... Patching component oracle.precomp.common, 19.0.0.0.0... Patching component oracle.precomp.lang, 19.0.0.0.0... Patching component oracle.jdk, 1.8.0.201.0... Patch 34133642 successfully applied. Sub-set patch [33515361] has become inactive due to the application of a super-set patch [34133642]. Please refer to Doc ID 2161861.1 for any possible further required actions. Log file location: /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-07-20_18-22-41PM_1.log OPatch succeeded.
Fine.
- datapatch
Just to clarify this – you should run datapatch only once at the very end of your patching exercise. I ran it two times now because I forgot to apply OJVM. I didn’t want to fake the output, hence I applied OJVM and then ran datapatch a second time. Normally you should apply the RU, OJVM is needed, one-offs – all ideally in a new separate home already installed – and then you execute datapatch at the very end only once!$ORACLE_HOME/OPatch/datapatch -verbose SQL Patching tool version 19.16.0.0.0 Production on Wed Jul 20 20:27:15 2022 Copyright (c) 2012, 2022, Oracle. All rights reserved. Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_24454_2022_07_20_20_27_15/sqlpatch_invocation.log Connecting to database...OK Gathering database info...done Note: Datapatch will only apply or rollback SQL fixes for PDBs that are in an open state, no patches will be applied to closed PDBs. Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1) Bootstrapping registry and package to current versions...done Determining current state...done Current state of interim SQL patches: Interim patch 33192694 (OJVM RELEASE UPDATE: 19.13.0.0.211019 (33192694)): Binary registry: Not installed PDB CDB$ROOT: Rolled back successfully on 19-JAN-22 10.14.44.327180 PM PDB PDB$SEED: Rolled back successfully on 19-JAN-22 10.14.44.347566 PM Interim patch 33561310 (OJVM RELEASE UPDATE: 19.14.0.0.220118 (33561310)): Binary registry: Installed PDB CDB$ROOT: Applied successfully on 19-JAN-22 10.14.44.331372 PM PDB PDB$SEED: Applied successfully on 19-JAN-22 10.14.44.350816 PM Current state of release update SQL patches: Binary registry: 19.16.0.0.0 Release_Update 220703022223: Installed PDB CDB$ROOT: Applied 19.14.0.0.0 Release_Update 211225122123 successfully on 19-JAN-22 09.48.19.540225 PM PDB PDB$SEED: Applied 19.14.0.0.0 Release_Update 211225122123 successfully on 19-JAN-22 09.48.19.756424 PM Adding patches to installation queue and performing prereq checks...done Installation queue: For the following PDBs: CDB$ROOT PDB$SEED No interim patches need to be rolled back Patch 34133642 (Database Release Update : 19.16.0.0.220719 (34133642)): Apply from 19.14.0.0.0 Release_Update 211225122123 to 19.16.0.0.0 Release_Update 220703022223 No interim patches need to be applied Installing patches... Patch installation complete. Total patches installed: 2 Validating logfiles...done Patch 34133642 apply (pdb CDB$ROOT): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/34133642/24865470/34133642_apply_CDB2_CDBROOT_2022Jul20_20_28_07.log (no errors) Patch 34133642 apply (pdb PDB$SEED): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/34133642/24865470/34133642_apply_CDB2_PDBSEED_2022Jul20_20_29_30.log (no errors) SQL Patching tool complete on Wed Jul 20 20:30:42 2022Good. Time for the next step.
- Doing OJVM now for a change – not a good idea since I need to run datapatch now twice.
$ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.32 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.32 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-07-20_20-54-21PM_1.log Verifying environment and performing prerequisite checks... -------------------------------------------------------------------------------- Start OOP by Prereq process. Launch OOP... Oracle Interim Patch Installer version 12.2.0.1.32 Copyright (c) 2022, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.32 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-07-20_21-00-20PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 34086870 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/19') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '34086870' to OH '/u01/app/oracle/product/19' Patching component oracle.javavm.server, 19.0.0.0.0... Patching component oracle.javavm.server.core, 19.0.0.0.0... Patching component oracle.rdbms.dbscripts, 19.0.0.0.0... Patching component oracle.rdbms, 19.0.0.0.0... Patching component oracle.javavm.client, 19.0.0.0.0... Patch 34086870 successfully applied. Sub-set patch [33561310] has become inactive due to the application of a super-set patch [34086870]. Please refer to Doc ID 2161861.1 for any possible further required actions. Log file location: /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2022-07-20_21-00-20PM_1.log OPatch succeeded.
- datapatch again, now for OJVM
$ORACLE_HOME/OPatch/datapatch -verbose SQL Patching tool version 19.16.0.0.0 Production on Wed Jul 20 21:15:16 2022 Copyright (c) 2012, 2022, Oracle. All rights reserved. Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_30826_2022_07_20_21_15_16/sqlpatch_invocation.log Connecting to database...OK Gathering database info...done Note: Datapatch will only apply or rollback SQL fixes for PDBs that are in an open state, no patches will be applied to closed PDBs. Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1) Bootstrapping registry and package to current versions...done Determining current state...done Current state of interim SQL patches: Interim patch 33192694 (OJVM RELEASE UPDATE: 19.13.0.0.211019 (33192694)): Binary registry: Not installed PDB CDB$ROOT: Rolled back successfully on 19-JAN-22 10.14.44.327180 PM PDB PDB$SEED: Rolled back successfully on 19-JAN-22 10.14.44.347566 PM Interim patch 33561310 (OJVM RELEASE UPDATE: 19.14.0.0.220118 (33561310)): Binary registry: Not installed PDB CDB$ROOT: Applied successfully on 19-JAN-22 10.14.44.331372 PM PDB PDB$SEED: Applied successfully on 19-JAN-22 10.14.44.350816 PM Interim patch 34086870 (OJVM RELEASE UPDATE: 19.16.0.0.220719 (34086870)): Binary registry: Installed PDB CDB$ROOT: Not installed PDB PDB$SEED: Not installed Current state of release update SQL patches: Binary registry: 19.16.0.0.0 Release_Update 220703022223: Installed PDB CDB$ROOT: Applied 19.16.0.0.0 Release_Update 220703022223 successfully on 20-JUL-22 08.30.37.698095 PM PDB PDB$SEED: Applied 19.16.0.0.0 Release_Update 220703022223 successfully on 20-JUL-22 08.30.38.297082 PM Adding patches to installation queue and performing prereq checks...done Installation queue: For the following PDBs: CDB$ROOT PDB$SEED The following interim patches will be rolled back: 33561310 (OJVM RELEASE UPDATE: 19.14.0.0.220118 (33561310)) No release update patches need to be installed The following interim patches will be applied: 34086870 (OJVM RELEASE UPDATE: 19.16.0.0.220719 (34086870)) Installing patches... Patch installation complete. Total patches installed: 4 Validating logfiles...done Patch 33561310 rollback (pdb CDB$ROOT): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/33561310/24538862/33561310_rollback_CDB2_CDBROOT_2022Jul20_21_15_51.log (no errors) Patch 34086870 apply (pdb CDB$ROOT): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/34086870/24803071/34086870_apply_CDB2_CDBROOT_2022Jul20_21_17_01.log (no errors) Patch 33561310 rollback (pdb PDB$SEED): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/33561310/24538862/33561310_rollback_CDB2_PDBSEED_2022Jul20_21_17_02.log (no errors) Patch 34086870 apply (pdb PDB$SEED): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/34086870/24803071/34086870_apply_CDB2_PDBSEED_2022Jul20_21_17_02.log (no errors) SQL Patching tool complete on Wed Jul 20 21:17:04 2022 - Optimizer fixes
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jul 20 20:48:50 2022 Version 19.16.0.0.0 Copyright (c) 1982, 2022, Oracle. All rights reserved. Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.16.0.0.0 SQL> set serverout on SQL> execute dbms_optim_bundle.getBugsforBundle; 19.16.0.0.220719DBRU: Bug: 28044739, fix_controls: 28044739 Bug: 30771009, fix_controls: 30771009 Bug: 33636280, fix_controls: 33636280 Bug: 33089096, fix_controls: 31545400 Bug: 30618406, fix_controls: 30618406 Bug: 32614157, fix_controls: 32614157 Bug: 33329027, fix_controls: 33329027 Bug: 33311488, fix_controls: 33311488 Bug: 32396085, fix_controls: 32396085 Bug: 32122197, fix_controls: 29972495 Bug: 32363981, fix_controls: 32363981 PL/SQL procedure successfully completed. SQL> execute dbms_optim_bundle.enable_optim_fixes('ON','BOTH', 'YES')
Applying the BP July 2022 to my Oracle 12.1.0.2 home
Since there is a delay with this BP, I will add it after my trip to YATRA.
Applying the PSU July 2022 to my Oracle 11.20.4 home
Since there is a delay with this PSU, I will add it after my trip to YATRA.
Your patch is not available right now?
In case you miss the patch bundle for your release and platform, please read this blog post.
Finally … clean up
Oh yes, my patching experience doesn’t get easier every time unfortunately. Since opatch does still not allow to define a “n-1” rule or pattern meaning “remove everything from my box except the previous patch bundle” I will need to do the cleanup by myself.
See here on how to do the cleanup of older bundles to free up space:
Further Information and Links
- July 2022 Security Alerts
- Database Server Products Risk Matrix
- MOS Note: 2867871.1 – Critical Patch Update (CPU) Program Jul 2022 Patch Availability Document (DB-only)
- Binary patching is slow – see this workaround
- Cleaup of Patch Bundles
- Your patch is not available yet?
–Mike
Hi Mike! Thank you for the post!
About OJVM patch and datapatch – why we can’t install 19.16 patch, OJVM patch after it and then run datapatch only one time?
You can – it was my failure 🙂
Cheers
Mike
Hi Mike,
is there any ETA for a merge of the following bundle patch?
Data Pump Recommended Proactive Patches For 19.10 and Above (Doc ID 2819284.1)
Also, when will they be consolidated into a RU?
Thanks in advance.
Regards,
Xabi.
Hi,
we need to upgrade from 12c to 19c. Which is better: 12.2.0.1 to 19.3 and upgrade to 19.16 after migration or upgrade 19.3 install to 19.16 and migrate then?
Thank you very much
Hi Ulrich,
always go directly to the designated RU. Never upgrade to the base release, and then patch afterwards.
Cheers
Mike
You mentioned 19.6 in the heading
I typically patch my 19c homes out of place. We usually have 10+ instances in each home and it makes for less down time to do things out of place.
I notice that in another article you suggest installing 19.3 base and then applying the RU on top of that. Will that potentially cause any issues with moving a database from let’s say a 19.14 home to a 19.16 home in case something needs to be rolled back for the upgrade (eg, typically Java)? I could technically rollback all in the current home before moving to the new home, but this can add a lot of time to the patching.
My current procedure is as follows (lets say 19.14 home and applying 19.16 RU):
-) copy over the existing 19.14 home to a new directory
-) runInstaller on this dir so it gets registered in oraInventory
-) apply latest OPatch and 19.16 DB & OJVM patch
-) copy over latest autoupgrade.jar
-) create a gold image into a zip file so I can then deploy this same home to other hosts (try to keep all identical in term of binaries)
-) Then I can one-by-one bring up databases in the new home and datapatch
This has been working well for me so far with all of my 19c patching… one negative side effect is that the .patch_storage directory is growing significantly with each RU I apply. I am sure the n-1 issue you bring up above is also a factor, but I don’t worry about that as much since after one install I use a gold image for all subsequent installs.
Do you think what I am doing is overkill (ie, applying newest RU on top of the previous RU).
Thanks.
Hi Ed,
no – datapatch will sort out everything correctly. From the binary side, there is no issue anyway. And datapatch does the rest for you.
Cheers
Mike
That would Oracle 19.16 and not 19.6 …
Mike,
You installed the 19.16 RU, ran Datapatch, then installed OJVM, and re-ran Datapatch.
Could you have installed both, and then run Datapatch only once?
Hi Ben,
I commented later that this was my fault. Usually I do it in one pass.
Cheers
Mike
In title “Applying RU and OJVM 19.6.0 July 2022 to my 19c home” missing the number 1 for 19.16
Thanks for the post. you are doing a great job .
Applying RU and OJVM 19.6.0 July 2022 to my 19c home
— i think above title has minor change about 19.6.
Hi Mike
After applying the July RU on our 19c environments we got “ORA-16705: internal error in Data guard broker” errors on all our DataGuard environments. Did you or your customers also face the issue? We’re patching our envs 4 times a year (meaning all RUs) but never had that issue…
best
Suli
Hi Suli,
I blogged about it soon after.
Cheers
Mike
Mike – sharing the following information only because you specifically called out the 9.1 – CVE-2020-35169.
Based on TCPS being listed in the protocol column, I suspected that this CVE was applicable only if your database listener was configured for TCPS connections.
I opened an SR with Support and they confirmed the same.
I’m not advocating ‘not to patch’. Just mentioning that if you are running your listener using sqlnet (Oracle Net) protocol, or any other that is not TCPS, this specific CVE is not applicable to you.
Hi Mike
Did you try this new Patch also in a Data Guard Environment?
We got some troubles after upgrading from 19.15. to 19.16. that the Data Guard Broker says:
ORA-16705: internal error in Data guard broker
Recreate the configuration or disable/enable configuration didn’t help.
Hi Manuel,
I didn’t. But I hope you’d sorted it out by now.
Sorry for the late reply.
Cheers
Mike
I’m confused about that first one (the 9.1). When I look at that CVE, it says it is for “Dell BSAFE Crypto-C Micro Edition, versions before 4.1.5, and Dell BSAFE Micro Edition Suite, versions before 4.5.2, contain an Improper Input Validation Vulnerability.”
https://nvd.nist.gov/vuln/detail/CVE-2020-35169
So does that mean it only applies to the client? Or DB server when it is running on a Dell server?
I looked for more info on it, and came up with this discussion (but that doesn’t look conclusive)
https://community.oracle.com/mosc/discussion/4523879/components-affected-by-cve-2020-35169
Hi Gord,
sorry for my late reply – still working on the backlog.
Unfortunately, I can’t comment on CVEs since I have no insights.
You always need to please check with Oracle Support.
Cheers,
Mike
Hi Mike,
do you know if CVE-2020-35169 is really Enterprise Edition specific like mentioned in the Risk Matrix?
Cheers!
Hi Matt,
I can’t comment on CVEs. You always need to check with Oracle Support please since I have no insights.
Cheers
Mike
Hi Mike, thanks for your excellent blog post on patching your environment, I always open your site before searching for the quarterly update. I would like to mention that you don’t talk about OCW Release Update 19.16.0.0.220719 in your update of your 19c OH installation, i did some digging and could not find a straight answer, I applied it anyways all went good.
Thanks for your feedback!!
Cheers
Mike
>please patch always out-of-place with a separate home
in case of Oracle Restart, we will only need to change the config property to point out on the new OH afterwards, right?
What about Oracle Restart itself, should it be also patched out-of-place? Together with DB with opatchauto or within another maintenance window separately?
thanks!
Hi Dennis,
you please need to check with Support. I can’t comment regarding RESTART since I don’t know all pieces of the available documentation.
Cheers,
Mike
Hi again Mike, one last thing I would add to the upgrade procedure, check the FRA, in doing my upgrade it froze during datapatch with no error message but a few minutes later, reading trough the alert log I was able to see that the DB was not able to allocate space in the FRA.
Thanks
Jean-François Brodeur
Hi Jean-Francois,
we do check the FRA already.
Then in your case something must have been very different and unexpected.
Cheers
Mike
Mike, how do I patch a full Administrator client only with this patch? There doesn’t seem to be a clear patch for Jul 2022 PSU and the client-only home.
Hi Carol,
I think all those RUs can be applied to a client home as well. Please see the patch README documentation – but I’d rather recommend you the Instant Client instead if that is possible for you. Will make everything much easier.
Cheers
Mike
Please add below to known issue after applying 19.16. Fortunately there is a patch, but we have to manually apply it.
ORA-16705: Internal Error In Data Guard Broker After Applying Release Update 19.16 Patch (Doc ID 2887535.1)
Hi Mike,
thank you. One question: if we upgrade out-of-pace, into separate home, I assume we install this new home with all needed one-offs patches which have been built for new RU and exist in current home.
Am i right that if some one-offs are not needed (or deprecated, like merge requested patches for old home) or not released jet for the new RU, then we have to rollback them first from current home?
if yes, can we rollback them only by ./datapatch (sql part) w/o downtime (opatch) ?
thank you in advance
Hi Denis,
sorry for my late reply.
No, you don’t have to rollback any patches in this case. Datapatch will do all the necessary steps and roll out the SQL portion if needed, and do everything the right way.
Cheers
Mike
Mike, Could please write a blog pots of how to patch a read only oracle home in RAC environment. I followed few steps provided by oracle support engineer and I have encountered an issue. There are not a lot of blog posts that describes this, so it would really good to have some clarity as this might save a lot of time during patching.
Regards, Sandeep
Hi Sandeep,
please can you share which advice Oracle Support has given to you?
And let me put this on my loooong list of blog posts to write.
Thanks for the hint!
Cheers
Mike
Hi Mike,
Here is what the engineer proposed (SR 3-27624173921 : Question regarding Read only Oracle homes).
I fixed all of the issues that I encountered and I was able to complete patching of one of our biggest clients. Using ROOH and -switchgrid home to patch GI, we were able to complete patching of a 2 node PROD RAC in just 45mins(with more that 20 database instances). The same process earlier was done over 2 days and took >60mins per Node.
If you can provide me your email address, I can send you the document that outlines the steps that I followed and see if the method I used is accurate or is there another way to perform this patching.
Steps to use ROOH & Gold Image to reduce Patching
======================================================
Since ORACLE_HOME retains/contains the Static Files(i,e mostly Oracle Binaries),We can swap it with a patched version as below-
1. Suppose ORACLE_HOME in the Path – /u01/app/oracle/product/19c/dbhome_1 is of version 19.3 is a ROOH
2. To reduce downtime of Patching in the future,We’d create a Gold Image Zip from /u01/app/oracle/product/19c/dbhome_1,as below-
$OH/runInstaller -createGoldImage -destinationLocation
3. Unzip the newly created Gold Image Zip in a different location For eg – /u01/app/oracle/product/19c/goldimagedbhome
4. Go through the Install Flow and install the OH Binaries under /u01/app/oracle/product/19c/goldimagedbhome
5. Apply the newer RUs/One-Offs to the newly installed OH.
6. Create a Gold Image Zip File from /u01/app/oracle/product/19c/goldimagedbhome using the earlier command
7. The newly created Zip File can simply be unzipped in /u01/app/oracle/product/19c/dbhome_1 (older binaries can be moved out somewhere else as Backup or After DB Services have been shutdown,the Folder dbhome_1 can be renamed)
8. Once done,We can run Post Patch Steps to complete patching Activity.
Regards,
Sandeep
Hi Mike. Got a hopefully simple question for you. I have a single instance 19c install – No grid, no ASM, no Oracle restart, no pdb/cdb, just a single instance and a listener. However, OEM (I have just begun playing with it) has reported that the system is missing the GI patch. Is it necessary to apply the GI patch?
Specifically, I have my DB patched with the 19.16 RU as well as the 19.16 OJVM. OEM shows that the DB is missing 34130714: GI RELEASE UPDATE 19.16.0.0.0. Not sure if this is an issue with how OEM looks at the system or if there might be some component of GI installed that actually needs patched.
Hi Bryan,
no – this is an error in EM or in the way how the patch gets labeled in MyOracle Support. As far as I am aware, OEM just relies on information displayed in MOS where it sources its information from. If this is labeled incorrectly, OEM will display a wrong patch.
Cheers
Mike
Hi Mike, sorry to bother with this, but regarding 19.16, a quick question about 19c client home patching which always confused me a bit. Does it make any sense to patch an 19.3 client home to 19.16 RU with anything else than the following?
34133642;Database Release Update : 19.16.0.0.220719 (34133642)
I mean, do we also apply OCW 19.16, a certain version of DSTvXX patch that we also use at the DB side, etc?
The usual README file that comes with the patch does say “Database client home -> RU/RUR -> opatch apply), and in this case this certainly refers to
Patch number: 34133642
Description: Database Release Update 19.16.0.0.220719
Applicable homes: Only Oracle home for non-Oracle RAC setup. Both Oracle home and Grid home for Oracle RAC setup.
However, these notes are sometime confusing, as it talks about ‘Oracle home’. Yes… but which one? Only RDBMS home or does it also apply to DB Client home?
Example from README:
Patch number: 34160635
Description: OCW Release Update 19.16.0.0.220719
Applicable homes: Both Oracle home and Grid home.
-> does this mean DB client home as well?
Thanks a lot in advance, kind regards.
Hi Zarko,
as far as I understand the concept of CLIENT patching, you will apply the regular RU to your client home – but nothing else.
So I agree with your understanding – I understand it exactly the same way as you do.
And I agree as well, the README should be WAY more clear on this topic.
Cheers
Mike
Certain security patches are applicable to client home where oracle client is installed. I routinely apply db patch, ojvm patch, jdk patch to client homes on windows and linux servers that are used for processing. It will be nice if oracle will update the base client image with latest patch every six month or so. Sometimes there is vulnerability that impacts client install and other times it does not. 19.3 client is too old.
I fully agree with you – and I think we do something for the instant client already but not for the base release full client unfortunately (yet).
Cheers
Mike