Yesterday I blogged about the Oct 2019 Patch Bundles being available for download. And while I wrote this blog post, I downloaded all of them for my environments. Here I’d like to share with you the most simple path to apply them. As I have no cluster or ASM, I don’t have to patch Grid Infrastructure. And I don’t even have OJVM in any of my databases right now. Due to space constraints in my lab environment I will apply the patches in-place. You shouldn’t do this but instead use always a new home you patch. This allows you a quick fallback and is much faster as you prepare the new patched home while your databases are still using the previous home.
Patching all my environments with the Oct 2019 Patch Bundles – Photo by Piron Guillaume on Unsplash
Let’s start.
Oracle Database 19.5.0
I downloaded:
- Database Update 19.5.0.0.191015 Patch 30125133 for UNIX
- Readme
- Opatch 12.2.0.1.17 minimum required
Quick check for the installed opatch version:
$ $ORACLE_HOME/OPatch/opatch version OPatch Version: 12.2.0.1.17 OPatch succeeded.
Fine, I’m good to go.
I unzip the patch and run the patch precheck just mentioned in the readme.
$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2019-10-16_19-21-55PM_1.log Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed. OPatch succeeded.
This looks promising. I will apply 19.5.0 to my environment after stopping the database services. I cut out a portion of the log information.
$ $ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2019-10-16_19-26-21PM_1.log Verifying environment and performing prerequisite checks... -------------------------------------------------------------------------------- Start OOP by Prereq process. Launch OOP... Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2019-10-16_19-26-36PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 30125133 Do you want to proceed? [y|n] Y (auto-answered by -silent) User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/19') Is the local system ready for patching? [y|n] Y (auto-answered by -silent) User Responded with: Y Backing up files... Applying interim patch '30125133' to OH '/u01/app/oracle/product/19' ApplySession: Optional component(s) [ oracle.network.gsm, 19.0.0.0.0 ] , [ oracle.rdbms.ic, 19.0.0.0.0 ] , [ oracle.network.cman, 19.0.0.0.0 ] , [ oracle.net.cman, 19.0.0.0.0 ] , [ oracle.options.olap.awm, 19.0.0.0.0 ] , [ oracle.oraolap.mgmt, 19.0.0.0.0 ] , [ oracle.assistants.usm, 19.0.0.0.0 ] , [ oracle.assistants.asm, 19.0.0.0.0 ] , [ oracle.tfa, 19.0.0.0.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.rdbms, 19.0.0.0.0... ...cutout... Patching component oracle.precomp.common, 19.0.0.0.0... Patching component oracle.precomp.lang, 19.0.0.0.0... Patch 30125133 successfully applied. Sub-set patch [29517242] has become inactive due to the application of a super-set patch [30125133]. Please refer to Doc ID 2161861.1 for any possible further required actions. Log file location: /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2019-10-16_19-26-36PM_1.log OPatch succeeded.
Final task is to run datapatch with all PDBs open:
$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 19.5.0.0.0 Production on Wed Oct 16 19:40:28 2019
Copyright (c) 2012, 2019, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_8582_2019_10_16_19_40_28/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of interim SQL patches:
No interim patches found
Current state of release update SQL patches:
Binary registry:
19.5.0.0.0 Release_Update 190909180549: Installed
PDB CDB$ROOT:
Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 28-APR-19 12.51.36.895975 AM
PDB PDB$SEED:
Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 28-APR-19 12.51.37.670858 AM
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED
No interim patches need to be rolled back
Patch 30125133 (Database Release Update : 19.5.0.0.191015 (30125133)):
Apply from 19.3.0.0.0 Release_Update 190410122720 to 19.5.0.0.0 Release_Update 190909180549
No interim patches need to be applied
Installing patches...
Patch installation complete. Total patches installed: 2
Validating logfiles...done
Patch 30125133 apply (pdb CDB$ROOT): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30125133/23151502/30125133_apply_CDB2_CDBROOT_2019Oct16_19_41_08.log (no errors)
Patch 30125133 apply (pdb PDB$SEED): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30125133/23151502/30125133_apply_CDB2_PDBSEED_2019Oct16_19_41_54.log (no errors)
SQL Patching tool complete on Wed Oct 16 19:42:20 2019
Done with 19.5.0 patching.
Just a final check I’d like to do for new parameters and underscores since 19.4.0:
_bug29302220_tcpinfo_statistics_save_atexit _bug29903454_ksws_enable_alb _session_modp_list
These are new in 19.5.0.
Oracle Database 18.8.0
I downloaded:
- Database Update 18.8.0.0.191015 Patch 30112122 for UNIX
- Readme
- Opatch 12.2.0.1.17 minimum required
Quick check for the current opatch version in my home:
$ $ORACLE_HOME/OPatch/opatch version OPatch Version: 12.2.0.1.17 OPatch succeeded.
According to the patches Readme, this is sufficient.
Then I unpack the downloaded zip and check for conflicts:
$ cd 30112122 $ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/18 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/18/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.4.0 Log file location : /u01/app/oracle/product/18/cfgtoollogs/opatch/opatch2019-10-16_17-09-11PM_1.log Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed. OPatch succeeded.
Then I’m going forward to apply the patch. This is just an excerpt of the messages opatch displays:
$ $ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/18 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/18/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.4.0 Log file location : /u01/app/oracle/product/18/cfgtoollogs/opatch/opatch2019-10-16_17-25-16PM_1.log Verifying environment and performing prerequisite checks... -------------------------------------------------------------------------------- Start OOP by Prereq process. Launch OOP... Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/18 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/18/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.4.0 Log file location : /u01/app/oracle/product/18/cfgtoollogs/opatch/opatch2019-10-16_17-25-30PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 30112122 Do you want to proceed? [y|n] Y (auto-answered by -silent) User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/18') Is the local system ready for patching? [y|n] Y (auto-answered by -silent) User Responded with: Y Backing up files... Applying interim patch '30112122' to OH '/u01/app/oracle/product/18' ApplySession: Optional component(s) [ oracle.assistants.server.oui, 18.0.0.0.0 ] , [ oracle.has.crs, 18.0.0.0.0 ] , [ oracle.network.gsm, 18.0.0.0.0 ] , [ oracle.rdbms.tg4db2, 18.0.0.0.0 ] , [ oracle.network.cman, 18.0.0.0.0 ] , [ oracle.crs, 18.0.0.0.0 ] , [ oracle.assistants.usm, 18.0.0.0.0 ] , [ oracle.tfa, 18.0.0.0.0 ] , [ oracle.rdbms.tg4tera, 18.0.0.0.0 ] , [ oracle.rdbms.tg4msql, 18.0.0.0.0 ] , [ oracle.rdbms.tg4ifmx, 18.0.0.0.0 ] , [ oracle.assistants.asm, 18.0.0.0.0 ] , [ oracle.rdbms.tg4sybs, 18.0.0.0.0 ] , [ oracle.ons.daemon, 18.0.0.0.0 ] , [ oracle.options.olap, 18.0.0.0.0 ] , [ oracle.net.cman, 18.0.0.0.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.rdbms.rsf, 18.0.0.0.0... Patching component oracle.rdbms, 18.0.0.0.0... ... Patching component oracle.precomp.common, 18.0.0.0.0... Patch 30112122 successfully applied. Sub-set patch [29301631] has become inactive due to the application of a super-set patch [30112122]. Sub-set patch [27908644] has become inactive due to the application of a super-set patch [30112122]. Please refer to Doc ID 2161861.1 for any possible further required actions. Log file location: /u01/app/oracle/product/18/cfgtoollogs/opatch/opatch2019-10-16_17-25-30PM_1.log OPatch succeeded.
Finally, I need to run datapatch on my databases with all PDBs being up.
$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 18.0.0.0.0 Production on Wed Oct 16 17:38:46 2019
Copyright (c) 2012, 2019, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_5075_2019_10_16_17_38_46/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of interim SQL patches:
Interim patch 27923415 (OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)):
Binary registry: Installed
PDB CDB$ROOT: Applied successfully on 25-JUL-18 03.20.20.172886 PM
PDB PDB$SEED: Applied successfully on 25-JUL-18 03.20.21.628515 PM
Current state of release update SQL patches:
Binary registry:
18.8.0.0.0 Release_Update 1909032123: Installed
PDB CDB$ROOT:
Applied 18.3.0.0.0 Release_Update 1806280943 successfully on 25-JUL-18 03.20.20.166046 PM
PDB PDB$SEED:
Applied 18.3.0.0.0 Release_Update 1806280943 successfully on 25-JUL-18 03.20.21.623225 PM
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED
No interim patches need to be rolled back
Patch 30112122 (Database Release Update : 18.8.0.0.191015 (30112122)):
Apply from 18.3.0.0.0 Release_Update 1806280943 to 18.8.0.0.0 Release_Update 1909032123
No interim patches need to be applied
Installing patches...
Patch installation complete. Total patches installed: 2
Validating logfiles...done
Patch 30112122 apply (pdb CDB$ROOT): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30112122/23124066/30112122_apply_CDB2_CDBROOT_2019Oct16_17_39_11.log (no errors)
Patch 30112122 apply (pdb PDB$SEED): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30112122/23124066/30112122_apply_CDB2_PDBSEED_2019Oct16_17_40_36.log (no errors)
SQL Patching tool complete on Wed Oct 16 17:41:41 2019
Looks good.
Oracle Database 12.2.0.1.0 with Oct 2019 RU
I downloaded:
- Database Oct 2019 Update 12.2.0.1.191015 Patch 30138470 for UNIX
- Readme
- Opatch 12.2.0.1.17 minimum required
At first, I check for the correct opatch version in this home as well.
$ $ORACLE_HOME/OPatch/opatch version OPatch Version: 12.2.0.1.17 OPatch succeeded.
That is more or less obvious as the opatch tools labeled 19c, 18c and 12.2.0.1 are all the same, just with different download tags on the MOS page. But as all these releases are from the same family, it makes sense to have the identical opatch tool, and not three different ones. If only MOS would show this clearly …
Next task: I will unpack the patch and run the conflict check.
$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/12.2.0.1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.2.0.1/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.1.4 Log file location : /u01/app/oracle/product/12.2.0.1/cfgtoollogs/opatch/opatch2019-10-16_19-57-38PM_1.log Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed. OPatch succeeded.
Then I apply the patch bundle – and I cut some of the output to shorten it:
$ $ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.2.0.1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.2.0.1/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.1.4 Log file location : /u01/app/oracle/product/12.2.0.1/cfgtoollogs/opatch/opatch2019-10-16_19-58-47PM_1.log Verifying environment and performing prerequisite checks... -------------------------------------------------------------------------------- Start OOP by Prereq process. Launch OOP... Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.2.0.1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.2.0.1/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.1.4 Log file location : /u01/app/oracle/product/12.2.0.1/cfgtoollogs/opatch/opatch2019-10-16_19-59-01PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 30138470 Do you want to proceed? [y|n] Y (auto-answered by -silent) User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/12.2.0.1') Is the local system ready for patching? [y|n] Y (auto-answered by -silent) User Responded with: Y Backing up files... Applying interim patch '30138470' to OH '/u01/app/oracle/product/12.2.0.1' ApplySession: Optional component(s) [ oracle.oid.client, 12.2.0.1.0 ] , [ oracle.nlsrtl.companion, 12.2.0.1.0 ] , [ oracle.ons.daemon, 12.2.0.1.0 ] , [ oracle.has.crs, 12.2.0.1.0 ] , [ oracle.network.cman, 12.2.0.1.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.network.rsf, 12.2.0.1.0... ...cutout-here... Patching component oracle.precomp.lang, 12.2.0.1.0... OPatch found the word "error" in the stderr of the make command. Please look at this stderr. You can re-run this make command. Stderr output: chmod: changing permissions of ‘/u01/app/oracle/product/12.2.0.1/bin/extjobO’: Operation not permitted make: [iextjob] Error 1 (ignored) Patch 30138470 successfully applied. Sub-set patch [29314339] has become inactive due to the application of a super-set patch [30138470]. Please refer to Doc ID 2161861.1 for any possible further required actions. OPatch Session completed with warnings. Log file location: /u01/app/oracle/product/12.2.0.1/cfgtoollogs/opatch/opatch2019-10-16_19-59-01PM_1.log OPatch completed with warnings.
Interesting. I saw this in a previous patch run before. A quick check on MOS tells me two things:
- It is a known issue and the explanation is in MOS Note: 2265726.1 (Applying Proactive Bundle / PSU Patch fails with Error: “chmod: changing permissions of `$ORACLE_HOME/bin/extjobO’: Operation not permitted”)
- As this can be safely ignored according to the MOS note (
"The Issue / Warning "chmod: changing permissions of '$ORACLE_HOME/bin/extjobO'': Operation not permitted" can be ignored safely."), nobody seemed to dare to fix it in the past 15 months.
Anyway, my final task is to execute datapatch in my two databases with all PDBs open. I copy only one output here:
$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 12.2.0.1.0 Production on Wed Oct 16 20:09:24 2019
Copyright (c) 2012, 2019, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_15340_2019_10_16_20_09_24/sqlpatch_invocation.log
Connecting to database...OK
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of SQL patches:
Bundle series DBRU:
ID 191015 in the binary registry and ID 190416 in PDB CDB$ROOT, ID 190416 in PDB PDB$SEED, ID 190416 in PDB PDB3
Adding patches to installation queue and performing prereq checks...
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED PDB3
Nothing to roll back
The following patches will be applied:
30138470 (DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015)
Installing patches...
Patch installation complete. Total patches installed: 3
Validating logfiles...
Patch 30138470 apply (pdb CDB$ROOT): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30138470/23136382/30138470_apply_CDB1_CDBROOT_2019Oct16_20_09_49.log (no errors)
Patch 30138470 apply (pdb PDB$SEED): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30138470/23136382/30138470_apply_CDB1_PDBSEED_2019Oct16_20_10_20.log (no errors)
Patch 30138470 apply (pdb PDB3): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30138470/23136382/30138470_apply_CDB1_PDB3_2019Oct16_20_10_20.log (no errors)
SQL Patching tool complete on Wed Oct 16 20:11:15 2019
And last but not least, I check for new parameters and unverscores compared to the previous patch bundle (July 2019 RU) but couldn’t find any differences.
Oracle Database 12.1.0.2.0 with Oct 2019 BP
I downloaded:
- Database Proactive Bundle Patch 12.1.0.2.191015 Patch 30070242 for UNIX
- Readme
- Opatch 12.2.0.1.17 minimum required
Unfortunately the Bundle Patch (BP) includes also GI/OCW and other things. Hence, the download package is 3.5GB in size which is huge and actually takes very long to download. The patch I’m going for, is in the subdirectory 29972716. The readme of the 12.1.0.2 Bundle Patches is unfortunately different from the other readmes – and IMHO kind of messy. I use the commands from the other readmes instead.
At first, I’m doing a quick check for my installed opatch version:
$ $ORACLE_HOME/OPatch/opatch version OPatch Version: 12.2.0.1.17 OPatch succeeded.
I don’t need to update opatch. If I would need to, the patch number is: Patch 6880880.
$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/12.1.0.2 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0.2/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.1.0.2.0 Log file location : /u01/app/oracle/product/12.1.0.2/cfgtoollogs/opatch/opatch2019-10-17_00-07-59AM_1.log Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed. OPatch succeeded.
Looks good to me. I can invoke the patch apply. As usual, I cut out a lot of information being displayed.
$ $ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0.2 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0.2/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.1.0.2.0 Log file location : /u01/app/oracle/product/12.1.0.2/cfgtoollogs/opatch/opatch2019-10-17_00-08-44AM_1.log Verifying environment and performing prerequisite checks... -------------------------------------------------------------------------------- Start OOP by Prereq process. Launch OOP... Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0.2 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0.2/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.1.0.2.0 Log file location : /u01/app/oracle/product/12.1.0.2/cfgtoollogs/opatch/opatch2019-10-17_00-09-12AM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 28259867 28731800 29141038 29496791 29972716 Do you want to proceed? [y|n] Y (auto-answered by -silent) User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/12.1.0.2') Is the local system ready for patching? [y|n] Y (auto-answered by -silent) User Responded with: Y Backing up files... Applying sub-patch '28259867' to OH '/u01/app/oracle/product/12.1.0.2' ApplySession: Optional component(s) [ oracle.has.crs, 12.1.0.2.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.oracore.rsf, 12.1.0.2.0... ...cutout-here... Patching component oracle.rdbms.crs, 12.1.0.2.0... Patching component oracle.ldap.rsf.ic, 12.1.0.2.0... Composite patch 29972716 successfully applied. Log file location: /u01/app/oracle/product/12.1.0.2/cfgtoollogs/opatch/opatch2019-10-17_00-09-12AM_1.log OPatch succeeded.
The last step I need to execute in all my databases with all PDBs been up, is datapatch.
$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 12.1.0.2.0 Production on Thu Oct 17 00:15:14 2019
Copyright (c) 2012, 2017, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_10717_2019_10_17_00_15_14/sqlpatch_invocation.log
Connecting to database...OK
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of SQL patches:
Bundle series DBBP:
ID 191015 in the binary registry and ID 180717 in PDB CDB$ROOT, ID 180717 in PDB PDB$SEED
Adding patches to installation queue and performing prereq checks...
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED
Nothing to roll back
The following patches will be applied:
29972716 (DATABASE BUNDLE PATCH 12.1.0.2.191015)
Installing patches...
Patch installation complete. Total patches installed: 2
Validating logfiles...
Patch 29972716 apply (pdb CDB$ROOT): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/29972716/23132651/29972716_apply_CDB1_CDBROOT_2019Oct17_00_15_41.log (no errors)
Patch 29972716 apply (pdb PDB$SEED): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/29972716/23132651/29972716_apply_CDB1_PDBSEED_2019Oct17_00_16_09.log (no errors)
SQL Patching tool complete on Thu Oct 17 00:16:33 2019
Done!
Oracle Database 11.2.0.4 with Oct 2019 PSU
I downloaded:
- Database PSU 11.2.0.4.191015 Patch 29913194 for UNIX
- Readme
- Opatch 11.2.0.3.20 minimum required
First check again is the opatch version:
$ $ORACLE_HOME/OPatch/opatch version OPatch Version: 11.2.0.3.21 OPatch succeeded.
That looks good. My installed opatch version is newer than the minum required one.
Now I unzip the patch and run the conflict check according to the readme:
$ cd 29913194/ $ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 11.2.0.3.21 Copyright (c) 2019, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/11.2.0.4 Central Inventory : /u01/app/oraInventory from : /etc/oraInst.loc OPatch version : 11.2.0.3.21 OUI version : 11.2.0.4.0 Log file location : /u01/app/oracle/product/11.2.0.4/cfgtoollogs/opatch/opatch2019-10-16_21-06-50PM_1.log Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed. OPatch succeeded.
Ok, I can apply the patch bundle. I had almost forgotten that with the older opatch version I will have to answer these annoying questions:
$ $ORACLE_HOME/OPatch/opatch apply Oracle Interim Patch Installer version 11.2.0.3.21 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/11.2.0.4 Central Inventory : /u01/app/oraInventory from : /etc/oraInst.loc OPatch version : 11.2.0.3.21 OUI version : 11.2.0.4.0 Log file location : /u01/app/oracle/product/11.2.0.4/cfgtoollogs/opatch/opatch2019-10-16_21-07-48PM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 29497421 29913194 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name: You have not provided an email address for notification of security issues. Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/11.2.0.4') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying sub-patch '29497421' to OH '/u01/app/oracle/product/11.2.0.4' Patching component oracle.rdbms, 11.2.0.4.0... Patching component oracle.rdbms.dbscripts, 11.2.0.4.0... Patching component oracle.rdbms.rman, 11.2.0.4.0... Patching component oracle.rdbms.rsf, 11.2.0.4.0... Patching component oracle.ldap.rsf, 11.2.0.4.0... Patching component oracle.ldap.rsf.ic, 11.2.0.4.0... Patching component oracle.oracore.rsf, 11.2.0.4.0... Patching component oracle.ctx, 11.2.0.4.0... Applying sub-patch '29913194' to OH '/u01/app/oracle/product/11.2.0.4' Patching component oracle.rdbms, 11.2.0.4.0... Patching component oracle.rdbms.dbscripts, 11.2.0.4.0... Patching component oracle.rdbms.rsf, 11.2.0.4.0... Patching component oracle.network.rsf, 11.2.0.4.0... Patching component oracle.ldap.rsf, 11.2.0.4.0... Patching component oracle.ldap.rsf.ic, 11.2.0.4.0... Patching component oracle.rdbms.util, 11.2.0.4.0... Composite patch 29913194 successfully applied. Log file location: /u01/app/oracle/product/11.2.0.4/cfgtoollogs/opatch/opatch2019-10-16_21-07-48PM_1.log OPatch succeeded.
Afterwards, I need to apply the SQL/PLSQL changes to my two databases once I started them up again.
SQL> @?/rdbms/admin/catbundle.sql psu apply
The output tells a lot about which packages get adjusted.
Final task
As a final task I clean up at least a little bit in my homes with opatch util cleanup. I did this with other homes, too.
$ $ORACLE_HOME/OPatch/opatch util cleanup Oracle Interim Patch Installer version 12.2.0.1.17 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.2.0.1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.2.0.1/oraInst.loc OPatch version : 12.2.0.1.17 OUI version : 12.2.0.1.4 Log file location : /u01/app/oracle/product/12.2.0.1/cfgtoollogs/opatch/opatch2019-10-16_20-15-00PM_1.log Invoking utility "cleanup" OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories. You will be still able to rollback patches after this cleanup. Do you want to proceed? [y|n] y User Responded with: Y Backup area for restore has been cleaned up. For a complete list of files/directories deleted, Please refer log file. OPatch succeeded.
–Mike
Hi Mike,
does “use always a new home you patch” mean we should use Fleet Patching and Provisioning formerly Rapid Home Provisioning?
Or are there other tools to switch between a home?
I’m asking because FPP doesn’t seem to work in Oracle Solaris Local Zones due to missing ACFS support. I’ve raised a SR for this.
Best regards
Sven
Hi Sven,
no, not at all – it would be nice if you use RHP/FPP but you don’t have to.
I call it the “dual home” strategy. Lets say you’d like to move from 19.3.0 to 19.5.0.
Then you install 19.3.0 into a second home. I call my homes in such cases “/u01/app/oracle/product/19.3.0” and “../19.5.0”. Once 19.3.0 software is installed into the 19.5.0 home, you will apply the 19.5.0 RU to this home. And once this is completed, you shutdown your database running from the 19.3.0 home, change the environment now pointing to the 19.5.0 home, start your database again – and invoke “datapatch -verbose”.
That’s it. Minimum downtime. Simple fallback in case anything goes wrong. And no additional tools required for this.
Cheers,
Mike
Thanks Mike! I might test this in case my FPP SR doesn’t work out.
Just in case you haven’t stumbled accross this cool new feature: Starting with 18c you can install a db home and apply patches in one step to save some time.
./runInstaller -applyRU -applyOneOffs
Best regards
Sven
Hi Sven,
I didn’t know that – will try it asap.
Cheers and thanks for the hint!
Mike
Hello Mike,
You don’t have JVM in your local databases, but I do. In that case, would I apply the combo patch, and then run datapatch -verbose twice? (one after the db patch, with DB open, and one time after the java patch, with DB ‘startup upgrade’ state?) or is it still valid to run it only once with DB open?
STARTUP UPGRADE stage for JVM is not required – we are working on correcting the readmes.
And even when you apply the COMBO patch, you have different “opatch apply” runs for each of them.
Actually my understanding of datapatch tells me that you need to run it only once – but let me double check.
CHeers,
Mike
Paul,
you run datapatch only ONCE!
It will take care!
Cheers,
Mike
I know this is different than your environment since you do not have ASM, but thought I would check and see if you have any input. We are building a new system, and have installed GI/ASM 19c and database 18c. The GI home has been patched with the October RU (30116789), but when we try to apply the October RU GI/DB combo (30116795) to the 18 DB home it fails with multiple make errors. The first one is:
[Oct 24, 2019 10:06:58 AM] [WARNING]OUI-67200:Make failed to invoke “/usr/bin/make -f ins_precomp.mk procob ORACLE_HOME=/u01/app/oracle/product/18/dbhome_1″….’/u01/app/oracle/product/19/grid/lib/libons.so: undefined reference to `memcpy@GLIBC_2.14′
/bin/chmod: cannot access ‘/u01/app/oracle/product/18/dbhome_1/precomp/lib/procob’: No such file or directory
make[1]: *** [/u01/app/oracle/product/18/dbhome_1/precomp/lib/procob] Error 1
make: *** [procob] Error 2
‘
[Oct 24, 2019 10:06:58 AM] [INFO] Re-link fails on target “procob”.
[Oct 24, 2019 10:06:58 AM] [INFO] Running make for target “proc”
[Oct 24, 2019 10:06:58 AM] [INFO] Start invoking ‘make’ at Thu Oct 24 10:06:58 MDT 2019Thu Oct 24 10:06:58 MDT 2019
[Oct 24, 2019 10:06:58 AM] [INFO] Finish invoking ‘make’ at Thu Oct 24 10:06:58 MDT 2019
[Oct 24, 2019 10:06:58 AM] [WARNING]OUI-67200:Make failed to invoke “/usr/bin/make -f ins_precomp.mk proc ORACLE_HOME=/u01/app/oracle/product/18/dbhome_1″….’/u01/app/oracle/product/19/grid/lib/libons.so: undefined reference to `memcpy@GLIBC_2.14′
/bin/chmod: cannot access ‘/u01/app/oracle/product/18/dbhome_1/precomp/lib/proc’: No such file or directory
make[1]: *** [/u01/app/oracle/product/18/dbhome_1/precomp/lib/proc] Error 1
make: *** [proc] Error 2
‘
[Oct 24, 2019 10:06:58 AM] [INFO] Re-link fails on target “proc”.
[Oct 24, 2019 10:06:58 AM] [INFO] ——————————————————————————–
Failed to run make commands. Please contact Oracle Support.
[Oct 24, 2019 10:06:58 AM] [SEVERE] OUI-67115:OPatch failed to restore OH ‘/u01/app/oracle/product/18/dbhome_1’. Consult OPatch document to restore the home manually before proceeding.
[Oct 24, 2019 10:06:58 AM] [WARNING]OUI-67124:
NApply was not able to restore the home. Please invoke the following scripts:
– restore.[sh,bat]
– make.txt (Unix only)
to restore the ORACLE_HOME. They are located under
“/u01/app/oracle/product/18/dbhome_1/.patch_storage/NApply/2019-10-24_10-05-08AM”
——–
This is a fresh install on OEL 7 using the RedHat kernel.
Hi Ross,
I did a quick MOS search assuming that you did this already as well.
I couldn’t find a matching solution.
Please open an SR – and I feel very sorry reading this experience 🙁
Thanks,
Mike
Thanks Mike, we have opened one as well. With a 19c DB home the patch applies. We haven’t tried it with 12.2.0.1 or anything else lower. And we know 18c ASM home the 18c DB patch applies because that’s how our other systems are configured.
Vielen Dank,
Ross
During the applied of GI Update OCT2019 (12.2.0.1.191015) 30116802 on my RDBMS home 12.2.0.1. I receive this error from opatchauto apply command:
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target “proc”.
Re-link fails on target “procob”.
I could not find a clear MOS on the issues and following the logs I saw this message
/bin/chmod: cannot access ‘/u01/app/oracle/product/12.2.0/db/db_1/precomp/lib/procob’: No such file or directory
make[1]: *** [/u01/app/oracle/product/12.2.0/db/db_1/precomp/lib/procob] Error 1
make: *** [procob] Error 2
The problem was resolve it just with simple touching those files, that did not exists
touch $ORACLE_HOME/precomp/lib/procob
touch $ORACLE_HOME/precomp/lib/proc
and resuming the patching
# /u01/app/oracle/product/12.2.0/db/db_1/OPatch/opatchauto resume
it works like a charm!!
==Following patches were SUCCESSFULLY applied:
Patch: /u01/medios/30116802/30138470
Log: /u01/app/oracle/product/12.2.0/db/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2019-11-06_11-50-19AM_1.log
OPatchauto session completed at Wed Nov 6 13:43:42 2019
Time taken to complete the session 39 minutes, 12 seconds
Hi 🙂
Thanks for this hint – my question would be if you did open an SR for this or did you solve it by yourself?
Cheers,
Mike
Hi mike. I was in a hurry to patching testbed env that I didnt have a chance to open SR. That’s why I decide to give a try.
Thanks – and no worries, I just wanted to check if it has been documented on our side already.
Cheers,
Mike
Thank you! We did end opening up an SR with Oracle, and had to start with an earlier RU and and earlier version of patch. After applying those, we were then apply to apply the latest.
Hi Mike,
I am experiencing the very same error as here described:
https://www.dbarj.com.br/en/2019/10/datapatch-fails-with-ora-29516-ora-00604-and-ora-29532-java-lang-arrayindexoutofboundsexception/
As we install our systems via Ansible fresh from the start, installing 19.4.0 first and then 19.5.0 does not seems to be a valid option for us in the long run.
MOS also has no solution (or even a bug) for this.
Maybe you can help?
Thanks a lot and best regards,
Matthias.
Matthias,
please log an SR and insist until you receive a solution.
Thanks,
Mike
Hi mike. Nice and good work. I have a doubt relate to patching SIHA env with asm on mixed versions, specifically rdbms home. Which one to apply on RDBMS
GI/ASM Home 19c:
Applied RU GI Update OCT2019 (19.5.0.0.0)
RDBMS HOME 12.2
RU GI Update OCT2019 (12.2.0.1.191015)
.OR.
RU DB Update OCT2019(12.2.0.1.191015)
Hi Mike
Regarding “dual home” strategy is there any special requirements regarding OPATCH directory names which contain hard-coded value pointing to original rdbms home ?
CON_NAME
——————————
CDB$ROOT
SQL> select OWNER, DIRECTORY_NAME, DIRECTORY_PATH from dba_directories where DIRECTORY_NAME like ‘%OPATCH%’ ;
OWNER DIRECTORY_NAME DIRECTORY_PATH
———- —————————— ————————————————–
SYS OPATCH_LOG_DIR /u01/app/oracle/product/12.1.0.2/QOpatch
SYS OPATCH_SCRIPT_DIR /u01/app/oracle/product/12.1.0.2/QOpatch
SYS OPATCH_INST_DIR /u01/app/oracle/product/12.1.0.2/OPatch
SYS OPATCH_TEMP_DIR /u01/app/oracle/product/12.1.0.2/QOpatch
THanks by advance for your suggestion
Hi Christophe,
I have never seen any issues so far with it. In fact, most customers I work with, use dual-homes.
And nobody reported an issue with patching regarding these entries. You will need your opatch locally in your current home when patching.
But that’s it.
Thanks,
Mike