Actually I carry around this topic with me since the October 2019 workshop in Dubai. We’ve had one of these OJVM Patching discussions I’d like to escape usually. Anyhow, during this session, it dawned me that the STARTUP UPGRADE requirement in the OJVM readme may be not correct anymore. So the question will be: Do you need STARTUP UPGRADE for OJVM?
Photo by Nik Shuliahin on Unsplash
A little bit of history
You will find a good amount of blog posts about OJVM here. OJVM stands for Oracle Java Virtual Machine. It is a very powerful thing and allows you to execute java code within the database much quicker than from outside. I have attended sessions by ACE Directors who showed impressive flame graphs about the performance differences. But as powerful OJVM is, not everybody uses it. And the power aspect leads to a lot of security issues in the past years.
Hence, I think it is not the worst idea to deinstall it when you can ensure that nobody is using it.
Since Oracle Database 18.4.0 OJVM is does really allow RAC rolling patching. Before, it is a conditional rolling install. Please refer to MOS Note: 2217053.1 – RAC Rolling Install Process for the “Oracle JavaVM Component Database PSU/RU” (OJVM PSU/RU) Patches if you seek for further details on this complex topic.
This important note contains also sections covering “Is OJVM installed?” and “Is OJVM used?“.
And just on the side, while OJVM is RAC Rolling, it is still not Data Guard Standby First Installable.
Do you need STARTUP UPGRADE for OJVM?
But the question which bothered me is: Do you need STARTUP UPGRADE for OJVM?
Let’s have a quick look at the 19.6.0 OJVM January 2020 readme. Under 1.4.1 it has this section explaining that STARTUP UPGRADE is required – with one exception, I will explain in a second:
But what I didn’t understand: Why should a STARTUP UPGRADE be required when datapatch requires all databases and PDBs in OPEN mode? Especially since all the patch bundles, RUs, BPs and PSUs per se are rolling installable.
Well, we’ve had an internal discussion. And as a result, the “Footnote 1” got added to the note, saying:
Following the link gets you to:
Aha! When you follow the approach in MOS Note: 2217053.1 then there’s no need for STARTUP UPGRADE anymore.
Then let us check what MOS Note: 2217053.1 has to say about this topic:
- If a database has any current or past exposure to an OJVM PSU patchset that was installed via “startup upgrade”, it will insist that all future OJVM PSU patchset actions also use “startup upgrade”. To workaround this expectation of “startup upgrade” originating from a previous OJVM PSU patchset installed via “startup upgrade”, the new datapatch parameter of “-skip_upgrade_check” should be used, as shown in the following example:
### Replacing the Java system classes while database is up
### Do not use -force flag for datapatch
cd $ORACLE_HOME/OPatch
./datapatch -verbose -skip_upgrade_check
Now we come closer to it. To me this reads: If you used STARTUP UPGRADE before, then use the flag now to avoid the need for STARTUP UPGRADE. Hence, in a RAC environment – and those are the environments where you can avoid downtime patching OJVM in a rolling fashion, since at least 18.4.0 you can do the entire process fully rolling. Including of course the datapatch run.
And this was the crucial topic we discussed in Dubai in October 2019. What does it help to apply the binaries rolling when I will have downtime to run datapatch? I hope this is clarified now. Even though I would have wished a way more obvious edit to the OJVM patches readme.
How about in non-RAC environments?
First of all, in a non-RAC environment you will encounter downtime anyway. You will at best case switch homes from the current one to the patched one. Hence, you will need to take your database down and start it up again. A little bit extra downtime for another STARTUP UPGRADE sequence won’t hurt too much. But actually I try to avoid such phases whenever possible.
Then, the crucial question is:
Do you need STARTUP UPGRADE in a non-RAC environment?
The OJVM patch readme says: Yes. The steps haven’t been removed. But datapatch patches the database while it is up and running without the need for STARTUP UPGRADE. What should be different from a situation in a 4-node RAC cluster when applying changes with datapatch from a non-RAC database?
I don’t see it – and I’d like to try out such things. So I installed the patch into my environment, and configured OJVM into my 19c CDB.
To configure OJVM into my existing CDB, I could have used the guidance of MOS Note: 2262919.1 – Repairing/Reinstalling JVM To A Multitenant Database (CDB and all PDBs) which by the way explains at first how to remove OJVM from a CDB database.
But instead I used the DBCA:
See the summary:
Quick check after I configured it:
CON_ID COMP_ID STATUS VERSION ---------- -------- ------------ ---------- 1 CATALOG VALID 19.0.0.0.0 1 CATJAVA VALID 19.0.0.0.0 1 CATPROC VALID 19.0.0.0.0 1 JAVAVM VALID 19.0.0.0.0 1 OLS VALID 19.0.0.0.0 1 OWM VALID 19.0.0.0.0 1 RAC OPTION OFF 19.0.0.0.0 1 XDB VALID 19.0.0.0.0 1 XML VALID 19.0.0.0.0 2 CATALOG VALID 19.0.0.0.0 2 CATJAVA VALID 19.0.0.0.0 2 CATPROC VALID 19.0.0.0.0 2 JAVAVM VALID 19.0.0.0.0 2 OLS VALID 19.0.0.0.0 2 OWM VALID 19.0.0.0.0 2 RAC OPTION OFF 19.0.0.0.0 2 XDB VALID 19.0.0.0.0 2 XML VALID 19.0.0.0.0
Looks ok.
Applying the 19.6.0 OJVM without STARTUP UPGRADE
First step is to verify that my opatch version is good enough to apply the OJVM Jan 2020 RU to my existing 19.6.0 home. The readme requires only the “10” version. As I have the “17” version installed, all should be fine.
Next step, after I downloaded and unzipped the patch 30484981, is the conflict check.
$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/19
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19/oraInst.loc
OPatch version : 12.2.0.1.17
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2020-01-21_22-55-00PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
Ok, I’m safe.
Let’s apply the patch to my home.
$ $ORACLE_HOME/OPatch/opatch apply
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/19
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19/oraInst.loc
OPatch version : 12.2.0.1.17
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2020-01-21_23-15-41PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 30484981
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/19')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30484981' to OH '/u01/app/oracle/product/19'
Patching component oracle.javavm.server, 19.0.0.0.0...
Patching component oracle.javavm.server.core, 19.0.0.0.0...
Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...
Patching component oracle.rdbms, 19.0.0.0.0...
Patch 30484981 successfully applied.
Log file location: /u01/app/oracle/product/19/cfgtoollogs/opatch/opatch2020-01-21_23-15-41PM_1.log
OPatch succeeded.
But why should I start my database now with STARTUP UPGRADE?
Test 1 – No STARTUP UPGRADE and no datapatch flag
In my first test, I pretend to have no idea and do exactly the same I do for my databases. This means, I don’t use the datapatch flag mentioned in MOS Note: 2217053.1 to skip the upgrade check. I started my database in regular mode (no STARTUP UPGRADE) and ran datapatch:
$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 19.6.0.0.0 Production on Tue Jan 21 23:21:41 2020
Copyright (c) 2012, 2019, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_8154_2020_01_21_23_21_41/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of interim SQL patches:
Interim patch 30484981 (OJVM RELEASE UPDATE: 19.6.0.0.200114 (30484981)):
Binary registry: Installed
PDB CDB$ROOT: Not installed
PDB PDB$SEED: Not installed
Current state of release update SQL patches:
Binary registry:
19.6.0.0.0 Release_Update 191217155004: Installed
PDB CDB$ROOT:
Applied 19.6.0.0.0 Release_Update 191217155004 successfully on 21-JAN-20 09.23.28.157993 PM
PDB PDB$SEED:
Applied 19.6.0.0.0 Release_Update 191217155004 successfully on 21-JAN-20 09.23.29.338333 PM
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED
No interim patches need to be rolled back
No release update patches need to be installed
The following interim patches will be applied:
30484981 (OJVM RELEASE UPDATE: 19.6.0.0.200114 (30484981))
Installing patches...
Patch installation complete. Total patches installed: 2
Validating logfiles...done
Patch 30484981 apply (pdb CDB$ROOT): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30484981/23248235/30484981_apply_CDB2_CDBROOT_2020Jan21_23_22_56.log (no errors)
Patch 30484981 apply (pdb PDB$SEED): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30484981/23248235/30484981_apply_CDB2_PDBSEED_2020Jan21_23_23_37.log (no errors)
SQL Patching tool complete on Tue Jan 21 23:23:51 2020
To my big surprise, the patch applied flawless. Without thinking a second about STARTUP UPGRADE. This may happen because I never applied any OJVM patches in STARTUP UPGRADE mode before, and hence, the -skip_upgrade_check flag was not needed. I won’t complain about this behavior.
Test 2 – No STARTUP UPGRADE and the datapatch flag
Still, I’d like to check whether using the flag “-skip_upgrade_check” works and leads to the same result.
$ $ORACLE_HOME/OPatch/datapatch -verbose -skip_upgrade_check
SQL Patching tool version 19.6.0.0.0 Production on Tue Jan 21 23:31:43 2020
Copyright (c) 2012, 2019, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_8288_2020_01_21_23_31_43/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of interim SQL patches:
Interim patch 30484981 (OJVM RELEASE UPDATE: 19.6.0.0.200114 (30484981)):
Binary registry: Installed
PDB CDB$ROOT: Not installed
PDB PDB$SEED: Not installed
Current state of release update SQL patches:
Binary registry:
19.6.0.0.0 Release_Update 191217155004: Installed
PDB CDB$ROOT:
Applied 19.6.0.0.0 Release_Update 191217155004 successfully on 21-JAN-20 09.23.28.157993 PM
PDB PDB$SEED:
Applied 19.6.0.0.0 Release_Update 191217155004 successfully on 21-JAN-20 09.23.29.338333 PM
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED
No interim patches need to be rolled back
No release update patches need to be installed
The following interim patches will be applied:
30484981 (OJVM RELEASE UPDATE: 19.6.0.0.200114 (30484981))
Installing patches...
Patch installation complete. Total patches installed: 2
Validating logfiles...done
Patch 30484981 apply (pdb CDB$ROOT): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30484981/23248235/30484981_apply_CDB2_CDBROOT_2020Jan21_23_32_44.log (no errors)
Patch 30484981 apply (pdb PDB$SEED): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30484981/23248235/30484981_apply_CDB2_PDBSEED_2020Jan21_23_33_23.log (no errors)
SQL Patching tool complete on Tue Jan 21 23:33:38 2020
My personal summary
As I explained above, we discussed this internally. But I can’t update readmes by myself. And I don’t want to pretend that I’m right. But to me it looks as if the STARTUP UPGRADE requirement in the OJVM patches’ readme is wrong.
In a RAC environment, MOS Note: 2217053.1 explains clearly how to patch with datapatch without the need for STARTUP UPGRADE. The readme should be much clearer here instead of hiding the important piece under a footnote and another click. Especially under the consideration that this is really impacting – and not needing a STARTUP UPGRADE in a RAC environment is a big relief.
Even though you could argue that it doesn’t make a big difference in a non-RAC system where you can’t apply the OJVM or any other patch binaries rolling anyway, whether you use STARTUP UPGRADE or not, I can see that especially in an environment with many PDBs it may take a bit until datapatch is finished. As STARTUP UPGRADE means downtime, I’d rather like to avoid it. And so far, with everything I tested and everything I know, I don’t see any reason to apply the changes with datapatch while my database is in UPGRADE mode.
But again, the non-RAC test is my personal view on this topic, not an official Oracle standpoint according to the OJVM readmes. I’d just like to share my experience and ideas with you.
Further Information and Links
- MOS Note: 2217053.1 – RAC Rolling Install Process for the “Oracle JavaVM Component Database PSU/RU” (OJVM PSU/RU) Patches
- MOS Note: 2262919.1 – Repairing/Reinstalling JVM To A Multitenant Database (CDB and all PDBs)
- OJVM January 2020 RU for Oracle 19c – patch 30484981
- Does OJVM require a specific RU?
- Are OJVM Patches included in RUs and RURs?
- The OJVM Patching Saga – and how to solve it – Part.5
- OJVM Patching – Standby First, yes or no?
- JAVAVM and XML cleanup in the database
- Patching all my environments with the January 2020 Patch Bundles
–Mike
Been applying PSUs then RUs since they were created on a non-CDB single instance Oracle database that is currently at 19.5.0.0.0. Stopped using “startup upgrade” a while back when it was not indicated to do so in readme file. Have always been applying the OJVM/DB PSU/RU bundle. Have never used the ‘skip_update_check’ option. The updates always apply just fine. Just applied the Oct 2019 19c bundle and it worked great!
Can we get an “offical” response on this topic from Oracle? Are the readme’s going to get updates soon?
There is one already – scroll down to the MOS Note I linked – this is official!
Cheers,
Mike
it looks like that the out-of-place patching is “must” for this (per the readme doc). However – you seems not using out of place patching here. Just wondering it’s really a must or nice-to-do thing? (I haven’t tested it yet; but will try sometime soon).
from “Patch 30484981 – Oracle JavaVM Component Release Update 19.6.0.0.200114” readme file –
This patch is now Oracle RAC Rolling installable. To use the Oracle RAC Rolling approach, out-of-place patching of the Oracle home is mandatory, as is the use of database services and SRVCTL to control instance and service operations. For further information, refer to MOS NOTE 2217053.1, RAC Rolling Install Process for the “Oracle JavaVM Component Database PSU/RU” (OJVM PSU/RU) Patches.
Well, I don’t have enough space in my lab environment – and I don’t have a RAC system.
Generally, out of place patching is the much safer way – and if you’d like to do RAC rolling, then you’ll have to.
But technically you can do in-place, too, but with a risk involved. And not RAC rolling.
Cheers,
Mike
I don’t see the issue as I don’t release the database for any use, until Datapatch has completed anyway. So it doesn’t matter what mode it runs in, since I don’t expect and want anything/anybody using it. RAC or not doesn’t matter in that context.
But, Mike, could i apply RU OJVM 19.16 for example, in combination with DBRU 19.16 completely rolling as usual at in-place mode ? i have a 2 nodes rac database in ODA .
obvious adding the parameter in datapatch:
./datapatch -verbose -skip_upgrade_check
you Note is very explained , but docid 2217053.1 is not clearly.
thanks.
Hi Raul,
yes you can.
Cheers
Mike