Upgrading a PDB with JVM may result in ORA-7445 joevm_invokevirtual

As you know, this blog is also my brain dump about Oracle topics. Since this one happened yesterday, mostly at night with the help of my team mates and the Java team, I’d like to write it down into this blog post: Upgrading a PDB with Java may result in ORA-7445 joevm_invokevirtual() as it is still fresh and may affect you as well. If you don’t have PDBs, if you don’t have Java (or JVM) in it or if you upgraded already to Oracle Database 19c, you can stop reading now.

Photo by Fahmi Fakhrudin on Unsplash

What happens?

As you didn’t stop reading you may have:

• One or more PDBs
• Not upgraded yet to Oracle Database 19c
• Java installed in your CDB/PDB

Our customer here upgraded a single tenant database having only one PDB hosting a very important assent management application. And AutoUpgrade failed.

Or, to be more precise, the upgrade of CDB$ROOT and PDB$SEED went fine, but there was an issue with this particular PDB. In phase 53 the upgrade failed. But let me describe the sequence of issues since the actual root cause happened already before.

This is the issue the customer saw:

2021-09-28 16:22:12.749 INFO SQL*Plus data: alter pluggable database all open read write restricted instances=all
*
ERROR at line 1:
ORA-65014: invalid SHARING clause

At this point, AutoUpgrade threw an exception. But luckily my team mates have created AutoUpgrade in a way that it logs everything. Always keep in mind that time stamps are your “primary key” in such situations. You can put logs side by side, and reconstruct the order of events.

Hence, the next step I learned a long time ago in my Support days, is always: Check the alert.log.

alter pluggable database all open read write restricted instances=all
[...]
2021-09-28T16:22:10.576995+02:00
ABC(3):Opening pdb with no Resource Manager plan active
2021-09-28T16:22:12.712195+02:00
ABC(3):joxcsys_required_dirobj_exists: directory object does not exist, pid 37148 cid 3
ABC(3):joxcsys_ensure_directory_object: error 65014 while creating directory object with path /u01/app/oracle/product/19c/home/javavm/admin/, pid 37148 cid 3
ABC(3):Error 65014 during pluggable database ABC opening in read write
2021-09-28T16:22:12.717148+02:00
ABC(3):Errors in file /oradump/CDBABC/diag/rdbms/cdbabc/CDBABC/trace/CDBABC_ora_37148.trc:

Time stamps match, this looks like the issue, doesn’t it?

What really happens!

But in fact, that is just an issue following the root cause. When you check the alert.log, don’t stop at this point but scroll a bit backwards, too. The real issue has happened minutes before already.

ABC(3):joxcsys: begin CREATE JAVA SYSTEM in pid 279841
ABC(3):JIT: pid 279841 requesting full stop
2021-09-28T16:03:04.966217+02:00
PDB$SEED(2):SERVER COMPONENT id=CATPROC: status=LOADED, version=19.11.0.0.0, timestamp=2021-09-28 16:03:04 Container=PDB$SEED Id=2
2021-09-28T16:03:06.994403+02:00
ABC(3):Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x4] [PC:0x7261AC1, joevm_invokevirtual()+1425] [flags: 0x0, count: 1]
Errors in file /oradump/CDBABC/diag/rdbms/cdbabc/CDBABC/trace/CDBABC_ora_279841.trc  (incident=348781) (PDBNAME=ABC):
ORA-07445: exception encountered: core dump [joevm_invokevirtual()+1425] [SIGSEGV] [ADDR:0x4] [PC:0x7261AC1] [Address not mapped to object] []
ABC(3):Incident details in: /oradump/CDBABC/diag/rdbms/cdbabc/CDBABC/incident/incdir_348781/CDBABC_ora_279841_i348781.trc
ABC(3):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

Don’t be puzzled by the logs from PDB$SEED. Since this is a single tenant, and the box has enough CPU power, the PDB$SEED and the PDB called ABC get upgraded in parallel. But the core dump, signaled by an ORA-7445 happens in the PDB called ABC. And it happens when the Java system gets created:

begin CREATE JAVA SYSTEM

Did I tell you that AutoUpgrade is great?!

The core dump happened at 16:03:06h in PDB “ABC”. And AutoUpgrade logs everything. Then I check the two worker logs of PDB “ABC”. And in one of them I find the error (or actually Joe from our team found it at first):

16:02:58 SQL>
16:02:58 SQL> -- Load all the Java classes
16:02:58 SQL> begin if initjvmaux.startstep('CREATE_JAVA_SYSTEM') then
16:02:58   2      initjvmaux.rollbacksetup;
16:02:58   3      commit;
16:02:58   4      initjvmaux.rollbackset;
16:02:58   5      initjvmaux.exec('create or replace java system');
16:02:58   6      commit;
16:02:58   7      initjvmaux.rollbackcleanup;
16:02:58   8      initjvmaux.endstep;
16:02:58   9      dbms_registry.update_schema_list('JAVAVM',
16:02:58  10            dbms_registry.schema_list_t('OJVMSYS'));
16:02:58  11  end if; end;
16:02:58  12  /
begin if initjvmaux.startstep('CREATE_JAVA_SYSTEM') then
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
ERROR:
ORA-03114: not connected to ORACLE


SP2-1519: Unable to write to the error log table SYS.REGISTRY$ERROR
Process ID: 279841
Session ID: 1752 Serial number: 22171

This is where is fails.

The root cause

At this point Hector had found already the matching MOS Note:

MOS Note: 2691097.1 – 12.2 or higher Database Upgrade may fail with ORA-07445: exception encountered: core dump [joevm_invokevirtual()+1414]

In case the document is not visible on MOS, you will find the workaround it contains below. If you are unsure about the workaround, please check with Oracle Support and ask the support engineer to provide the contents of the document.

This note refers to the unpublished BUG 28745878 – FAILURES IN PDB DERIVED FROM NONCDB DURING OR AFTER OJVM UPGRADE OR PATCH. And since our Java dev team reacted super-fast (thanks a lot!!) we understood what has happened.

This bug got fixed in Oracle 19.2 code – but that doesn’t mean you can’t hit it. Only if you are already on Oracle 19c you won’t see it anymore. But for those of you who had a non-CDB not having Long Identifiers (either before 12.2.0.1 or COMPATIBLE was below 12.2.0), plugged it into a CDB having Long Identifiers (default since 12.2.0.1 as soon as COMPATIBLE is 12.2.0 or higher), you may hit this issue.

And of course, you will only see it in these cases when JVM is present in DBA_REGISTRY.

So in these cases, you have a sleeping corruption which may not cause trouble. Unless you upgrade this PDB to a newer release. Then you may see it. And then it make your upgrade fail with the above error sequence.

How do you check it?

Checking whether you may be affected is simple. At first, please verify whether JVM is installed in your PDB:

select count(*) from DBA_REGISTRY where COMP_ID='JVM';

If the result is “1” you have JVM and you want to execute now the following query.

select count(*) from OBJ$ where bitand(flags,65600)=64;

If the result is non-zero, then you most likely hit unpublished BUG 28745878.

How do you fix it?

At first, you can fix this once the error has happened, and then resume the upgrade. As you may be aware already, AutoUpgrade is fully resumable. But it will be better to fix this before you invoke the upgrade.

So once the above query return non-zero results you should open your 12.2.0.1 or 18c PDB in read-write mode, apply the workaround – and then invoke the upgrade to Oracle Database 19c.

The workaround is fully described in MOS Note: 2691097.1.

You need to do the first step in CDB$ROOT, then the next steps happen in your PDB(s). Please make sure to exchange the <affected pdb name> placeholder below with the name(s) of your PDB(s) before executing the script. If you have multiple PDBs, you need to run the PDB steps in every PDB:

alter session set container=cdb$root;
alter session set "_ORACLE_SCRIPT"=true;
create or replace view rootobj sharing=object as select obj#,o.name,u.name
uname,o.type#,o.flags from obj$ o,user$ u where owner#=user#;

alter session set container=<affected pdb name>;
shutdown
startup upgrade
create or replace view rootobj sharing=object as select obj#,o.name,u.name
uname,o.type#,o.flags from obj$ o,user$ u where owner#=user#;

update obj$ set flags=flags+65536 where type# in (28,29,30,56) and
bitand(flags,65536)=0 and obj# in (select o.obj# from obj$ o,user$ u,rootobj
r where o.name=r.name and o.type#=r.type# and o.owner#=u.user# and
u.name=r.uname and bitand(r.flags,65536)!=0 union select obj# from obj$ where
bitand(flags,4259840)=4194304);

delete from sys.idl_ub1$ where obj# in (select obj# from sys.obj$ where
bitand(flags, 65536)=65536 and type# in (28,29,30,56));

commit;

shutdown
startup

Summary

The above corruption happens in the following case:

• You plugged in a non-CDB having no Long Identifiers
• The CDB you plugged into was either an Oracle 12.2.0.1 or Oracle 18c CDB with COMPATIBLE >= 12.2.0 – then Long Identifiers are present
• Your PDB has JVM in DBA_REGISTRY present

If all of the above are true, then you need to run the check query:

• select count(*) from OBJ$ where bitand(flags,65600)=64;

And if the result of it is a non-zero value, you need to apply the workaround as otherwise you will receive a core dump resulting in a failing upgrade to Oracle Database 19c.

Certainly, this issue can happen with upgrades to Oracle Database 21c and the next long term support release, too, if the above conditions apply.

Further Links and Information

• MOS Note: 2691097.1 – 12.2 or higher Database Upgrade may fail with ORA-07445: exception encountered: core dump [joevm_invokevirtual()+1414]
• Unpublished BUG 28745878 – FAILURES IN PDB DERIVED FROM NONCDB DURING OR AFTER OJVM UPGRADE OR PATCH
• Long Identifiers may cause trouble
• Video: Long Identifiers – Cool Features for DBAs

–Mike