OJVM datapatch fails with ORA-29532 – but the root cause is noncdb_to_pdb.sql

Kudos to Robert Ortel who brought this nice misbehavior to my attention. And even though it looks like this would be an OJVM issue, it is caused by noncdb_to_pdb,sql, the script which is used to convert a non-CDB to a PDB. When you apply an OJVM patch, OJVM datapatch fails with ORA-29532 – but the root cause is noncdb_to_pdb.sql.

Photo by Alexandra Gorn on Unsplash

It’s a bit tricky

First things first. This is not a blog post to blame OJVM. The problem just happens because datapatch for an OJVM patch touches data in the dictionary which hasn’t been adjusted correctly by noncdb_to_pdb.sqlduring plugin of a non-CDB. Not the first issue with noncdb_to_pdb.sql I come across.

Second, let me give a brief summary what happened for Robert as this can happen to you as well.

Robert operates a CDB with more than 40 PDBs, and attempted to upgrade from 18.9.0 to 19.6.0. In addition he also applied the OJVM patch to his 19.6.0 home:

• Patch 30484981 : applied on Wed Feb 05 16:57:35 CET 2020
  Unique Patch ID: 23248235
  Patch description: "OJVM RELEASE UPDATE: 19.6.0.0.200114 (30484981)"

And while the CDB$ROOT upgrade went fine, some PDB upgrades failed with ORA-07445 [joevm_invokevirtual()+1425].

Is this a bug?

Of course this is a bug. And a terrible one to be honest. noncdb_to_pdb.sql causes a corruption.

• BUG 28745878 – POST-INSTALLATION FOR 12.2.0.1.180717 OJVM PSU ON 12.2 PDB FAILING WITH ORA-29532 ERROR

The bug is non-public. Hence, I didn’t add a link. And there is only one single one-off patch available – for Oracle 12.2.0.1. It contains a fixed noncdb_to_pdb.sql.

• POST-INSTALLATION FOR 12.2.0.1.180717 OJVM PSU ON 12.2 PDB FAILING WITH ORA-29532 ERROR

And the fix went into 19.2.0 as well. It should be available in all subsequent RUs. But no other RUs or patch bundles contain this a fixed version of the conversion script.

Everything ok? Of course not. The fix will fix the root cause only when you plugin a non-CDB into 19c. But in many cases, you plugged in a non-CDB into 12.1.0.2, 12.2.0.1 or 18c with various patch levels. Such as Robert did here. His non-CDB databases were 12.1.0.2 non-CDBs. And he plugged them in either in 12.2.0.1 or 18c. And there the problem still exists. For 12.2.0.1 – BEFORE plugging in – you can use the one-off patch I linked above, and exchange noncdb_to_pdb.sql with a working version.

If you already plugged in, the bug fix won’t help you anymore. This was an issue, Robert’s SR got delayed with as the support engineer waited for a backport. But this backport wouldn’t have solved anything.

The origin of this issue is coming from a 12.2 feature, the Long Identifiers. noncdb_to_pdb.sql does an incorrect translation for Java classes. A metadata link does not get set. And this gets revealed only later when an upgrade happens.

When does it happen?

This issue happens only when:

• You have JAVAVM (aka OJVM) in your databases
• You plugged into 12.2.0.1 or 18c
• COMPATIBLE is set to 12.2.0 or higher

It does not happen if you plugged into Oracle 12.1.0.2 or Oracle 19c. And as far as my understanding goes, it does not happen if COMPATIBLE was set to 12.1.0 while you plugged in. If you don’t have JAVAVM, you won’t see this issue. And hence, no worry as well if you plan to go to Multitenant in Oracle 19c for the first time. The issue is fixed in 19c.

But what if it happens to you? Then you need to know how to fix it.

How do you fix this?

You can find the manual fix in:

• MOS Note:2488247.1 – OJVM Datapatch Fails With ORA-29532 java.lang.BootstrapMethodError Error in Multitenant\PDB Environment

Be aware that your will patch your data dictionary to fix a corruption:

alter session set container=cdb$root;

alter session set "_ORACLE_SCRIPT"=true;

create or replace view rootobj sharing=object as select obj#,o.name,u.name 
uname,o.type#,o.flags from obj$ o,user$ u where owner#=user#; 

alter session set container=<AFFECTED_PDB_NAME>;

create or replace view rootobj sharing=object as select obj#,o.name,u.name 
uname,o.type#,o.flags from obj$ o,user$ u where owner#=user#; 

update obj$ set flags=flags+65536 where obj# in (select o.obj# from obj$ 
o,user$ u,rootobj r where o.name=r.name and o.type#=r.type# and 
o.owner#=u.user# and u.name=r.uname and o.type# in (28,29,30,56) and 
bitand(r.flags,65536)!=0 and bitand(o.flags,65536)=0); 

delete from sys.idl_ub1$ where obj# in (select obj# from sys.obj$ where 
bitand(flags, 65536)=65536 and type# in (28,29,30,56)); 

commit;

Again, be aware that you are patching the dictionary here manually. You must ensure that you have a valid backup which can be restored and recovered. Test this beforehand.

Once you patched your PDBs with the above commands, you can repeat now your upgrades (dbupgrade -R or autoupgrade.jar are both resumable, DBUA isn’t!).

More Information and Links

• noncdb_to_pdb.sql – why does it take so long?
• noncdb_to_pdb.sql – Concurrent Stats Gathering – bug 26434999
• Database Migration from non-CDB to PDB – Typical Plugin Issues and Workarounds
• MOS Note:2488247.1 – OJVM Datapatch Fails With ORA-29532 java.lang.BootstrapMethodError Error in Multitenant\PDB Environment

–Mike