AutoUpgrade in trouble when you are short on RAM

Actually I blog about this topic for the simple reason that currently you won’t find helpful information in MyOracle Support. And I’ve seen this issue now twice in a row at Enterprise customers. You may see AutoUpgrade in trouble when you are short on RAM. During one of the restarts AutoUpgrade initiates you may get an ORA-600.

Photo by Vadim Sadovski on Unsplash

What you may see

At first, the alert.log – and this is the strange pattern here – does not give you any indication about this ORA-600. You will find it only in the logs written by autoupgrade.jar:

2020-05-29 17:15:03.254 ERROR
DATABASE NAME: hugo01
         CAUSE: ERROR at Line 870491 in [/autoupgrade/hugo01/100/dbupgrade/catupgrd20200529161301hugo010.log]
        REASON: ORA-00600: internal error code, arguments: [907], [], [], [], [], [], [], [],

catupgrd20200529161301hugo010.log
========================================
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
ERROR:
ORA-00600: internal error code, arguments: [907], [], [], [], [], [], [], [], [], [], [], []
IBM AIX RISC System/6000 Error: 13: Permission denied
Additional information: 8
Additional information: 51315686
SQL>
SQL> startup restrict pfile=/autoupgrade/hugo01/100/dbupgrade/catupgrd20200529161301hugo01_20200529170657_24577676.ora;
SP2-0640: Not connected
SQL> SP2-0640: Not connected
SQL> SP2-0640: Not connected
SQL> 
SQL> ========== Process Terminated by catcon ==========

But in the alert.log you will see only this:

alert.log
============
2020-05-29T17:13:05.199299+02:00
Shutting down instance (immediate) (OS id: 657404)
Shutting down instance: further logons disabled
...
2020-05-29T17:13:45.070166+02:00
Instance shutdown complete (OS id: 657404)
2020-05-29T17:14:27.749130+02:00
Starting ORACLE instance (normal) (OS id: 52626574)
...
2020-05-29T17:14:43.791515+02:00
Database mounted in Exclusive Mode
Lost write protection disabled
...
(PID:6619978): Using STANDBY_ARCHIVE_DEST parameter default value as /.../arch/ [krsd.c:17775]
Completed: ALTER DATABASE   MOUNT
2020-05-29T17:14:44.290773+02:00
ALTER DATABASE OPEN MIGRATE
...
2020-05-29T17:14:48.510007+02:00
CJQ0 started with pid=42, OS id=64947190
Completed: ALTER DATABASE OPEN MIGRATE

And no indication about the ORA-600 anywhere.

I’ve seen the exact same pattern a few weeks ago at another customer on Exadata within an OVM. And when I tried to debug this case, it looked to me as if there was a memory shortage.

In the above case, my ACS colleague Gisela confirmed that this environment is short on RAM, too.

How do you solve it?

Today Gisela reported back to me that the solution was:

1. Increase the RAM for this environment
2. And as the following error has happened, too:

   KUP-04095: preprocessor command /opt/app/oracle/database/18.9.0.1.0_A/QOpatch/qopiprep.bat encountered error "pipe read timeout"

   they set these underscores as remedy:

   "_xt_preproc_timeout"=1000;
   "_enable_ptime_update_for_sys"=TRUE;

   And please, as a general rule, don’t set underscores just blindly. I’m documenting the workaround here in case you ran into the same error and land here via a search engine search (and as reminder to myself in case I need it someday).

Summary

Actually I’m wondering sometimes. There are more and more virtual environments consolidated together. And RAM assignments get tighter and tighter. Well, you know already that Multitenant would be the correct answer here – or Cloud.

Generally, please don’t forget that autoupgrade is spawning processes in order to upgrade your database(s) quickly and unattended. It needs air (aka RAM) to breath. And in this particular case, a “resume” may not solve it unless you increase the available RAM to your virtual environment.

Further Information and Links

• AutoUpgrade Troubleshooting, Restarting and Restoring
• AutoUpgrade – Running two or more sessions in parallel
• AutoUpgrade may fail when patch ID column is NULL

–Mike