Actually I blog about this topic for the simple reason that currently you won’t find helpful information in MyOracle Support. And I’ve seen this issue now twice in a row at Enterprise customers. You may see AutoUpgrade in trouble when you are short on RAM. During one of the restarts AutoUpgrade initiates you may get an ORA-600.
What you may see
At first, the alert.log – and this is the strange pattern here – does not give you any indication about this ORA-600. You will find it only in the logs written by autoupgrade.jar:
2020-05-29 17:15:03.254 ERROR DATABASE NAME: hugo01 CAUSE: ERROR at Line 870491 in [/autoupgrade/hugo01/100/dbupgrade/catupgrd20200529161301hugo010.log] REASON: ORA-00600: internal error code, arguments: , , , , , , , , catupgrd20200529161301hugo010.log ======================================== SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. ERROR: ORA-00600: internal error code, arguments: , , , , , , , , , , ,  IBM AIX RISC System/6000 Error: 13: Permission denied Additional information: 8 Additional information: 51315686 SQL> SQL> startup restrict pfile=/autoupgrade/hugo01/100/dbupgrade/catupgrd20200529161301hugo01_20200529170657_24577676.ora; SP2-0640: Not connected SQL> SP2-0640: Not connected SQL> SP2-0640: Not connected SQL> SQL> ========== Process Terminated by catcon ==========
But in the alert.log you will see only this:
alert.log ============ 2020-05-29T17:13:05.199299+02:00 Shutting down instance (immediate) (OS id: 657404) Shutting down instance: further logons disabled ... 2020-05-29T17:13:45.070166+02:00 Instance shutdown complete (OS id: 657404) 2020-05-29T17:14:27.749130+02:00 Starting ORACLE instance (normal) (OS id: 52626574) ... 2020-05-29T17:14:43.791515+02:00 Database mounted in Exclusive Mode Lost write protection disabled ... (PID:6619978): Using STANDBY_ARCHIVE_DEST parameter default value as /.../arch/ [krsd.c:17775] Completed: ALTER DATABASE MOUNT 2020-05-29T17:14:44.290773+02:00 ALTER DATABASE OPEN MIGRATE ... 2020-05-29T17:14:48.510007+02:00 CJQ0 started with pid=42, OS id=64947190 Completed: ALTER DATABASE OPEN MIGRATE
And no indication about the ORA-600 anywhere.
I’ve seen the exact same pattern a few weeks ago at another customer on Exadata within an OVM. And when I tried to debug this case, it looked to me as if there was a memory shortage.
In the above case, my ACS colleague Gisela confirmed that this environment is short on RAM, too.
How do you solve it?
Today Gisela reported back to me that the solution was:
- Increase the RAM for this environment
- And as the following error has happened, too:
KUP-04095: preprocessor command /opt/app/oracle/database/188.8.131.52.0_A/QOpatch/qopiprep.bat encountered error "pipe read timeout"
they set these underscores as remedy:
And please, as a general rule, don’t set underscores just blindly. I’m documenting the workaround here in case you ran into the same error and land here via a search engine search (and as reminder to myself in case I need it someday).
Actually I’m wondering sometimes. There are more and more virtual environments consolidated together. And RAM assignments get tighter and tighter. Well, you know already that Multitenant would be the correct answer here – or Cloud.
Generally, please don’t forget that autoupgrade is spawning processes in order to upgrade your database(s) quickly and unattended. It needs air (aka RAM) to breath. And in this particular case, a “resume” may not solve it unless you increase the available RAM to your virtual environment.
Further Information and Links
- AutoUpgrade Troubleshooting, Restarting and Restoring
- AutoUpgrade – Running two or more sessions in parallel
- AutoUpgrade may fail when patch ID column is NULL