New version of our Hands-On Lab available for download “Upgrade/Migrate/Consolidate to Oracle 12c” HOL

guest says:

Hi Folks, Facing big issue while upgrading database using catctl.pl. My upgrade is failing at phase 65 with below error.
ORACLE_BASE—> /var/tmp/oracle/upgrade/LNTOREDBA1
.
Argument list for [/ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl]
SQL Process Count n = 8
SQL PDB Process Count N = 0
Input Directory d = /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin
Phase Logging Table t = 0
Log Dir l = /var/tmp/oracle/upgrade
Script s = 0
Serial Run S = 0
Upgrade Mode active M = 0
Start Phase p = 0
End Phase P = 0
Log Id i = 0
Run in c = 0
Do not run in C = 0
Echo OFF e = 1
No Post Upgrade x = 0
Reverse Order r = 0
Open Mode Normal o = 0
Debug catcon.pm z = 0
Debug catctl.pl Z = 0
Display Phases y = 0
Child Process I = 0

catctl.pl version: 12.1.0.2.0
Oracle Base = /var/tmp/oracle/upgrade/LNTOREDBA1
catcon: ALL catcon-related output will be written to /var/tmp/oracle/upgrade/catupgrd_catcon_28703.lst
catcon: See /var/tmp/oracle/upgrade/catupgrd*.log files for output generated by scripts
catcon: See /var/tmp/oracle/upgrade/catupgrd_*.lst files for spool files, if any
…….Number of Cpus = 8
SQL Process Count = 8
..
——————————————————
Phases [0-73] Start Time:[2016_01_07 08:20:31]
——————————————————
Serial Phase #: 0 Files: 1 ……………….. Time: 193s
Serial Phase #: 1 Files: 5 ……………… Time: 186s
Restart Phase #: 2 Files: 1 Time: 0s
Parallel Phase #: 3 Files: 18 ….. Time: 45s
Restart Phase #: 4 Files: 1 Time: 0s
Serial Phase #: 5 Files: 5 …….. Time: 85s
Serial Phase #: 6 Files: 1 …. Time: 39s
Serial Phase #: 7 Files: 4 … Time: 24s
Restart Phase #: 8 Files: 1 Time: 0s
Parallel Phase #: 9 Files: 62 ………… Time: 123s
Restart Phase #:10 Files: 1 Time: 1s
Serial Phase #:11 Files: 1 ….. Time: 48s
Restart Phase #:12 Files: 1 Time: 0s
Parallel Phase #:13 Files: 91 . Time: 11s
Restart Phase #:14 Files: 1 Time: 0s
Parallel Phase #:15 Files: 111 .. Time: 25s
Restart Phase #:16 Files: 1 Time: 0s
Serial Phase #:17 Files: 3 . Time: 3s
Restart Phase #:18 Files: 1 Time: 0s
Parallel Phase #:19 Files: 32 ….. Time: 51s
Restart Phase #:20 Files: 1 Time: 0s
Serial Phase #:21 Files: 3 .. Time: 20s
Restart Phase #:22 Files: 1 Time: 0s
Parallel Phase #:23 Files: 23 ……………….. Time: 201s
Restart Phase #:24 Files: 1 Time: 0s
Parallel Phase #:25 Files: 11 …….. Time: 82s
Restart Phase #:26 Files: 1 Time: 1s
Serial Phase #:27 Files: 1 Time: 1s
Restart Phase #:28 Files: 1 Time: 0s
Serial Phase #:30 Files: 1 Time: 0s
Serial Phase #:31 Files: 257 …… Time: 55s
Serial Phase #:32 Files: 1 Time: 0s
Restart Phase #:33 Files: 1 Time: 1s
Serial Phase #:34 Files: 1 . Time: 9s
Restart Phase #:35 Files: 1 Time: 0s
Restart Phase #:36 Files: 1 Time: 1s
Serial Phase #:37 Files: 4 …….. Time: 79s
Restart Phase #:38 Files: 1 Time: 1s
Parallel Phase #:39 Files: 13 …………. Time: 134s
Restart Phase #:40 Files: 1 Time: 1s
Parallel Phase #:41 Files: 10 . Time: 9s
Restart Phase #:42 Files: 1 Time: 1s
Serial Phase #:43 Files: 1 . Time: 12s
Restart Phase #:44 Files: 1 Time: 0s
Serial Phase #:45 Files: 1 . Time: 9s
Serial Phase #:46 Files: 1 Time: 0s
Restart Phase #:47 Files: 1 Time: 1s
Serial Phase #:48 Files: 1 Time: 0s
Restart Phase #:49 Files: 1 Time: 1s
Serial Phase #:50 Files: 1 . Time: 0s
Restart Phase #:51 Files: 1 Time: 1s
Serial Phase #:52 Files: 1 Time: 0s
Restart Phase #:53 Files: 1 Time: 1s
Serial Phase #:54 Files: 1 ……… Time: 96s
Restart Phase #:55 Files: 1 Time: 0s
Serial Phase #:56 Files: 1 Time: 1s
Restart Phase #:57 Files: 1 Time: 0s
Serial Phase #:58 Files: 1 Time: 0s
Restart Phase #:59 Files: 1 . Time: 1s
Serial Phase #:60 Files: 1 Time: 1s
Restart Phase #:61 Files: 1 Time: 0s
Serial Phase #:62 Files: 1 Time: 1s
Restart Phase #:63 Files: 1 Time: 1s
Serial Phase #:64 Files: 1 Time: 2s
Serial Phase #:65 Files: 1 Calling sqlpatch with LD_LIBRARY_PATH=/ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/lib; export LD_LIBRARY_PATH; LIBPATH=/ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/lib; export LIBPATH; LD_LIBRARY_PATH_64=/ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/lib; export LD_LIBRARY_PATH_64; DYLD_LIBRARY_PATH=/ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/lib; export DYLD_LIBRARY_PATH; /ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/perl/bin/perl -I /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin -I /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/../../sqlpatch /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/../../sqlpatch/sqlpatch.pl -verbose -upgrade_mode_only > /var/tmp/oracle/upgrade/catupgrd_datapatch_upgrade.log 2> /var/tmp/oracle/upgrade/catupgrd_datapatch_upgrade.err
returned from sqlpatch
LOG FILES: (catupgrd*.log) (catupgrd_trace.log)
==============================================
======================
SQL Patching tool version 12.1.0.2.0 on Thu Jan 7 08:46:29 2016
Copyright (c) 2015, Oracle. All rights reserved.

Log file for this invocation: /var/tmp/oracle/upgrade/LNTOREDBA1/cfgtoollogs/sqlpatch/sqlpatch_21506_2016_01_07_08_46_29/sqlpatch_invocation.log

SQL Patching arguments:
verbose: 1
force: 0
prereq: 0
upgrade_mode_only: 1
oh:
bundle_series:
ignorable_errors:
bootstrap:
pdbs:

Connecting to database…OK
catcon: ALL catcon-related output will be written to /var/tmp/oracle/upgrade/LNTOREDBA1/cfgtoollogs/sqlpatch/sqlpatch_21506_2016_01_07_08_46_29/sqlpatch_catcon__catcon_21506.lst
catcon: See /var/tmp/oracle/upgrade/LNTOREDBA1/cfgtoollogs/sqlpatch/sqlpatch_21506_2016_01_07_08_46_29/sqlpatch_catcon_*.log files for output generated by scripts
catcon: See /var/tmp/oracle/upgrade/LNTOREDBA1/cfgtoollogs/sqlpatch/sqlpatch_21506_2016_01_07_08_46_29/sqlpatch_catcon__*.lst files for spool files, if any

Bootstrapping registry and package to current versions…done

verify_queryable_inventory returned ORA-20001: Latest xml inventory is not loaded into table

Queryable inventory could not determine the current opatch status.
Execute ‘select dbms_sqlpatch.verify_queryable_inventory from dual’ and/or check the invocation log
/var/tmp/oracle/upgrade/LNTOREDBA1/cfgtoollogs/sqlpatch/sqlpatch_21506_2016_01_07_08_46_29/sqlpatch_invocation.log
for the complete error.
Prereq check failed, exiting without installing any patches.

Please refer to MOS Note 1609718.1 and/or the invocation log
/var/tmp/oracle/upgrade/LNTOREDBA1/cfgtoollogs/sqlpatch/sqlpatch_21506_2016_01_07_08_46_29/sqlpatch_invocation.log
for information on how to resolve the above errors.

SQL Patching tool complete on Thu Jan 7 08:46:32 2016

Any help would be useful. Mike if you can help as I spoke with you during Oracle Open World 2015 in SFO.

ram sharma says:

Hi Mike,

I have sent you an email with log files attached created by phase 65 if they can help you. The upgrade process fails at phase 65 and does not proceed to next phase.

The below error also I found from upgrade log.

Restart Phase #:63 Files: 1 Time: 1s
Serial Phase #:64 Files: 1 Time: 2s
Serial Phase #:65 Files: 1 Calling sqlpatch with LD_LIBRARY_PATH=/ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/lib; export LD_LIBRARY_PATH; LIBPATH=/ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/lib; export LIBPATH; LD_LIBRARY_PATH_64=/ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/lib; export LD_LIBRARY_PATH_64; DYLD_LIBRARY_PATH=/ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/lib; export DYLD_LIBRARY_PATH; /ms/dist/orcl/PROJ/product/12.1.0.2.3/.exec/@sys/perl/bin/perl -I /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin -I /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/../../sqlpatch /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/../../sqlpatch/sqlpatch.pl -verbose -upgrade_mode_only > /var/tmp/oracle/upgrade/catupgrd_datapatch_upgrade.log 2> /var/tmp/oracle/upgrade/catupgrd_datapatch_upgrade.err
returned from sqlpatch

Unexpected error encountered in catctlMain; Error Stack Below; exiting

Unexpected error encountered in catctlCreateSqlFile; exiting Failed to open – /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catupgrd80.595592058337392.sql 30 at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 4515.
at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 4515
main::catctlCreateSqlFile(‘@/ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/utluc…’, 0) called at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 4704
main::catctlCreatePFile() called at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 2526
main::catctlExecutePhaseFiles(65, 1, undef, undef, undef) called at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 1371
main::catctlRunPhase(65, 1, undef, undef, undef) called at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 1487
main::catctlRunPhases(0, 74, undef, undef, undef) called at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 1574
main::catctlRunMainPhases() called at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 930
main::catctlMain() called at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 854
eval {…} called at /ms/dist/orcl/PROJ/product/12.1.0.2.3/exec/rdbms/admin/catctl.pl line 852 LOG FILES: (catupgrd*.log) (catupgrd_trace.log)

Regards,
Ram Sharma

Mike says:

Ram,

as we exchanged emails already my recommendation was:

Run catctl.pl with the "-x" option which will skip the execution of datapatch during upgrade. I’d suppose that datapatch can’t deal with the read-only file system you are using. Just a guess.

After the upgrade run datapatch -verbose to make sure the required changes will be applied.

Cheers
Mike