A customer from the US did contact me past week via LinkedIn and raised a question:
“Is it expected that my patch set upgrade from Oracle 18.104.22.168 to Oracle 22.214.171.124 takes over 3 hours?”
Of course, no – this is not expected.
This is the upgrade stats gathered post upgrade with utlu112s.sql:
SQL> @?/rdbms/admin/utlu112s.sql ; .
Oracle Database 11.2 Post-Upgrade Status Tool
Component Current Version Elapsed Time
Name Status Number HH:MM:SS
. VALID 126.96.36.199.0 02:46:19
JServer JAVA Virtual Machine
. VALID 188.8.131.52.0 00:08:34
Total Upgrade Time: 03:06:47
No, this is not really expected. So we tried to nail down the root cause finding out these statements in the upgrade script c1102000.sql are causing the trouble:
194 — wri$_optstat_histhead_history2.
195 execute immediate
196 q’#create unique index i_wri$_optstat_hh_obj_icol_st on
197 wri$_optstat_histhead_history (obj#, intcol#, savtime, colname)
198 tablespace sysaux #’;
200 execute immediate
201 q’#create index i_wri$_optstat_hh_st on
202 wri$_optstat_histhead_history (savtime)
203 tablespace sysaux #’;
It’s index rebuilds on histogram tables. And the customer has a large amount of stats data in his database as the default stats retention is 31 days.
Obviously the index rebuild is not done very efficiently (not done in parallel, no nologging clause). Those things can happen and sometimes this may not cause any issues. But in this case it lead to over 2 hours for just those index rebuilds.
Luckily my colleague Cindy is an excellent resource for such things – after asking our team I’ve got the reply that this is tagged with a bug number and code fix already got checked in (under review right now):
Upgrade from 184.108.40.206 to 220.127.116.11 is slow
PS: Credits go to Tan for bringing this to my attention – and sorry for the inconvenience!
How were you able to detect the slow sql?
SET TIMING ON
SET ECHO ON
when you run the upgrade – not the default in 11.2 but is the default beginning with 12.1’s catctl.pl
Thanks Mike, I wish I had checked the blog a few moments earlier as already started the upgrade. I wasn’t however sure if my question got updated as I couldn’t see it immediately and I tried to update the blog a couple of time.
Mike, In following up on my previous question, I can’t find catctl.pl in the 11204 binaries, this maybe only a 12c thing or platform specific.
catctl.pl is a pure 12c tool – we had just given it to a number of selected high profile customers for 18.104.22.168 evalutation – but it is not available for 11g databases to anybody anymore.
OK, so what’s the point of the post if it’s not available to the majority of the customers.
Not sure what you mean with your post???
The post or article is for 11204 however the script catpl used for determining the perf issue doesn’t exist on 11204 and
most customers will not have access to it, so it kind of defeats the purpose of the article/post. Hope the explanation is clearer.
Sorry … my fault – now I got your point and you are correct.
Please exchange "catctl.pl" with "catupgrd.sql". For catctl.pl the ECHO ON won’t help you as it is a SQL Plus setting but catctl.pl runs outside of sql*plus.
So appologize 🙂
I’ve meant catupgrd.sql