Grid Infrastructure PSU Jan 2015 – Am I too intolerant?

Germans are not only known for being precise and timely – but sometimes also for being too direct. Well, Roy could tell you stories … and I always honor his politeness 🙂

Enough about stereotypes. I work with a customer at the moment on their 12c upgrades. And I did recommend the most recent PSU (Patch Set Updates) for their Grid Infrastructure environments running Oracle Restart. Same of course for the database homes but this blog post will just name some findings I’ve had the other night when trying to apply the January 2015 GI PSU to my Oracle Restart environment.

First of all, start here:

• Drill down from MOS Note:161818.1 (Click on the Release Link for 12.1.0!) into
• Oracle 12.1 Support Status and Alerts MOS Note:1565065.1 and further into
• 12.1.0.2 Patch Set – Availability and Known Issues MOS Note:1683799.1 to get
  • Recommendations for patches on top of Oracle Database 12.1.0.2 – and of course for Grid Infrastructure (GI), Engineered Systems and Database In-Memory as well.

For GI you’ll get the following recommendations:

Grid Infrastructure

Document	Description	Rolling RAC	Patch Download
Note:20132450.8	Combo of 12.1.0.2.2 OJVM PSU and 12.1.0.2.2 GI PSU (Jan 2015)	Part	Patch:20132450
Note:19954978.8	12.1.0.2.2 (Jan 2015) Grid Infrastructure Patch Set Update (GI PSU)	Yes	Patch:19954978

Well, first question:
Do you need the Combo Patch or the non-Combo? 

My personal recommendation: Take the non-Combo Patch as the combo patch includes not only the GI PSU, but also the Database PSU, the OJVM Patch and some other things you will not need for patching your Grid Infrastructure only. Of course I can see the benefit of downloading just one big piece and applying everything all together to my environment. But first of all parts of the patch (speaking of OJVM) are not rolling applicable – and second I’d like to control and script things separately. But please feel free to see and do this in a different way.

So I did download Patch:19954978 to my environment. Unzipped it.

You all know it already – you will need a new OPatch!

Of course my OPatch version is too old. You will need at least OPatch 12.1.0.1.5 – the most recent one is OPatch 12.1.0.1.6 – and you simply download it via MOS patch 6880880 and install it after removing the old directory into your $ORACLE_HOME.
.

First learning experience? OPatch doesn’t do anything without a response file.

The reason for this (at least for me as I don’t patch daily) new requirement seems to be the new opatchauto call which scripts the entire apply process in a silent way. Well, at least the patch readme tells me what to do. Please read MOS Note:966023.1 to learn about how to create this response file.

If you do not have the OCM response file (ocm.rsp), see the following My Oracle Support Document 966023.1 How To Create An OCM Response File For Opatch Silent Installation.

I did create it with OCM OFF as I don’t see a benefit in my environment for using OCM. And just on the side: I was a bit worried that this note does not contain the new opatchauto syntax but instead lists an example (which is always good and nice and helpful) from the old days:

$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u02/unconfig.rsp

Second learning experience? Analyze analyzes … not everything!

In the readme under 2.2 you’ll find also the requirement to run an anylze for conflict detection and resolution first. Maybe I’m too naive as I’m so happy with all the stuff ORAchk (credits for this gem of a tool go to Sandesh Rao’s team – I will write something about it later) can deliver – and expected too much. In my case the anylyze signaled: All good, sky is bright and nice and clear – ready to fly!

#GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/19954978 -analyze -ocmrf <ocm response file>

You’ll read below why my assumption was incorrect.

Third learning experience? An error is an error is an error!

Can you expect a patch to get applied correctly in the first run?

opatchauto apply
/oradata/grid/12.1.0/19954978 -oh
/oradata/grid/12.1.0 -ocmrf
/oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp

I did. Well … mine failed. Must be Murphy’s Law See the log below – I have marked the failing part in red.

2015-02-16_19-45-09:
Failed to run this command:
/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp

oracle.opatchauto.gi.RunExecutionSteps.runGenericShellCommands(RunExecutionSteps.java:913)
oracle.opatchauto.gi.RunExecutionSteps.processAllSteps(RunExecutionSteps.java:215)
oracle.opatchauto.gi.GIPatching.processPatchingSteps(GIPatching.java:544)
oracle.opatchauto.gi.OPatchautoExecution.main(OPatchautoExecution.java:141)
Command
"/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.

Log file Location for the failed command:
/oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-45-03PM_1.log

2015-02-16_19-45-09:
--------------
After fixing the cause of failure you have two options shown below:
Run 'opatchauto resume' or
Manually run the commands listed below
---------------------------------------------------------------------------------
/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp
(Run as oracle) - (TRIED BUT FAILED)

Now the question is:
How much space does it really require to apply this PSU? 

At this point I was wondering as the analyze passed successfully – and I couldn’t find anything in the readme about specific space requirements. My disk had roughly 8GB of free space – and as my GI Restart installation’s footprint was around 6GB I don’t had any bad thoughts.

Forth learning experience? Always see the logfile first …

Just rerunning the mentioned command gave me the correct information (just wondering why OPatch couldn’t tell me this during the failed run?). So I did execute as oracle user:

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp

Receiving finally this result:

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(12773.858MB) is not available.
UtilSession failed: 
Prerequisite check "CheckSystemSpace" failed.
Log file location:
/oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-49-04PM_1.log

Ouch – 12.7GB of free space??? Really? Seriously?? Now I was scared.

The patch for Linux x86-64 has a tip size of 873 MB – but unzipped it takes 3.3 GB. So why does it require 12.7 GB of free space? Actually I don’t know the answer yet but I will follow up here as soon I know the details. Colleagues in Development ensure that especially on AIX you will need even more free space, such as in the area of 22GB!!!

One of the reasons for such a huge space requirement may be this:

Starting with 10.2, Opatch does not backup only the affected modules, it also takes a backup of the complete affected libraries to $ORACLE_HOME/.patch_storage/<patchid>/backup/<lib_directory_name>/<library_name>.

Fifth learning experience? There are workarounds …

Blogs are sometimes VERY helpful. I found two helpful entries from external bloggers (but didn’t bookmark them so I can’t credit them here – sorry).

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp OPatch.SKIP_VERIFY_SPACE=true

Of course you can’t give this variable at the end to opatchauto as it wouldn’t understand it.

Final learning lesson? Clean up is a great idea!

This is not as simple as I did expect it. Of course, it’s not. But please see this MOS Note:550522.1 – How To Avoid Disk Full Issues Because OPatch Backups Take Big Amount Of Disk Space

Famous Last Words?

Patches are great. In fact YOU NEED TO PATCH. Take the PSUs. There’s no way out as it will help you to avoid plenty of known issues. But I hope that my above learning experience may help you to sail around one or the other pitfall 🙂

–Mike