Grid Infrastructure PSU Jan 2015 – Am I too intolerant?

Germans are not only known for being precise and timely – but sometimes also for being too direct. Well, Roy could tell you stories … and I always honor his politeness 🙂

Enough about stereotypes. I work with a customer at the moment on their 12c upgrades. And I did recommend the most recent PSU (Patch Set Updates) for their Grid Infrastructure environments running Oracle Restart. Same of course for the database homes but this blog post will just name some findings I’ve had the other night when trying to apply the January 2015 GI PSU to my Oracle Restart environment.

First of all, start here:

  • Drill down from MOS Note:161818.1 (Click on the Release Link for 12.1.0!) into
  • Oracle 12.1 Support Status and Alerts MOS Note:1565065.1 and further into
  • 12.1.0.2 Patch Set – Availability and Known Issues MOS Note:1683799.1 to get
    • Recommendations for patches on top of Oracle Database 12.1.0.2 – and of course for Grid Infrastructure (GI), Engineered Systems and Database In-Memory as well.

For GI you’ll get the following recommendations:

Grid Infrastructure

Document Description Rolling RAC Patch Download
Note:20132450.8 Combo of 12.1.0.2.2 OJVM PSU and 12.1.0.2.2 GI PSU (Jan 2015) Part Patch:20132450
Note:19954978.8 12.1.0.2.2 (Jan 2015) Grid Infrastructure Patch Set Update (GI PSU) Yes Patch:19954978

Well, first question:
Do you need the Combo Patch or the non-Combo? 

My personal recommendation: Take the non-Combo Patch as the combo patch includes not only the GI PSU, but also the Database PSU, the OJVM Patch and some other things you will not need for patching your Grid Infrastructure only. Of course I can see the benefit of downloading just one big piece and applying everything all together to my environment. But first of all parts of the patch (speaking of OJVM) are not rolling applicable – and second I’d like to control and script things separately. But please feel free to see and do this in a different way.

So I did download Patch:19954978 to my environment. Unzipped it.

You all know it already – you will need a new OPatch!

Of course my OPatch version is too old. You will need at least OPatch 12.1.0.1.5 – the most recent one is OPatch 12.1.0.1.6 – and you simply download it via MOS patch 6880880 and install it after removing the old directory into your $ORACLE_HOME.
.

First learning experience? OPatch doesn’t do anything without a response file.

The reason for this (at least for me as I don’t patch daily) new requirement seems to be the new opatchauto call which scripts the entire apply process in a silent way. Well, at least the patch readme tells me what to do. Please read MOS Note:966023.1 to learn about how to create this response file.

If you do not have the OCM response file (ocm.rsp), see the following My Oracle Support Document 966023.1 How To Create An OCM Response File For Opatch Silent Installation.

I did create it with OCM OFF as I don’t see a benefit in my environment for using OCM. And just on the side: I was a bit worried that this note does not contain the new opatchauto syntax but instead lists an example (which is always good and nice and helpful) from the old days:

$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u02/unconfig.rsp

Second learning experience? Analyze analyzes … not everything!

In the readme under 2.2 you’ll find also the requirement to run an anylze for conflict detection and resolution first. Maybe I’m too naive as I’m so happy with all the stuff ORAchk (credits for this gem of a tool go to Sandesh Rao’s teamI will write something about it later) can deliver – and expected too much. In my case the anylyze signaled: All good, sky is bright and nice and clear – ready to fly!

#GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/19954978 -analyze -ocmrf <ocm response file>

You’ll read below why my assumption was incorrect.

Third learning experience? An error is an error is an error!

Can you expect a patch to get applied correctly in the first run?

opatchauto apply
/oradata/grid/12.1.0/19954978 -oh
/oradata/grid/12.1.0 -ocmrf
/oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp

I did. Well … mine failed. Must be Murphy’s Law See the log below – I have marked the failing part in red.

2015-02-16_19-45-09:
Failed to run this command:
/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp

oracle.opatchauto.gi.RunExecutionSteps.runGenericShellCommands(RunExecutionSteps.java:913)
oracle.opatchauto.gi.RunExecutionSteps.processAllSteps(RunExecutionSteps.java:215)
oracle.opatchauto.gi.GIPatching.processPatchingSteps(GIPatching.java:544)
oracle.opatchauto.gi.OPatchautoExecution.main(OPatchautoExecution.java:141)
Command
"/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.

Log file Location for the failed command:
/oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-45-03PM_1.log

2015-02-16_19-45-09:
--------------
After fixing the cause of failure you have two options shown below:
Run 'opatchauto resume' or
Manually run the commands listed below
---------------------------------------------------------------------------------
/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp
(Run as oracle) - (TRIED BUT FAILED)

Now the question is:
How much space does it really require to apply this PSU? 

At this point I was wondering as the analyze passed successfully – and I couldn’t find anything in the readme about specific space requirements. My disk had roughly 8GB of free space – and as my GI Restart installation’s footprint was around 6GB I don’t had any bad thoughts.

Forth learning experience? Always see the logfile first …

Just rerunning the mentioned command gave me the correct information (just wondering why OPatch couldn’t tell me this during the failed run?). So I did execute as oracle user:

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp

Receiving finally this result:

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(12773.858MB) is not available.
UtilSession failed: 
Prerequisite check "CheckSystemSpace" failed.
Log file location:
/oradata/grid/12.1.0/cfgtoollogs/opatch/opatch2015-02-16_19-49-04PM_1.log

Ouch – 12.7GB of free space??? Really? Seriously?? Now I was scared.

The patch for Linux x86-64 has a tip size of 873 MB – but unzipped it takes 3.3 GB. So why does it require 12.7 GB of free space? Actually I don’t know the answer yet but I will follow up here as soon I know the details. Colleagues in Development ensure that especially on AIX you will need even more free space, such as in the area of 22GB!!!

One of the reasons for such a huge space requirement may be this:

Starting with 10.2, Opatch does not backup only the affected modules, it also takes a backup of the complete affected libraries to $ORACLE_HOME/.patch_storage/<patchid>/backup/<lib_directory_name>/<library_name>.

Fifth learning experience? There are workarounds …

Blogs are sometimes VERY helpful. I found two helpful entries from external bloggers (but didn’t bookmark them so I can’t credit them here – sorry).

/oradata/grid/12.1.0/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local -invPtrLoc /oradata/grid/12.1.0/oraInst.loc -oh /oradata/grid/12.1.0 -silent -ocmrf /oradata/grid/12.1.0/OPatch/ocm/bin/emocmrsp OPatch.SKIP_VERIFY_SPACE=true

Of course you can’t give this variable at the end to opatchauto as it wouldn’t understand it.

Final learning lesson? Clean up is a great idea!

This is not as simple as I did expect it. Of course, it’s not. But please see this MOS Note:550522.1 – How To Avoid Disk Full Issues Because OPatch Backups Take Big Amount Of Disk Space

Famous Last Words?

Patches are great. In fact YOU NEED TO PATCH. Take the PSUs. There’s no way out as it will help you to avoid plenty of known issues. But I hope that my above learning experience may help you to sail around one or the other pitfall 🙂

–Mike

6 thoughts on “Grid Infrastructure PSU Jan 2015 – Am I too intolerant?

  1. Why Oracle do not provide a tool to handle all of this?
    Is to hard to mimic yum, apt-get or even Windows Update?
    Red Hat, IBM, Microsoft, all others have tool to handle it, Oracle is the only one "special".

    It is very bad to have to dig into patch readme, note id and etc.

  2. Well, I see all your points. They are all quite obvious and understandable.

    But as you may already assume or know, I’m not in the position to change this. Actually it’s only customers who’ll be able to deliver feedback about the process. Please open SRs and ask Support to open Bugs if you are not satisfied with the PSU process, it’s documentation and such.

    Thanks a lot!
    Mike

  3. Thanks Mike.
    We’re about to apply latest PSU to our 12.1.0.2 rac database (gi + db home) and even before starting this seems like a nightmare.
    The note id’s + readme’s are not helpful at all.
    Whatever happend to good old opatch apply local?

  4. Hi,

    you may "hate" me for saying this – but please log SRs and force Support to file bugs. I have filed 21 review issues for the April PSU readme – and I hear complaints all over the place. But nothing seems to go into the direction "Support". Not sure why this is – and I fully understand your frustration, I even hear it from our own field experts.

    The only way I can think off is to log SRs. Many SRs. And insist to get bug numbers filed for issues you see, regardless if it’s a doc only thing or a misbehavior of opatch.

    Thanks
    Mike

  5. I think you’Ve got me wrong.

    People listen to me (at least some) but YOU are the customers and YOU have the power. If customers don’t open SRs for such issues it simply is way less powerful than myself and a few others moaning and logging issues.

    Honestly, I know that opening SRs costs time – but it costs less time than juggling around with issues during patching.

    Cheers
    Mike

Leave a Reply

Your email address will not be published. Required fields are marked *