SuperCluster opatch and shared client library errors following disk space problems during opatch

Asset ID:	1-72-2013546.1
Update Date:	2015-05-23
Keywords:

Solution Type Problem Resolution Sure

Solution 2013546.1 : SuperCluster opatch and shared client library errors following disk space problems during opatch

Applies to:

SPARC SuperCluster T4-4 Full Rack - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M6-32 Hardware - Version All Versions to All Versions [Release All Releases]
SPARC SuperCluster T4-4 - Version All Versions to All Versions [Release All Releases]
SPARC SuperCluster T4-4 Half Rack - Version All Versions to All Versions [Release All Releases]
Oracle Solaris on SPARC (64-bit)

Symptoms

Two separate stages of errors are possible after these types of failures

Please not these steps should be fine for any 11.2.0.3 or 11.2.0.4 GI. However a similar problem with 12.1.0.x should be re-evalauated as it handles inventory between the nodes differently in RAC than 11g.

1) Varoius opatch commands could fail with messages similar to the following

The precise messaging will vary with the offending patch number and Grid Home path

Unable to create patchObject
Possible causes are:
ORACLE_HOME/inventory/oneoffs/18193172 is corrupted
No read permission to above directory
Please remove one-off entry [ 18193172 ] from /u01/app/11.2.0.3/grid/inventory/ContentsXML/comps.xml and retry if above directory corrupted, or Please check the contents of the directory ORACLE_HOME/inventory/oneoffs/18193172

2) crsctl commands and rootcrs.pl commands can fail with messages similar to the following

ld.so.1: crsctl.bin: fatal: libclntsh.so.11.1: open failed: No such file or directory

Changes

opatch application failed at the LDom or local zone level due to running out of disk space .

This could occur during opatch auto or opatch napply. The directory you go to to recover will vary depending on command run and tme of problem.

Cause

Opatch failed while modifying binaries and client shared libraries, due to space issues, It had made home edits but had not finalized inventory file edits nor home relinks.

Solution

1) Correct the inventory file discrepencies

Obtain a listing of the oneoffs left that the home knows about , if there are none left you will have to remove reference to every one off in the $ORACLE_HOME/inventory/ContentsXML/comps.xml file.

#ls -al $ORACLE_HOME/inventory/oneoffs

As the grid home owner edit the $ORACLE_HOME/inventory/ContentsXML/comps.xml to remove the <ONEOFF ... </ONEOFF> entries between <ONEOFF_LIST> and </ONEOFF_LIST> that do not exist in the directory . If there are no folders left in the directory then remove everything from <ONEOFF_LIST> to </ONEOFF_LIST> inclusive.

Once this is done verify that you can do an inventory listing as the Gird home owner.

$ORACLE_HOME/OPatch/opatch lsinventory

2) Restore the pre patch binaries, from the backup directory fo the failed run, and libraries and then regenerate the client shared libraries.

Remember the path may be different depending on your installation approach and time of opatch failure

As grid home owner

$ORACLE_HOME/.patch_storage/NApply/2015-05-23_08-42-08AM/restore.sh
This script is going to restore the Oracle Home to the previous state.
It does not perform any of the following:
- Running init/pre/post scripts
- Oracle binary re-link
- Customized steps performed manually by user
Please use this script with supervision from Oracle Technical Support.
About to modify Oracle Home( /u01/app/11.2.0.3/grid )
Do you want to proceed? [Y/N]
y
User responded with : Y
Restore script completed.

Then as root

#cd $ORACLE_HOME/bin

#./genclntsh

#$ORACLE_HOME/bin/crsctl stop crs -f

#$ORACLE_HOME/crs/install/rootcrs.pl -unlock

3) proceed to patch the failed node or nodes that were manually corrected with opatch napply commands using the -local flag. Using -local will help you prevent other opatch errors just incase your different nodes failed at different stages of patching.

Community Discussions

Still have questions? Use the communities window below to search for similar discussions or start a new discussion on this subject. (Window is the live community not a screenshot)

Click here to open in main browser window

References

<NOTE:1567979.1> - Oracle SuperCluster Supported Software Versions - All Hardware Types
<NOTE:1452277.1> - SuperCluster Critical Issues

Attachments

This solution has no attachment