SuperCluster - How to recover from a failed 11.2.0.3 to 11.2.0.4 upgrade due to HAIP

Asset ID:	1-72-1996379.1
Update Date:	2015-04-22
Keywords:

Solution Type Problem Resolution Sure

Solution 1996379.1 : SuperCluster - How to recover from a failed 11.2.0.3 to 11.2.0.4 upgrade due to HAIP

Applies to:

Oracle SuperCluster T5-8 Half Rack - Version All Versions and later
SPARC SuperCluster T4-4 Half Rack - Version All Versions and later
SPARC SuperCluster T4-4 Full Rack - Version All Versions and later
Oracle SuperCluster T5-8 Full Rack - Version All Versions and later
Oracle SuperCluster M6-32 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

RAC alert log for the host shows something similar to this

[client(14264)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.4/grid/log/hostname/client/ocrcheck_14264.log.
2015-04-05 00:33:17.971:
[client(18393)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.4/grid/log/hostname/client/crsctl_grid.log.
2015-04-05 00:33:29.135:
[client(19012)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.4/grid/log/hostname/client/crsctl_grid.log.
2015-04-05 00:33:39.677:
[client(19799)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.4/grid/log/hostname/client/crsctl_grid.log.
2015-04-05 00:33:50.227:
[client(20622)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.4/grid/log/hostname/client/crsctl_grid.log.
2015-04-05 00:34:07.815:
[client(21927)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.4/grid/log/hostname/client/crsctl_grid.log.
2015-04-05 00:34:21.661:
[client(23310)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.4/grid/log/hostname/client/crsctl_grid.log.
2015-04-05 00:35:28.999:
[ctssd(1312)]CRS-2409:The clock on host hostname is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2015-04-05 00:35:42.338:
[/u01/app/11.2.0.4/grid/bin/orarootagent.bin(28629)]CRS-5818:Aborted command 'start' for resource 'ora.cluster_interconnect.haip'. Details at (:CRSAGF00113:) {0:0:242} in /u01/app/11.2.0.4/grid/log/hostname/agent/ohasd/orarootagent_root//orarootagent_root.log.
2015-04-05 00:35:46.343:
[ohasd(27426)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:242} in /u01/app/11.2.0.4/grid/log/hostname/ohasd/ohasd.log.
2015-04-05 00:35:46.674:
[/u01/app/11.2.0.4/grid/bin/oraagent.bin(28409)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0.4/grid/log/hostname/agent/ohasd/oraagent_grid//oraagent_grid.log"

crsctl status res -t -init shows

ora.cluster_interconnect.haip interface not starting and ASM not started.

crsctl status res ora.asm -p

shows a startup an shutdown dependency on ora.cluster_interconnect.haip.

Changes

Performing an 11.2.0.3 to 11.2.0.4 upgrade in either an LDom or zone host and failure occurs at rootupgrade.sh phase.

Cause

The reason for this is that HAIP is not supported on Solaris IPMP interfaces. Our cluster interconnect on SuperCluster uses ipmp.

There is a bug /enhancement filed to get the HAIP support on IPMP <Defect 17837626> SuperCluster / Solaris exadata rootupgrade to 11.2.0.4 fails at haip startup. To prevent this you have to edit the following file BEFORE the upgrade process , once you have run it without the edit nd this haip resource is created you have to follow the recovery process specified below.

cd <GRID_HOME>/crs/install# vi s_crsconfig_lib.pm

Look for the function s_is_sun_ipmp and make the following change
at the end of that function

/is_sun_ipmp

# made it all the way out without finding any IPMP private
#return FALSE;
return TRUE;
"s_crsconfig_lib.pm" 2811 lines, 83736 characters written

Solution

1) Modify the ora.asm parameters under the 11.2.0.4 home env

crsctl modify resource ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd)pullup(ora.cssd,ora.ctssd)weak(ora.drivers.acfs)'" -f -init

crsctl modify resource ora.asm -attr "STOP_DEPENDENCIES=hard(intermediate:ora.cssd)" -f -init

2) Remove the erroneous ora.cluster_interconnect.haip resource form crs in the 11.2.0.4 environment.

crsctl delete resource ora.cluster_interconnect.haip -f -init

3) Downgrade the GI from 11.2.0.4 to 11.2.0.3, form the 11.2.0.4 environment, using the force flag

$NEW_HOME/crs/install/rootcrs.pl -downgrade -force -oldcrshome $OLD_HOME -version $OLD_VERSION

4) Repeat the upgrade process form the begining but only after you verify the s_crsconfig_lib.pm is modified as above.

Attachments

This solution has no attachment