Failure to Backout IB Switch FW Upgrade - README Documentation Bug

Asset ID:	1-71-2048141.1
Update Date:	2015-11-10
Keywords:

Solution Type Technical Instruction Sure

Solution 2048141.1 : Failure to Backout IB Switch FW Upgrade - README Documentation Bug

Applies to:

Oracle SuperCluster T5-8 Full Rack - Version All Versions and later
Information in this document applies to any platform.

Goal

Downgrade fails with error that an up-rev version is applied and that it can't replace it with the old one:

# more ibswitch
ssc01-swi-01
# export EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION=2.1.3-4
# ./ibsw-patchmgr -sw ibswitch -downgrade -precheck
2015-07-09 09:42:33 -0500 1 of 1 :SUCCESS: DO: Initiate pre-downgrade validation check on InfiniBand switch(es).
----- InfiniBand switch update process started Thu Jul 9 09:42:33 CDT 2015 -----
[NOTE     ] Log file at /var/log/cellos/upgradeIBSwitch.log

[INFO     ] List of InfiniBand switches for rollback: ( ssc01-swi-01 )
[PROMPT   ] Use the default password for all switches? (y/n) [n]: n
[PROMPT   ] Use same password for all switches? (y/n) [n]:
Enter admin or root password for ssc01-swi-01:
Confirm password:
[PROMPT   ] Mismatch in the list of switches provided ( ssc01-swi-01 ) and the output of the command ibswitches ( adc-ib-bu141-swi- adc-ib-bu141-swi- adc-ib-bu141-swi- ). Are you sure you want to continue? (y/n) [n]: y
[PROMPT   ] Updating only 1 switch(es). Are you sure you want to continue? (y/n) [n]: y
[SUCCESS ] Verifying Network connectivity to ssc01-swi-01
[INFO     ] Master Subnet Manager is set to ssc01-swi-01- in all Switches
[PROMPT   ] Found 1 InfiniBand switches with subnet manager running. Are you sure you want to continue? (y/n) [n]: y
[INFO     ] Found 1 InfiniBand switches with subnet manager running. Continuing as per user request.

[INFO     ] ---------- Starting with IBSwitch ssc01-swi-01
[FAIL     ] ssc01-swi-01 is at 2.1.5-1. Does not meet minimum patching level 2.1.3-4 <<<<<<<< EH??? >>>>>>>>>
[SUCCESS ] Verifying that /tmp has 120M in ssc01-swi-01, found 249M
[SUCCESS ] Verifying that / has 80M in ssc01-swi-01, found 198M
[SUCCESS ] Verifying that ssc01-swi-01 has 120M free memory, found 413M
[SUCCESS ] Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth[0,1] for ssc01-swi-01
[SUCCESS ] Verifying that ssc01-swi-01 has at least 1 NTP Server, found 1
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-07-09 Time:(HH:MM:SS) 09:59:51
[FAIL     ] Pre-update validation on ssc01-swi-01
[FAIL     ] Overall status

----- InfiniBand switch update process ended Thu Jul 9 09:44:11 CDT 2015 -----
2015-07-09 09:44:11 -0500 1 of 1 :FAILED: Details in files /var/log/cellos/upgradeIBSwitch.log /opt/oracle.supercluster/patchmgr-ibsw/patchmgr.stdout, /opt/oracle.supercluster/patchmgr-ibsw/patchmgr.stderr
2015-07-09 09:44:11 -0500 1 of 1 :FAILED: DONE: Initiate pre-downgrade validation check on InfiniBand switch(es).

Checking the version information:

[root@ssc01-swi-01 ~]# version
SUN DCS 36p version: 2.1.5-1
Build time: Oct 6 2014 10:35:15
SP board info:
Manufacturing Date: 2013.08.14
Serial Number: "NCDBW0256"
Hardware Revision: 0x0007
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

README says that one should downgrade the InfiniBand Switches to target version as follows, which will fail:

export EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION=2.0.6-1

./ibsw-patchmgr -downgrade

or

./ibsw-patchmgr -sw ibswitch -downgrade

Solution

The Readme contains incorrect information as follows:

1. One needs to set both ENV variables while the readme only says to set one.

# export EXADATA_IMAGE_IBSWITCH_UPGRADE_VERSION=2.1.5-1

and

# export EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION=2.1.3-4

2. One must have a copy of each of the FW packages in the same directory.

The precheck utility does not look for the existence of the FW pkg. Therefore if the FW pkg is not in `pwd` the precheck will pass, but the installation will fail.

The precheck needs to check for presence of both of the FW PKG's before any upgrade or downgrade in versions.

References

<NOTE:1910505.1> - SuperCluster- July 2014 Quarterly Full Stack Download Patch (QFSDP) Companion Document
<NOTE:1567979.1> - Oracle SuperCluster Supported Software Versions - All Hardware Types
<NOTE:1569461.1> - SuperCluster - Patching Best Practices For The Quarterly Full Stack Download Patch

Attachments

This solution has no attachment