Exadata and SuperCluster Firmware Verification on InfiniBand Switch xxx -ib1 Failed, Possibly Due to Custom Configuration Changes

Asset ID:	1-72-2003562.1
Update Date:	2015-05-23
Keywords:

Solution Type Problem Resolution Sure

Solution 2003562.1 : Exadata and SuperCluster Firmware Verification on InfiniBand Switch xxx -ib1 Failed, Possibly Due to Custom Configuration Changes

Applies to:

Oracle Exadata Storage Server Software - Version 12.1.2.1.1 to 12.1.2.1.1 [Release 12.1]
Sun Datacenter InfiniBand Switch 36 - Version All Versions to All Versions [Release All Releases]
SPARC SuperCluster T4-4 - Version All Versions and later
Oracle SuperCluster T5-8 Hardware - Version All Versions and later
Oracle SuperCluster M6-32 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

While upgrading the ibswitches you encounter errors during the verification of the installed files
The most likely version is on 2.1.5-1

$ ./patchmgr -ibswitches ibswitches.lst -upgrade

2015-04-- 15:25:35 +000 1 of 1 :

SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.5-1.

Expect up to 15 minutes for each switch
----- InfiniBand switch update process started Sat Apr 2015 -----
[NOTE ] Log file at /var/log/cellos/upgradeIBSwitch.log
...
...
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-04-25 Time:(HH:MM:SS) 15:01:51
[WARNING ] Firmware verification on InfiniBand switch abc-ib01 failed, possibly due to custom configuration changes
[INFO ] spawn ssh root@abc-ib01 /usr/local/bin/fwverify
[INFO ] root@abc-ib01's password:
[INFO ] Checking all present packages:
[INFO ] ........................................................................................................................................... OK
[INFO ] Checking if any packages are missing:
[INFO ] ...................................................................................................................................... OK
[INFO ] Verifying installed files:
[INFO ] ......................................................................................................................................... FAILED
[INFO ] * Package tzdata-2012c-1.el5.i386:
[INFO ] .M...... /usr/share/zoneinfo/Etc/GMT
[INFO ] .M...... /usr/share/zoneinfo/Etc/GMT+0
[INFO ] .M...... /usr/share/zoneinfo/Etc/GMT-0
[INFO ] .M...... /usr/share/zoneinfo/Etc/GMT0
[INFO ] .M...... /usr/share/zoneinfo/Etc/Greenwich
[INFO ] .M...... /usr/share/zoneinfo/GMT
[INFO ] .M...... /usr/share/zoneinfo/GMT+0
[INFO ] .M...... /usr/share/zoneinfo/GMT-0
[INFO ] .M...... /usr/share/zoneinfo/GMT0
[INFO ] .M...... /usr/share/zoneinfo/Greenwich
[INFO ] Checking FW Coreswitch:
[INFO ] FW Version: 7.4.2202 OK
[INFO ] PSID: SUN_NM2-36p_004 OK
[INFO ] Verifying image integrity OK

Second Example from research
==========================

Upgrading IB switches from version 2.1.4-1 to 2.1.5-1 fails with the following error:

[root@xyzsw-ib3 ~]# fwverify

Checking all present packages:
..............................................................................
............................................................. OK

Checking if any packages are missing:
..............................................................................
........................................................ OK

Verifying installed files:
..............................................................................
........................................................... FAILED

* Package tzdata-2012c-1.el5.i386:
.M...... /usr/share/zoneinfo/America/Los_Angeles
.M...... /usr/share/zoneinfo/US/Pacific

Checking FW Coreswitch:
FW Version: 7.4.2202 OK
PSID: SUN_NM2-36p_004 OK
Verifying image integrity OK

Closer inspection confirmed that the permissions on the following files on the IB Switch are set to 777

* Package tzdata-2012c-1.el5.i386:
.M...... /usr/share/zoneinfo/America/Los_Angeles
.M...... /usr/share/zoneinfo/US/Pacific

Checking FW Coreswitch:
FW Version: 7.4.2202 OK
PSID: SUN_NM2-36p_004 OK
Verifying image integrity OK

Changes

patching IBSwitches to 2.1.5-1

Cause

Bug 20458988 when the permissions are not set as expected when upgrading IB switches to 2.1.5-1 Fails

"...I found that the Firmware install fails because the permission of this files is set to 777 ..."

Solution

Switch permissions for the reported tz files on the switch to 644

Rerun the patch

Then the switch firmware upgrade should be successful.

References

<BUG:20458988> - UPGRADING IB SWITCHES TO 2.1.5-1 FAILS

Attachments

This solution has no attachment