SuperCluster - IB Connectivity Issues in Expansion Storage Configuration After IBSwitch Reboot

Asset ID:	1-72-1928383.1
Update Date:	2015-03-24
Keywords:

Solution Type Problem Resolution Sure

Solution 1928383.1 : SuperCluster - IB Connectivity Issues in Expansion Storage Configuration After IBSwitch Reboot

Applies to:

Oracle SuperCluster T5-8 Full Rack - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M6-32 Hardware - Version All Versions to All Versions [Release All Releases]
SPARC SuperCluster T4-4 - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster T5-8 Half Rack - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster T5-8 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

After rebooting the IB switch, some IB Connections are reporting down.

We found the spine switch was not the master.

Changes

Customer had rebooted a IB Switch and we lost some IB connectivity.

Cause

The spine switch was not the master, but the switch in Exadata Storage Expansion Rack (IB4 in this case) was the master.

[root@pdcsc1sw-ib4 ~]# getmaster -l
Local SM enabled and running, state MASTER

[root@pdcsc1sw-ib1 ~]# getmaster -l
Local SM enabled and running, state STAND BY

Supercluster expansion storage config was incomplete in IBSwitch perspective.
The 2 IB Switches in expansion rack (3 in the main storage rack) was not added to the smnodes configuration.

The spine was supposed to be the master, when it want to the switch in expansion rack ib switch which was not part of config we had this issue.

per analysis in webex.

Solution

The following steps were performed to fix this configuration.

1) We disabled SM in all smnodes.
# disablesm

2) Then we did enablesm in spine (ib1 here)
# enablesm

3) Checked the configvalid (cat /conf/configvalid) on all switches (it is 1)

4) Also added the missing smnodes in IB1,IB2,IB3
smnodes add <management ipdaddr of IB>
smnodes list
-> shows all 5 IP

Added all 5 IPs in IB4 and IB5

# smnodes add <management ipdaddr of IB>
# smnodes list
-> shows all 5 IP in all IB Switches

5) Checked the smpartition list active on the spine. This should be same in all switches.

# smpartition list active

6) enablesm on all the other switches
# enablesm

7) then from the spine issue (ensure this Spine is Master and rest all Standby)
# smpartition start
then
# smpartition commit

References

<NOTE:1912199.1> - Configuring Infiniband Switches when Interconnecting an Oracle SuperCluster and a Exadata Expansion Rack

Attachments

This solution has no attachment