Sun Infiniband HCA
 
Sun Datacenter InfiniBand Switch 36
 
OpenSolaris Operating System
 

PLA-Support>Sun Systems>SAND>Network>SN-SND: Sun Network Infiniband
 

In this Document

Symptoms

Cause

Solution

Created from <SR 3-8042602700>

Applies to:

Symptoms

"ib" devices are found missing when a host running Solaris 10 is rebooted.  The host has an infiniband HCA.  This HCA is linked to infiniband switch (nm2-36P or nm2-Gateway).  The host is rebooted at a time when there is no master subnet manager in the infiniband network.  The "dladm show-dev" output on the Solaris host does not show any infiniband devices after reboot when they should be there.

Cause

For infiniband network to function properly, there has to be a master subnet manager running in that network.   A Solaris 10 host with an infiniband HCA connected to an infiniband network (connected to an infiniband switch) will not show the "ib" devices after a reboot if there is no master subnet manager in the network.

There are several factors which can lead to a situation with no master subnet manager in a network.  Missing infiniband cabling or faulty cables can be one such factor.  A misconfiguration of the infiniband nodes is another factor.   Enabling subnet manager and setting proper smpriorities are important to have a master subnet manager.   On infiniband switches (nm2-36P and nm2-Gateway) running firmware version lower than 2.0,  "/conf/configvalid" must be set to 1.    Issues (bugs) with firmware of the infinibnad switches can also result in this situation occasionally.  For example, the following bug in the firmware of the infiniband switch (nm2-36P and nm2-Gateway) can result in no master subnet manager when it recovers from a power outage.

BUG 15692591- SUNBT7013467 1.3.3 Too strict error handling in partitioned.

(This bug has been fixed in firmware version 2.0.1 or later.)

So, in an infiniband network with these IB switches running firmware version lower than 2.0.1,  it is possible that there is no master subnet manager when the switches recover from a power outage.  A power outage can also result in a reboot of the Solaris hosts connected to these switches.  If the host is running Solaris 10,  this can result in missing infiniband devices in that host.  The "dladm show-dev" output on the Solaris 10 host may not show "ib" devices.  This can also happen if the Solaris 10 host is manually rebooted during the period when there is no master subnet manager in the infiniband network.  This issue is not seen with hosts running Solaris 11 or Linux.

Solution

To resolve this, first the issues with subnet manager have to be fixed.  If the switches are running firmware version lower than 2.0.1, upgrade it to the latest.  This will take care of the known bug 15692591.

Make sure that subnet managers are enabled (i.e. run "enablesm") on the switches.  If this is an engineered system (exada, exalogic, supercluster),  refer to its documents to find out which switches should be running subnet manager.  Set smpriorities of each of these switches as per the same documents.   Make sure that "/conf/configvalid" is set to 1 on all infiniband switches running firmware version lower than 2.0.  In firmware versions 2.0 or newer,  setting of /conf/configvalid can be anything and it has no effect.  Instead, makle sure that smparttition is propagated correctly to all IB switch running opensm by executing the following two commands on the IB switch running the current Master.  Prior to running these two commands, make sure that "smnodes list" output on all those IB switch running opensm are identical and includes the management IP addresses of all those IB switches.

#smpartition start
#smpartition commit

Check the output of "sminfo" on any of the infiniband switch.  Also, check the output of "getmaster -l" on all the switches.

( The following document is relevant here in collecting these outputs.

      Exadata IB Switch - "getmaster" returns "No Master SubnetManager seen in the system" (Doc ID 1334834.1).

  So, in an exadata,  make sure that this data is not collected from a spine switch which is not a master)

A typical normal output should look as follows:

# sminfo
sminfo: sm lid 1 sm guid 0x212856cbe2c0a0, activity count 9131 priority 0 state 3 SMINFO_MASTER

   (This output shows that the master subnet manager is running on LID 1)

# getmaster -l
Local SM enabled and running    <<<<<<<<<  Note
Last ring buffer history listed:
20131111 20:20:59 No Master SubnetManager seen in the system
20131111 20:28:21 Master SubnetManager on sm lid 1 sm guid 0x212856cbe2c0a0 : SUN IB QDR GW switch infiniband36p-gat 10.186.85.157
20131111 20:38:24 No Master SubnetManager seen in the system
20131111 20:43:14 Master SubnetManager on sm lid 1 sm guid 0x212856cbe2c0a0 : SUN IB QDR GW switch infiniband36p-gat 10.186.85.157
20131111 20:51:57 No Master SubnetManager seen in the system
20131111 20:58:07 Master SubnetManager on sm lid 1 sm guid 0x212856cbe2c0a0 : SUN IB QDR GW switch infiniband36p-gat 10.186.85.157    <<<<<<<<<  Note

Note the last line in the getmaster -l output.  That shows that the master subnet manager is running on LID 1.

Once this is fixed, either reboot the solaris 10 host, or run the following command on that solaris host:

#devfsadm -C

After this, check the "ib" devices using dladm command:

Here is an example output:

# devfsadm -C

# dladm show-dev
igb0            link: up        speed: 1000  Mbps       duplex: full
igb1            link: unknown   speed: 0     Mbps       duplex: unknown
usbecm0         link: up        speed: 10    Mbps       duplex: full
ibd0            link: down      speed: 32000 Mbps       duplex: unknown
ibd1            link: unknown   speed: 0     Mbps       duplex: unknown   <<<<<<<   Note:  ib devices are now seen
igb2            link: unknown   speed: 0     Mbps       duplex: unknown   <<<<<<<
igb3            link: unknown   speed: 0     Mbps       duplex: unknown

ibd0 and ibd1 are the infiniband devices on this Solaris host.

This solution has no attachment