Solaris Operating System
 
Sun Datacenter InfiniBand Switch 36
 
Sun Network QDR InfiniBand Gateway Switch
 

PLA-Support>Sun Systems>SAND>Network>SN-SND: Sun Network Infiniband
 

In this Document

Symptoms

Cause

Solution

Created from <SR 3-9880013811>

Applies to:

Symptoms

IB link over particular partition stays down on Solaris node within Exalogic machine.

Customer have many pkeys which is created on infiniband ipoib port that are not working:

#dladm dladm create-part -l net9 -P 8101 p.8101
#ipadm create-ip p.8101

The outcome results was:

#dladm show-part
LINK PKEY OVER STATE FLAGS
p.8101 8101 net9 down ----                <<------ Stay down 

Cause

The mtu setting of the IP partition 8101 on  the IB switch is running on is 4096.  That's mismatched with the default MTU setting 2048 on Solaris node. 

On the SM master IB swtich, we can run "smpartition list active" to examine current mtu setting for the partitions. 

[root@ExIBSw01 ~]# smpartition list active
# Sun DCS IB partition config file
# This file is generated, do not edit
#! version_number : 608
Default=0x7fff, ipoib :
ALL_CAS=full,
ALL_SWITCHES=full,
SELF=full;
SUN_DCS=0x0001, ipoib :
ALL_SWITCHES=full;
..........................
dnfs-zfs-murex-prd = 0x8101,ipoib,rate=7,mtu=5,sl=2,defmember=full:          <------------ mtu=5, 5 means 4096. Default is 2048 when it is 4.
0x0010E0000133A6F1,                
0x0010E0000128C5D9,                 
0x0010E00001340FB1,
..........

Solution

There are two ways to fix this.

1) Change the mtu of ib partition to 2048.  This can be done as follows:

Re-creating the partition with default MTU. 

Regarding the parameters for command smpartition, we can find helps on IB switch by typing "smpartition".

Extract following information as the reference for better understanding this issue. 

Legal value for pkey is 1 to 0x7fff
All numbers are in hexadesimal apart from the following legal values:
mtu : 1, 2, 3 ,4 (default) or 5. These maps to a MTU of: 256, 512, 1024, 2048 or 4096
rate: 2, 3 (default), 4, 5, 6, 7, 8, 9 or 10: Which maps to: 2.5, 10, 30, 5, 20, 40, 60, 80 or 120 Gb/sec
sl: 0 to 15 (default 0) <<--- service level
scope: 1 - 14 (default 1). See the IBTA specification interpretation.
Valid value for defmember are limited, full or both

 Re-create Ib partition as follows:

Here is an example for this re-creation.

# smpartition start
# smpartition delete -n dnfs-zfs-murex-prd             <<---- Delete partition  dnfs-zfs-murex-prd
# smpartition create -n dnfs-zfs-murex-prd -pkey 0x8101 -flag ipoib rate 7 sl 5 -m full      <<--- Recreate the partition  dnfs-zfs-murex-prd with default mtu, rate 7 and sl 5
# smpartition add -n dnfs-zfs-murex-prd <port_guid1> <port_guid2>      <<--- Can add multiple port together
# smpartition list modified               <<---- Review the modification
# smpartition commit              <<---- Run the command to confirm the partition configuration
# smpartition list active           <<---- Double check the modification  

2) Change the mtu of the solaris host.

     MTU of solaris host can be changed as follows:

      Add the following in /etc/system, and reboot the machine.

          set hermon:hermon_enable_4K_MTU=0x1

      Upon reboot, check the mtu of the ports using the following command on this solaris host:

              #ibv_devinfo

This solution has no attachment