| Asset ID: |
1-72-1605955.1 |
| Update Date: | 2017-09-29 |
| Keywords: | |
Solution Type
Problem Resolution Sure
Solution
1605955.1
:
How to enable an Infiniband SUN DCS 36p switch port when it 's auto-disabled
| Related Items |
- Exadata X3-2 Hardware
- Exadata Database Machine X2-2 Hardware
- Sun Network QDR InfiniBand Gateway Switch
|
| Related Categories |
- PLA-Support>Sun Systems>SAND>Network>SN-SND: Sun Network Infiniband
- _Old GCS Categories>Announcements>All Product Lines>Communications>All Product Lines
|
In this Document
Created from <SR 3-8060803101>
Applies to:
Sun Network QDR InfiniBand Gateway Switch - Version All Versions and later
Exadata Database Machine X2-2 Hardware - Version All Versions and later
Exadata X3-2 Hardware - Version All Versions and later
All Platforms
autodisable feature has been introduced only in firmware 2.1 and does not apply to firmware 1.3.
Correct way to account for this is to check and ensure whether any auto-disabled ports exist and if present then re-enable using enableswitchport --automatic 'before' up/downgrading fw to a different version. This will ensure compatible settings when moving between different fw.
autodisable feature is in the 2.x series and should be in the /conf/disabledport.conf if autodisable is set for this port. This should not be here for the 1.x series and looks like a artifact from the downgrade Our internal team is looking at how to handle this in the future
Note: Using ibportstate to administer a port will not be preserved through subsequent reboots, hence should not be relied on. You should always use en(dis)ableswitchport.
since /conf/disabledports.conf does not apply to 1.3, you can safely delete it for now and issue a switch reset command 'ibdevreset Switch' or simply reboot the switch.
for switch with version 2.1.3-4 run enableswitchport --automatic command and port number
Symptoms
listlinkup show that port 29 is autodisabled.
# listlinkup
Connector 4B Present <-> Switch Port 27 is up (Enabled)
Connector 5B Present <-> Switch Port 29 is down (AutomaticBadSpeedOrWidth) <---------------
Connector 6B Present <-> Switch Port 36 is up (Enabled)
Changes
To enable the port , you first need to determine the firmware version that 's running on the switch using the "version" command.
I included below the output for version: 2.1.3-4 and version: 1.3.3-2 which is supported as per the latest release notes.
version
SUN DCS 36p version: 2.1.5-1
version
SUN DCS 36p version: 2.1.3-4
version
SUN DCS 36p version: 1.3.3-2
Cause
The issue can be caused by a bad partner link or misconfiguration
Solution
For version 1.3.3-2 , 2.1.3-4 and above , make sure the file /conf/disabledports.conf doesn't exist .
# version
SUN DCS 36p version: 1.3.3-2
Build time: Apr 4 2011 11:15:19
SP board info:
Manufacturing Date: 2011.12.25
Serial Number: "NCD7K0152"
Hardware Revision: 0x0006
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010
# getportstatus 29
Port status for connector 3B Switch Port 29
Adminstate:......................Disabled (AutomaticBadSpeedOrWidth)
LinkWidthEnabled:................1X or 4X
LinkWidthSupported:..............1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkState:.......................Down
PhysLinkState:...................Disabled
LinkSpeedActive:.................2.5 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
This below command works on version: 2.1.3-4 but not on version: 1.3.3-2
# enableswitchport --reason=AutomaticBadSpeedOrWidth 29
Invalid reason AutomaticBadSpeedOrWidth
Usage:
enableswitchport [--reason=reason] connector | [ibdevicename] port
Values for ibdevicdename: Switch
Values for port: 1-36
Values for connector: 0A-17A, 0B-17B
Values for reason: Blacklist, Partition
Somehow /conf/disabledports.conf had these settings. Only possibility I can think of is this system had a fw with the autodisable feature with 2 ports that had this condition and then the system was downgraded and this file did not get updated. the file should be renamed or removed completely.
cat /conf/disabledports.conf
# List of Disabled ports
# Format:
# ibdev port Adminstate
#Switch 34 AutomaticBadSpeedOrWidth
#Switch 29 AutomaticBadSpeedOrWidth
/var/log/installfw.log shows that the switch was downgraded from version: 2.1.3-4 to version: 1.3.3-2
[ 2 Oct 05:09:16][getShelfManagerDetails:709]: SUN DCS 36p version: 2.1.3-4
[ 2 Oct 05:09:16][executeMST:577]: Executing command: sudo mst start >>/var/log/installfw.stderr 2>&1
[ 2 Oct 05:09:17][getMSTAddr:267]: Executing command: mst status
[ 2 Oct 05:09:18][getFWVersion:147]: Executing command: flint -d /dev/mst/mt48436_pci_cr0 q
[ 2 Oct 05:09:19][executeFSwitchUpdateCommand:485]: Executing command: flint -y -allow_psid_change -d /dev/mst/mt48436_pci_cr0 -i SUN_DCS_36p_i4_7.3.000_v0.0.8.bin b 2>>/var/log/installfw.stderr
[ 2 Oct 05:09:28][executeFSwitchUpdateCommand:517]: Executing command: /usr/local/sbin/ibdevreset
[ 2 Oct 05:09:53][verifyFSwitchUpdate:843]: Executing command: cat /sys/class/infiniband/is4_0/fw_ver
[ 2 Oct 05:09:53][getFWVersion:147]: Executing command: flint -d /dev/mst/mt48436_pci_cr0 q
[ 2 Oct 05:09:54][shmmUpgradeCommand:324]: Executing command: SPFW_UPGRADE_OPTIONS='--oldpackage' sh spfw_upgrade_1.3.3-2.sh --f
after removing or renaming the file /conf/disabledports.conf, enable the port as follow
enableswitchport Switch 29
getportstatus 29
Port status for connector 5B Switch Port 29
Adminstate:......................Enabled
LinkWidthEnabled:................1X or 4X
LinkWidthSupported:..............1X or 4X
LinkWidthActive:.................1X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkState:.......................Active
PhysLinkState:...................LinkUp
LinkSpeedActive:.................10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
ibportstate 29 query
PortInfo:
# Port info: Lid 29 port 0
LinkState:.......................Active
PhysLinkState:...................LinkUp
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................10.0 Gbps
#listlinkup
Connector 4B Present <-> Switch Port 27 is up (Enabled)
Connector 5B Present <-> Switch Port 29 is up (Enabled)
Connector 6B Present <-> Switch Port 36 is up (Enabled)
to disable the port again you can use
disableswitchport switch 29
Now for version SUN DCS 36p version: 2.1.3-4 and above
# listlinkup
Connector 5B Present <-> Switch Port 29 is down (AutomaticBadSpeedOrWidth) <----------
# enableswitchport --automatic Switch 29
Enable connector 5B Switch port 29
Adminstate:......................Enabled
LinkWidthEnabled:................1X or 4X
LinkWidthSupported:..............1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkState:.......................Down
PhysLinkState:...................PortConfigurationTraining
LinkSpeedActive:.................2.5 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
NeighborMTU:.....................2048
OperVLs:.........................VL0-7
# ibportstate 3 29
PortInfo:
# Port info: Lid 3 port 29
LinkState:.......................Down
PhysLinkState:...................Polling
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................10.0 Gbps
# ibportstate 3 29
PortInfo:
# Port info: Lid 3 port 29
LinkState:.......................Down
PhysLinkState:...................Polling
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................10.0 Gbps
#
# ibportstate 3 29
PortInfo:
# Port info: Lid 3 port 29
LinkState:.......................Active
PhysLinkState:...................LinkUp
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................2.5 Gbps
Peer PortInfo:
# Port info: Lid 3 DR path slid 65535; dlid 65535; 0,29 port 2
LinkState:.......................Active
PhysLinkState:...................LinkUp
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................2.5 Gbps
ibwarn: [2687] validate_speed: Peer ports operating at active speed 1 rather than 4 (10.0 Gbps)
[root@scam07sw-ibb0 IBdata]# ibportstate 3 29
PortInfo:
# Port info: Lid 3 port 29
LinkState:.......................Down
PhysLinkState:...................Disabled
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................2.5 Gbps
[root@scam07sw-ibb0 IBdata]#
This might explain why the port got disabled in the first place. You have an SDR link speed on peer port instead of QDR. It could be either a cable or HCA problem. try switching cables?
Attachments
This solution has no attachment
