Exacheck Reports Topology Errors On Infiniband Network

Asset ID:	1-72-1483604.1
Update Date:	2013-10-03
Keywords:

Solution Type Problem Resolution Sure

Solution 1483604.1 : Exacheck Reports Topology Errors On Infiniband Network

Applies to:

Exadata Database Machine X2-2 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Exacheck Reports some infiniband topology errors.
verify-topology also reports errors as seen below.

Leaf Switch dm02sw-ib2.example.com with GUID 0x2128469d62a0a0has fewer than 8 links to compute nodes
It has 5 links (14A 13A 13B 12B 5A) to compute nodes
[ERROR]

However an investigation usng listlinkup output and ibnetdiscover does not show any cable connectivity issues.

ibnetdiscover output has entries like the following

[28] "H-00212800013f1346"[2](212800013f1348) # "dm02db07 HCA-1" lid 62 4xQDR

Ideally the correct entry in ibnetdiscover should be like the following. In the previous output IP address is misisng.

[12] "H-00212800013f22d6"[1](212800013f22d7) # "dm02db04 S 192.168.10.26 HCA-1" lid 44 4xQDR

Changes

A software upgrade.

Cause

<Bug 14373741>: IB_SET_NODE_DESC.SH DOES NOT HANDLE BOND0 INTERFACES.

The node description reported in ibnetdiscover output is set by ib_set_node_desc.sh script in each node.

Certain versions of this script expect on the name "bondib0" for the infiniband interface. If the infiniband bond interface name is "bond0" the script fails to set the node description.

Solution

Fix for <Bug 14373741> corrects the script ib_set_node_desc.sh.

To resolve this issue you will need to download file ib_set_node_desc.sh script attached to this note. This file is part of the 11.2.3.2.0 image. Run the script on each compute node which will fix the node description. and allow ibnetdiscover to correctly identifed the compute nodes.

References

<BUG:14373741> - IB_SET_NODE_DESC.SH DOES NOT HANDLE BOND0 INTERFACES

Attachments

This solution has no attachment