Running "bdacheckib -s" Command to Verifiy Infiniband Topology on New Install of Oracle Big Data Appliance X4-2 Shows Many Links as DOWN

Asset ID:	1-72-1663348.1
Update Date:	2014-10-03
Keywords:

Solution Type Problem Resolution Sure

Solution 1663348.1 : Running "bdacheckib -s" Command to Verifiy Infiniband Topology on New Install of Oracle Big Data Appliance X4-2 Shows Many Links as DOWN

Applies to:

Big Data Appliance X4-2 Hardware - Version All Versions and later
Linux x86-64

Symptoms

Running "bdacheckib -s" command to verifiy Infiniband Topology on new Oracle Big Data Appliance X4-2 but many links are showing as down. The -s switch uses the BDAShip.json to verify the Infiniband Topology.

[root@bda01 bda]# bdacheckib -s
using switch names <switch prefix name>-bda1sw-i and <switch prefix name>-bda1sw-i
LINK <switch prefix name>-bda1sw-i.0B   ... <switch prefix name>-bda1sw-i.8B    UP
LINK <switch prefix name>-i.1B   ... <switch prefix name>-bda1sw-i.8B    UP
LINK <switch prefix name>-bda1sw-i.15A ... bda02.HCA-1.2           UP
LINK <switch prefix name>-i.15B ... bda01.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.14A ... bda04.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.14B ... bda03.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.13A ... bda06.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.13B ... bda05.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.12A ... bda07.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.12B ... <switch prefix name>-bda1sw-i.11B   UP
LINK <switch prefix name>-bda1sw-i.9B   ... <switch prefix name>-bda1sw-i.9A    UP
LINK <switch prefix name>-bda1sw-i.9A   ... <switch prefix name>-bda1sw-i.9B    UP
LINK <switch prefix name>-bda1sw-i.10B ... <switch prefix name>-bda1sw-i.10A   UP
LINK <switch prefix name>-bda1sw-i.10A ... <switch prefix name>-bda1sw-i.10B   UP
LINK <switch prefix name>-bda1sw-i.11B ... <switch prefix name>-bda1sw-i.12B   UP
LINK <switch prefix name>-bda1sw-i.11A ... bda08.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.3B   ... bda11.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.3A   ... bda12.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.4B   ... bda09.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.4A   ... bda10.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.8A   ... <switch prefix name>-bda1sw-i.8A    UP
LINK <switch prefix name>-bda1sw-i.8B   ... <switch prefix name>-bda1sw-i.0B    UP
LINK <switch prefix name>-bda1sw-i.15A ... bda02.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.15B ... bda01.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.14A ... bda04.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.14B ... bda03.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.13A ... bda06.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.13B ... bda05.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.12A ... bda07.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.12B ... <switch prefix name>-bda1sw-i.11B   UP
LINK <switch prefix name>-bda1sw-i.9B   ... <switch prefix name>-bda1sw-i.9A    UP
LINK <switch prefix name>-bda1sw-i.9A   ... <switch prefix name>-bda1sw-i.9B    UP
LINK <switch prefix name>-bda1sw-i.10B ... <switch prefix name>-bda1sw-i.10A   UP
LINK <switch prefix name>-bda1sw-i.10A ... <switch prefix name>-bda1sw-i.10B   UP
LINK <switch prefix name>-bda1sw-i.11B ... <switch prefix name>-bda1sw-i.12B   UP
LINK <switch prefix name>-bda1sw-i.11A ... bda08.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.3B   ... bda11.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.3A   ... bda12.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.4B   ... bda09.HCA-1.1           UP
LINK <switch prefix name>-bda1sw-i.4A   ... bda10.HCA-1.2           UP
LINK <switch prefix name>-bda1sw-i.8A   ... <switch prefix name>-bda1sw-i.8A    UP
LINK <switch prefix name>-bda1sw-i.8B   ... <switch prefix name>-bda1sw-i.1B    UP
LINK <switch prefix name>-bda1sw-i.0A   ... bda18.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.0B   ... bda17.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.1A   ... bda16.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.1B   ... bda15.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.2A   ... bda14.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.2B   ... bda13.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.3A   ... bda12.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.3B   ... bda11.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.4A   ... bda10.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.0A   ... bda18.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.0B   ... bda17.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.1A   ... bda16.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.1B   ... bda15.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.2A   ... bda14.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.2B   ... bda13.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.3A   ... bda12.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.3B   ... bda11.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.4A   ... bda10.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.4B   ... bda09.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.11A ... bda08.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.12A ... bda07.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.13A ... bda06.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.13B ... bda05.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.14A ... bda04.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.14B ... bda03.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.15A ... bda02.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.15B ... bda01.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.4B   ... bda09.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.11A ... bda08.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.12A ... bda07.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.13A ... bda06.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.13B ... bda05.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.14A ... bda04.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.14B ... bda03.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.15A ... bda02.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.15B ... bda01.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.0A   ... bda18.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.0B   ... bda17.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.1A   ... bda16.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.1B   ... bda15.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.2A   ... bda14.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.2B   ... bda13.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.0A   ... bda18.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.0B   ... bda17.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.1A   ... bda16.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.1B   ... bda15.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.2A   ... bda14.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.2B   ... bda13.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.0A   ... bda18.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.0B   ... bda17.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.1A   ... bda16.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.1B   ... bda15.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.2A   ... bda14.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.2B   ... bda13.HCA-1.1           DOWN
LINK <switch prefix name>-bda1sw-i.0A   ... bda18.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.0B   ... bda17.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.1A   ... bda16.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.1B   ... bda15.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.2A   ... bda14.HCA-1.2           DOWN
LINK <switch prefix name>-bda1sw-i.2B   ... bda13.HCA-1.2           DOWN

This was run on a 12-node Oracle Big Data Appliance (BDA) Cluster so some of the nodes would show as DOWN and this would be expected.

Cause

This was caused by spine switch and leaf switches with names consisting of 19 characters in length. This caused the switch names to be truncated which also caused the switch names to appear to have the same name due to the truncation. This is an issue with the Infiniband script of bdacheckib. The BDA Configurator version 2.4.0/2.5.0 did not warn of the limit of 17 characters. In a future version there will be a character limit added in the BDA Configurator.

Solution

1. Change the names of the switches in DNS to be a shorter length of 16 characters or less.

2. Change the configuration of the leaf switches (Gateway switches) to the new names. Referred to as ib2 and ib3 in a default configuration.

To do this perform the following steps:

a. Connect to the first of the Leaf switches (Gateway switches) referred to as ib2 in a default configuration with a serial cable between the IB switch's USB serial adapter and a laptop.

b. Using Putty application change connection type to serial.

c. The Serial line will need to be changed to COM<n> for whatever the laptop has been configured for.

d. The default serial port speed is 115200 baud. Change the Speed to 115200.

e. Click on Open

f. Login as ilom-admin. The switch OS is Linux-based but has an ILOM interface that will be used to make the necessary configuration changes.

localhost: ilom-admin
password: welcome1

g. Set and verify the switch hostname without using the domain name:

Change to the new hostname as set or will be set in DNS
Example:

-> set /SP hostname=bda1sw-ib2
-> show /SP hostname
/SP
Properties:
hostname = bda1sw-ib2

h. Logout from the IB leaf switch.

Exit the ILOM shell:

-> exit

i. Login to the IB switch linux as root and reboot the switch to ensure all the changes take effect.

localhost: root
password:welcome1
[root@localhost ~]# reboot

j. Repeat the steps a-i on the second leaf switch (referred to with ib3 extension in default configuration).

3. Change the configuration of the spine switch to the new name in DNS.

To do this perform the following steps:

a. Connect to the Spine switch with a serial cable between the IB switch's USB serial adapter and a laptop.

b. Using Putty change connection type to serial.

c. The Serial line will need to be changed to COM<n> for whatever the laptop has been configured for.

d. The default serial port speed is 115200 baud. Change the Speed to 115200.

e. Click on Open

f. Login as ilom-admin. The switch OS is Linux-based but has an ILOM interface that will be used to make the necessary configuration changes.

localhost: ilom-admin
password: welcome1

g. Set and verify the switch hostname without using the domain name:

Change to the new hostname as set in DNS or will be set in DNS in step 1 above. Set the switch hostname without using the domain name:

-> set /SP hostname=bda1sw-ib1
-> show /SP hostname
/SP
Properties:
hostname = bda1sw-ib1

h. Logout from the IB spine switch.

Exit the ILOM shell:

-> exit

i. Login to the IB spine switch linux as root and reboot the switch to ensure all the changes take effect.

localhost: root
password:welcome1
[root@localhost ~]# reboot

4. Change the spine switches and leaf switches to use shorter names in the BDA Configurator 2.4.0/2.5.0.

5. Regenerate the files with BDA Configurator so that the BdaDeploy.json is updated as well with the shorter switch names.

6. Run the "bdacheckib -s" command again and verify that the results are showing up correctly now and the switch names are not being truncated.

[root@bda01 bda]# bdacheckib -s

References

<BUG:18554218> - INFINIBAND NODE_DESC AND GW SWITCHES TRUNCATED IF 19 CHARACTERS FOR HOSTNAME

Attachments

This solution has no attachment