Big Data Appliance Hardware
 
Exadata X4-2 Full Rack
 

PLA-Support>Sun Systems>SAND>Network>SN-SND: Sun Network Infiniband
 

This document describes how to migrate an existing IPoIB subnet from default to custom InfiniBand Partition.

In this Document

Goal

Solution

Steps to accomplish the task of migrating IPoIB from default to custom partition

Step 1: Identify an unused InfiniBand Partition key

Step 2: Determine if any node need to be added as limited member to the IB partition

Step 3: Collect all IB port GUIDs of the host compute nodes that need to be part of this new IB partition

Step 4: Identify the IB switch running master subnet manager

Step 5: Start IB partition editing on master SM switch

Step 6: Create IB partion on master SM switch

Step 7: Add all IB port GUIDs in the new IB partition

Step 8: Save and commit the switch configuration

Step 9: Verify that the IB partition information is propagated into all standby SM switches

Step 10: Backup current IPoIB configuration on all host compute nodes

Step 11: Migrate current IPoIB configuration into new IB partition based configuration

Step 12: Reboot first two host compute nodes one at a time

Step 13: Verify the connectivity between the rebooted nodes

Step 14: Reboot remaining host compute nodes one at a time

References

Applies to:

Goal

 To use a custom InfiniBand Partition instead of default and migrate the existing IPoIB subnet keeping all IP addresses same. 

Solution

 This activity can be broken down into following tasks which are required to accomplish this.

1. Configuring the InfiniBand Switches

2. Configuring the host compute nodes 

Steps to accomplish the task of migrating IPoIB from default to custom partition

Step 1: Identify an unused InfiniBand Partition key

Login to an IB switch and run the following command to display current partitions on the IB network. 

Pick a unique and unused partition key e.g. 0x0045

Step 2: Determine if any node need to be added as limited member to the IB partition

This step is optional if restrictions are not needed among the nodes in same IB partition. However, if the environment needs some nodes not to communicate with each other inside the new IB partition then these nodes shall be flagged as 'limited' members.

Step 3: Collect all IB port GUIDs of the host compute nodes that need to be part of this new IB partition

Run the following command on all host compute nodes to find out the IB port GUIDs.

Step 4: Identify the IB switch running master subnet manager

From any IB switch in this network, run the following command to identify the IB switch running master subnet manager.

Once you have permissions to run write-enabled administrative tasks on the network, proceed to the following steps

Step 5: Start IB partition editing on master SM switch

Login to the IB switch running master SM and execute following command to enable editing of IB partitions.

Step 6: Create IB partion on master SM switch

Now we can create a new IB partition on the IB network as follows. In this example, we are using pkey 0x0045 and naming it as bda_ipoib_0045

Step 7: Add all IB port GUIDs in the new IB partition

Once the partition is created, we can add all IB port GUIDs in it now.

 In the above command, include all the port GUIDs in hexadecimal notation (e.g. 0x0021280001a16d4c) separated by comma. You can also use one GUID at a time and repeat for all. 

If some port IB GUIDs are required to be added as 'limited' members, then use following syntax for those specific GUIDs only.

Step 8: Save and commit the switch configuration

Once all port GUIDs are added, we can save and commit the new configuration.

Step 9: Verify that the IB partition information is propagated into all standby SM switches

A quick verification must be done before proceeding to host configurations. Run following command to verify all port GUIDs have been added. Make sure to verify that some port GUIDs are added as limited if you did so in previous step.

Step 10: Backup current IPoIB configuration on all host compute nodes

Following are the three configuration files for default IB partition based IPoIB interfaces. We need to backup these files on each individual host compute node.

You can backup these files on each host as follows. 

Step 11: Migrate current IPoIB configuration into new IB partition based configuration

Now we can migrate default IB partition based configuration to new IB partition based by changed the file names and contents.

The file ifcfg-bondib0 can remain as-is. This file contains the IP address, subnet and other useful information which is transparent to underlying IB partition. 

Step 12: Reboot first two host compute nodes one at a time

After making the above changes in step 10-11 on any two nodes, we can reboot them one at a time. This is to ensure that our new configuration works before we change the entire system. Before rebooting, you may also need privileges to do so from system or site administrator. Watch the console closely to make sure the systems come back normally after a reboot. This is just a recommendation but not mandatory.

Step 13: Verify the connectivity between the rebooted nodes

Once the two rebooted nodes come back, its time to verify the connectivity across these two. You can login to the nodes and ping each other's IPoIB address. Also make sure that you are 'not able' to ping any other IPoIB address on which the new partition has not been applied yet.  

Step 14: Reboot remaining host compute nodes one at a time

Once step 13 is successful, we are now ready to apply configuration to all remaining nodes using steps 10 onwards. Once all nodes are completed, we have accomplished our task to migrate IPoIB subnet from default IB partition to a new custom IB partition.

References

This solution has no attachment