OVN :: Double meshed IB topology

Asset ID:	1-72-1598662.1
Update Date:	2017-07-11
Keywords:

Solution Type Problem Resolution Sure

Solution 1598662.1 : OVN :: Double meshed IB topology

Applies to:

Oracle Fabric Interconnect F1-15 - Version All Versions and later
Oracle Fabric Interconnect F1-4 - Version All Versions and later
Information in this document applies to any platform.

Symptoms

When an IB switch is rebooted - both (active and redundant) paths go down. Impact on Data path.

Cause

If the IB Topology is Double Meshed (meshed at both layers) it is possible that both (active and redundant) paths are passing via the same IB switch.

Solution

Referring to the diagram below:

Double mesh

We see that the IB topology is double meshed (at both layers) - between Blade chassis and the IB Switches & IB switches and Fabric Directors.

In the above topology it is possible that the Queue Pairs (QPs) from both Fabric Directors to a particular host/blade could pass through the same IB switch and, if an IB switch happens to go down (due to failure, reboots, or IB Switch F/W upgrades) - the QPs will take a few seconds to come alive on the other active IB paths. This delay can cause an outage to the data paths - as both active and redundant QPs go down together.

FIX:

Recommended fix is to remove the mesh at either Layer-1 or Layer-2 in the above diagram (remove either the RED / BLUE cables). With single mesh -the two QPs from both Fabric Directors to a particular host/blade will pass via both IB switches, in other words we will have active and redundant QPs passing via different switches.

Cabling the IB Fabric so it is singled meshed instead of double meshed ensures that redundancy across all IB switches is maintained.

Attachments

This solution has no attachment