Big Data Appliance Hardware
 

PLA-Support>Eng Systems>BDA>Big Data Appliance>DB: BDA_EST
 

In this Document

Symptoms

Changes

Cause

Solution

Applies to:

Symptoms

After replacing an Infiniband (IB) HCA card on an Oracle Big Data Appliance (BDA), bondeth0 is not up.

The symptoms observed are like those below:

1. Traffic is going through the ports.

2. However eth8, eth9 and bondeth0 are not up.

a) Nothing is returned from:

b) Nothing is visible for eth8/eth9/bondeth0 from:

Changes

 The IB HCA card on the server missing bondeth0 was replaced.

Cause

The cause is that after IB HCA card replacement on BDA both post-configuration steps below are required:

1. Run /opt/oracle/bda/network/BdaUserConfigEoib on the server to reset private and client networks.

2. Delete the old VNICs from the switches which correspond to the old HCA that was replaced.

If the first step is missed the symptoms reported above can be observed.

Solution

There are 2 solutions for post-configuration one for on-premise environments and one for BDCS environments.

1. For on-premise environment:

a) First run /opt/oracle/bda/network/BdaUserConfigEoib on the server where the card was replaced to reset private and client networks.

This will not delete the "old" VNICS because the MAC addresses have changed. However a new VNIC will be created and the client network will function with the new VNIC.

b) Delete the old VNICs from the switches which correspond to the old HCA that was replaced. Do so with the deletevinc command.

i. On the BDA server where the IB HCA card was replaced find the current MAC address. For example:

The current MAC addresses are *:*:*:*:*:x and *:*:*:*:*:y, respectively.

ii.  Log into the 2 gateway switches e.g.

iii. On each  run:

Output is like:

From <rack>-sw-ib2: 

From <rack>-sw-ib3 

There should be 2 VNICs. The one not matching with the current server MAC address for either eth8 or eth9 should be removed with the deletevnic command on the gateway switch.

iv. Based on the details in the above steps delete the VNIC returned by showvnics on the gateway switch which does not match the "new" HW address for either eth8 or eth9 on the server (as returned by ifconfig eth8 |grep HW/ifconfig eth9 |grep HW).  

Use:

For example if the output from showvnics on the switch is as below (the state of the old VNIC will be WAIT-IOA not UP):

Where the MAC address above (*:*:*:*:*:z) does not match the output returned on the server (*:*:*:*:*:x/*:*:*:*:*:y) delete the VNIC with:

Note: The plan is for post-BDA V4.6 versions to have a bdanetworktool command to delete VNICs in WAIT-IOA state.  The command will be:  "bdanetworktool none-except delete-fabric-vnics-in-wait" This will find all gateway switches on the fabric and delete such VNICs. It can also be limited to one switch by adding having="switch" argument. 

Note: Fixing the VNICs as documented above may not work when VNICs have pkeys.  But in case of pkeys being applied, no network traffic over the IB interface will be occurring after HCA replacement, so it does not really apply to the symptoms of this case.

2. For BDCS environments, open an SR with Oracle Support to get the post-configuration IB HCA card replacement steps.

The internal steps are documented at: http://adc00owq.us.oracle.com/doku.php?id=bdcs_hca_replacement_procedure

This solution has no attachment