How to Configure Ethernet Over InfiniBand on Oracle Big Data Appliance Nodes

Asset ID:	1-71-1905407.1
Update Date:	2017-10-19
Keywords:

Solution Type Technical Instruction Sure

Solution 1905407.1 : How to Configure Ethernet Over InfiniBand on Oracle Big Data Appliance Nodes

Applies to:

Big Data Appliance Hardware - Version All Versions and later
Linux x86-64

Goal

This document provides instructions on how configure Ethernet Over InfiniBand (EOIB) i.e eth8 or eth9 devices on Oracle Big Data Appliance (BDA) node/s.

Below output shows that both eth8/eth9 are not configured correctly

# service network status

Configured devices:
lo bondib0 eth0 eth1 eth2 eth3 ib0 ib1
Currently active devices:
lo eth0 ib0 ib1 bondib0

# ifup eth8

/sbin/ifup: configuration for eth8 not found.
Usage: ifup <device name>

# ifup eth9

/sbin/ifup: configuration for eth8 not found.
Usage: ifup <device name>

Also bdachecklcuster reports that

ERROR: 2 virtual NICs not available on host <BDANode>

Solution

Execute /opt/oracle/bda/network/BdaUserConfigEoib as root user on each of the BDA node/s where Ethernet Over InfiniBand(EOIB) is not configured correctly.

Note:- If all the nodes in the cluster don't have EOIB configured then execute /opt/oracle/bda/bin/bdaredoclientnet from the fist node of the cluster

# cd /opt/oracle/bda/network

# ./BdaUserConfigEoib

INFO LEAF1_NAME=10.***.17.***
INFO LEAF2_NAME=10.***.17.***
INFO SPINE_NAME=<Switch-Name>
INFO ibswitches: < list of switches ...>
check and delete vNIC for <NodeName> on switch <IBSwitch-Name>
root@10.128.17.198's password:
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use "help" at linux prompt.
check and delete vNIC for <NodeName> on switch <IBSwitch-Name>
root@10.***.17.***'s password:
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use "help" at linux prompt.
vNIC ID 1334 deleted
IO Adapter for vNIC deleted
check and delete vNIC for <NodeName> on switch <IBSwitch-Name>
root@10.128.17.198's password:
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use "help" at linux prompt.
vNIC ID 1300 deleted
IO Adapter for vNIC deleted
check and delete vNIC for <NodeName> on switch <IBSwitch-Name>
root@10.128.17.199's password:
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use "help" at linux prompt.
create vLan 0A-ETH-3 if it did not exist.
INFO check existence of default vlan for port 0A-ETH-3 on <IBSwitch-Name>
INFO use existing default vlan for port 0A-ETH-3 on <IBSwitch-Name>
create vNIC eth9 <NodeName> on switch <IBSwitch-Name>
root@10.128.17.198's password:
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use "help" at linux prompt.
vNIC created
create vLan 0A-ETH-3 if it did not exist.
INFO check existence of default vlan for port 0A-ETH-3 on <IBSwitch-Name>
INFO use existing default vlan for port 0A-ETH-3 on <IBSwitch-Name>
create vNIC eth8 <NodeName> on switch <IBSwitch-Name>
root@10.128.17.199's password:
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use "help" at linux prompt.
vNIC created
Create /etc/sysconfig/network
./BdaUserConfigEoib: line 196: udevadm: command not found
Shutting down interface bondib0: [ OK ]
Shutting down interface eth0: [ OK ]
Shutting down loopback interface: [ OK ]
Disabling IPv4 packet forwarding: net.ipv4.ip_forward = 0
[ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface bondeth0: [ OK ]
Bringing up interface bondib0: [ OK ]
Bringing up interface eth0: [ OK ]
Stopping sshd: [ OK ]
Starting sshd: [ OK ]
Shutting down ntpd: [ OK ]
Starting ntpd: [ OK ]

Check the status of the network to ensure EOIB is configured correctly

# service network status

Configured devices:
lo bondeth0 bondib0 eth0 eth1 eth2 eth3 eth8 eth9 ib0 ib1
Currently active devices:
lo eth0 ib0 ib1 bondib0 eth9 eth8 bondeth0

Attachments

This solution has no attachment