networksetup-two Fails with "Ping gtw error" when LACP is Configured

Asset ID:	1-72-2279628.1
Update Date:	2017-06-30
Keywords:

Solution Type Problem Resolution Sure

Solution 2279628.1 : networksetup-two Fails with "Ping gtw error" when LACP is Configured

Applies to:

Big Data Appliance X6-2 Hardware - Version All Versions and later
Linux x86-64

Symptoms

networksetup-two fails with "Ping gtw error" when LACP is configured.

For example:

[root@bdanode01 network]# ./networksetup-two

networksetup-two: do basic sanity checks on /opt/oracle/bda/network.json
networksetup-two: passed
networksetup-two: checking for networkexpansion.json
networksetup-two: ping server ips on admin network
networksetup-two: passed
networksetup-two: test ssh server ips on admin network
networksetup-two: passed
networksetup-two: run connected network post script on each server
networksetup-two: post network setup for bdanode02-adm
networksetup-two: post network setup for bdanode03-adm
networksetup-two: post network setup for bdanode04-adm
networksetup-two: post network setup for bdanode05-adm
networksetup-two: post network setup for bdanode06-adm
networksetup-two: post network setup for bdanode07-adm
networksetup-two: post network setup for bdanode08-adm
networksetup-two: post network setup for bdanode09-adm
networksetup-two: post network setup for this node
networksetup-two: ping servers by name on admin network
networksetup-two: passed
networksetup-two: verify infiniband topology
networksetup-two: passed
networksetup-two: start setup client network (10gigE over Infiniband)
networksetup-two: ping both gtw leaf switches
networksetup-two: passed
networksetup-two: confirm switches occur in ib fabric
networksetup-two: passed
networksetup-two: verify existence of gateway ports
networksetup-two: passed
networksetup-two: removing existing eoib setup for this server
Shutting down interface bondeth0: [ OK ]
Shutting down interface bondib0: [ OK ]
Shutting down interface eth0: [ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface bondib0: Determining if ip address 192.xxx.xxx.x is already in use for device bondib0...
[ OK ]
Bringing up interface eth0: Determining if ip address 10.xxx.xxx.xx is already in use for device eth0...
[ OK ]
networksetup-two: ping server ips on admin network
networksetup-two: passed
networksetup-two: ping servers by name on admin network
networksetup-two: passed
networksetup-two: test ssh server ips on admin network
networksetup-two: passed
networksetup-two: test unique gw instance id
networksetup-two: passed
networksetup-two: copy /opt/oracle/bda/network.json to servers
networksetup-two: verify transfer of /opt/oracle/bda/network.json
networksetup-two: passed
networksetup-two: apply eoib on each server
networksetup-two: wait a few seconds for the network to restart on bdanode02-adm
networksetup-two: wait a few seconds for the network to restart on bdanode03-adm
networksetup-two: wait a few seconds for the network to restart on bdanode04-adm
networksetup-two: wait a few seconds for the network to restart on bdanode05-adm
networksetup-two: wait a few seconds for the network to restart on bdanode06-adm
networksetup-two: wait a few seconds for the network to restart on bdanode07-adm
networksetup-two: wait a few seconds for the network to restart on bdanode08-adm
networksetup-two: wait a few seconds for the network to restart on bdanode09-adm
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode02 on switch bdasw-ib2
networksetup-two: vNIC ID 9 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode02 on switch bdasw-ib3
networksetup-two: vNIC created
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode02 on switch bdasw-ib2
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode02 on switch bdasw-ib3
networksetup-two: vNIC ID 9 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode03 on switch bdasw-ib2
networksetup-two: vNIC ID 1 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode03 on switch bdasw-ib3
networksetup-two: vNIC created
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode03 on switch bdasw-ib2
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode03 on switch bdasw-ib3
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode04 on switch bdasw-ib2
networksetup-two: vNIC ID 2 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode04 on switch bdasw-ib3
networksetup-two: vNIC created
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode04 on switch bdasw-ib2
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode04 on switch bdasw-ib3
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode05 on switch bdasw-ib2
networksetup-two: vNIC ID 3 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode05 on switch bdasw-ib3
networksetup-two: vNIC created
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode05 on switch bdasw-ib2
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode05 on switch bdasw-ib3
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode06 on switch bdasw-ib2
networksetup-two: vNIC ID 4 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode06 on switch bdasw-ib3
networksetup-two: vNIC created
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode06 on switch bdasw-ib2
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode06 on switch bdasw-ib3
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode07 on switch bdasw-ib2
networksetup-two: vNIC ID 5 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode07 on switch bdasw-ib3
networksetup-two: vNIC created
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode07 on switch bdasw-ib2
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode07 on switch bdasw-ib3
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode08 on switch bdasw-ib2
networksetup-two: vNIC ID 6 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode08 on switch bdasw-ib3
networksetup-two: vNIC created
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode08 on switch bdasw-ib2
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode08 on switch bdasw-ib3
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode09 on switch bdasw-ib2
networksetup-two: vNIC ID 10 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode09 on switch bdasw-ib3
networksetup-two: vNIC created
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode09 on switch bdasw-ib2
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode09 on switch bdasw-ib3
networksetup-two: vNIC ID 10 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: vNIC created
networksetup-two: vNIC ID 1 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: vNIC created
networksetup-two: vNIC ID 2 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: vNIC created
networksetup-two: vNIC ID 3 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: vNIC created
networksetup-two: vNIC ID 4 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: vNIC created
networksetup-two: vNIC ID 5 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: vNIC created
networksetup-two: vNIC ID 6 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: vNIC created
networksetup-two: vNIC created
networksetup-two: apply eoib on this node
networksetup-two: network will restart momentarily
RTNETLINK answers: File exists
RTNETLINK answers: No such file or directory
RTNETLINK answers: No such file or directory
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode01 on switch bdasw-ib2
networksetup-two: vNIC ID 11 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode01 on switch bdasw-ib3
networksetup-two: vNIC created
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode01 on switch bdasw-ib2
networksetup-two: bdanetworktool.delete_vnic_on_switch: check and delete vNIC for bdanode01 on switch bdasw-ib3
networksetup-two: vNIC ID 11 deleted
networksetup-two: IO Adapter for vNIC deleted
networksetup-two: vNIC created
networksetup-two: test client network (eoib) resolve and reverse resolve
networksetup-two: passed
networksetup-two: test client name array matches ip array
networksetup-two: passed
networksetup-two: ping servers on client network by ip
networksetup-two: passed
networksetup-two: ssh into servers on client network by name
networksetup-two: passed
networksetup-two: test ntp servers
networksetup-two: passed
networksetup-two: ping client gateway
networksetup-two: passed
networksetup-two: test arp -a
networksetup-two: passed
networksetup-two: test vnics for all nodes

host if status actv primary switch gw port ping gw vlan
============================= === ====== ==== ======= ====================== ========= ======= ======
bdanode01 eth8 up no no bdasw-ib2 0A-ETH-1 yes N/A
bdanode01 eth9 up yes yes bdasw-ib3 0A-ETH-1 yes N/A
bdanode02 eth8 up no yes bdasw-ib2 0A-ETH-2 yes N/A
bdanode02 eth9 up no no bdasw-ib3 0A-ETH-2 yes N/A
bdanode03 eth8 up no no bdasw-ib2 0A-ETH-2 yes N/A
bdanode03 eth9 up yes yes bdasw-ib3 0A-ETH-2 yes N/A
bdanode04 eth8 up yes yes bdasw-ib2 0A-ETH-3 yes N/A
bdanode04 eth9 up no no bdasw-ib3 0A-ETH-3 yes N/A
bdanode05 eth8 up no no bdasw-ib2 0A-ETH-3 no N/A
bdanode05 eth9 up yes yes bdasw-ib3 0A-ETH-3 yes N/A
bdanode06 eth8 up no yes bdasw-ib2 0A-ETH-4 yes N/A
bdanode06 eth9 up no no bdasw-ib3 0A-ETH-4 yes N/A
bdanode07 eth8 up no no bdasw-ib2 0A-ETH-4 yes N/A
bdanode07 eth9 up yes yes bdasw-ib3 0A-ETH-4 yes N/A
bdanode08 eth8 up yes yes bdasw-ib2 0A-ETH-1 yes N/A
bdanode08 eth9 up no no bdasw-ib3 0A-ETH-1 yes N/A
bdanode09 eth8 up no no bdasw-ib2 0A-ETH-1 no N/A
bdanode09 eth9 up yes yes bdasw-ib3 0A-ETH-1 yes N/A
1Ping gtw error on host bdanode05, interface eth8, switch bdasw-ib2, port 0A-ETH-3
1Ping gtw error on host bdanode09, interface eth8, switch bdasw-ib2, port 0A-ETH-1
networksetup-two: end setup client network

Cause

The cause is that LAG/LACP is not supported in BDA rpm package versions v4.5.0-<x> which are less than BDA rpm package version v4.5.0-6.

BDA rpm package version v4.5.0-6 is the first version to support LAG/LACP.

Hence running networksetup-two on v4.5.0-<x> BDA rpm package versions lower than v4.5.0-6 will not create the required LAG connector. This will cause LAG/LACP to not be configured and cause errors accessing the network.

Solution

1. Apply the bda 4.5.0-6 patch from here: Network Configuration Instructions for Shipped BDA Racks with a BDA Base Image Less Than V4.5.0 (Doc ID 2135358.1). From the document click on the patch and download p23241894_450_Linux-x86-64.zip. Follow the steps to apply the patch from the document. After applying the patch, "rpm -q bda" should show 4.5.0-6.

2. Verify /opt/oracle/bda/network.json with json-select. "ok" should be returned.

# json-select /opt/oracle/bda/network.json

3. Erase all VNICs and VLANs on both gateways switches.

3.1. ssh to one of the switches.

a) Delete all the VNICs:

i. Run showvnics:

# ssh root@bdasw-b2

[root@bdasw-ib2 ~]# showvnics

ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW
--- -------- --- ----------------------- -------------------------------- ---- ----------------- --- ------ --------
30 UP N 0010E0000132F989 bdanode02 BDA 1xx.xxx.xx.xx H 0000 FA:89:0A:80:04:08 NO 0xffff 0A-ETH-1
31 UP N 0010E0000132F3B9 bdanode01 BDA 1xx.xxx.xx.xx H 0000 F2:B9:0A:80:04:07 NO 0xffff 0A-ETH-1
32 UP N 0010E0000132FAC9 bdanode03 BDA 1xx.xxx.xx.xx H 0000 FA:C9:0A:80:04:09 NO 0xffff 0A-ETH-1
63 UP N 0010E0000132FE69 bdanode04 BDA 1xx.xxx.xx.xx H 0000 FE:69:0A:80:04:0A NO 0xffff 0A-ETH-3

ii. Delete all the VNICs with the information shown in showvnics.

Run the command "deletevnice" as: deletevnic <connector> <VNIC_Id>.

For example:

[root@bdasw-ib2 ~]# deletevnic 0A-ETH-1 30

vNIC ID 30 deleted
IO Adapter for vNIC deleted

iii. Repeat the same for the rest of the VNICs.

b) Delete all the VLANs.

i. Run showvlan from the switch:

[root@bdasw-ib2 ~]# showvlan

Connector/LAG VLN PKEY
------------- --- ------
0A-ETH-1 0 0xffff
0A-ETH-3 0 0xffff
1A-ETH-1 0 0xffff
0A-LAG-2 0 0xffff

ii. Then delete the VLANs:

[root@bdasw-ib2 ~]# deletevlan 0A-ETH-1 -vlan 0

iii. Perform the same steps for the rest of the VLANs.

3.2. Repeat the entire set of steps for the second gateway switch to delete all VNICs and VLANs there.

4. From Node 1 run /opt/oracle/bda/network/BdaUserConfigEoib and collect the output so it can be verified to be correct.

# /opt/oracle/bda/network/BdaUserConfigEoib

Note: If needed open an SR with Oracle Support to verify the output.

5. If everything works, the VNIC created should be related to LAG-01. At this point rerun networksetup-two.

# networksetup-two

References

<NOTE:2135358.1> - Network Configuration Instructions for Shipped BDA Racks with a BDA Base Image Less Than V4.5.0

Attachments

This solution has no attachment