Network Interfaces Not Operational After Motherboard Replacement on Exalogic Nodes

Asset ID:	1-72-1569243.1
Update Date:	2017-10-12
Keywords:

Solution Type Problem Resolution Sure

Solution 1569243.1 : Network Interfaces Not Operational After Motherboard Replacement on Exalogic Nodes

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.0.0.0 and later
Oracle Exalogic Elastic Cloud X2-2 Hardware - Version X2 to X6 [Release X2 to X6]
Linux x86-64
Oracle Solaris on x86-64 (64-bit)
Oracle Virtual Server x86-64

Symptoms

After system board replacement on Exalogic node, Network interfaces cannot be used. Network Interfaces responds from the local host, but fails to connect to other hosts.

Below is an example where eth0 interface cannot be used. ( Applies to eth1,eth2,eth3 - any onboard eth devices )

eth0 network interface shows status up, but does not function as expected.

Example:

(Original IP and netmask is masked by x.x.x.x and y.y.y.y in the output below)

[oracle@el01cn05 ~]$ /sbin/ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:10:E0:32:06:C4
inet addr:x.x.x.x Bcast:z.z.z.z Mask:y.y.y.y
UP BROADCAST MULTICAST MTU:1500 Metric:1 <-- eth0 is up, but is not running
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

[oracle@el01cn05 ~]$ ping x.x.x.x
PING x.x.x.x (x.x.x.x) 56(84) bytes of data. <-- ping from localhost works.
64 bytes from x.x.x.x: icmp_seq=1 ttl=64 time=0.015 ms
64 bytes from x.x.x.x: icmp_seq=2 ttl=64 time=0.028 ms
64 bytes from x.x.x.x: icmp_seq=3 ttl=64 time=0.062 ms

[oracle@el01cn05 ~]$ ping x.x.x.x
PING x.x.x.x (x.x.x.x) 56(84) bytes of data. <-- but cannot ping other node eth0 ( no external connectivity )
From x.x.x.x icmp_seq=2 Destination Host Unreachable
From x.x.x.x icmp_seq=3 Destination Host Unreachable
From x.x.x.x icmp_seq=4 Destination Host Unreachable

Changes

This issue is noticed after system board replacement in Exalogic nodes.

Cause

The issue is caused due to mismatching MAC address on the replaced system board and the old MAC address existing under the config /etc/sysconfig/network-scripts/ifcfg-eth0.

Solution

Steps to isolate the issue.

a) Login to ILOM via ssh and verify the MAC address from ILOM

-> show /SYS/MB/NET0

/SYS/MB/NET0
Targets:

Properties:
type = Network Interface
ipmi_name = MB/NET0
fru_description = 10G Ethernet Controller
fru_manufacturer = INTEL
fru_part_number = X540
fru_macaddress = 00:10:e0:32:06:c2 <----
fault_state = OK
clear_fault_action = (none)

b) Verify the MAC address in /etc/sysconfig/network-scripts/ifcfg-eth0

(Original IP and netmask is masked by x.x.x.x and y.y.y.y in the below output )

# cat ifcfg-eth0
# Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
DEVICE=eth0
HWADDR=00:10:e0:32:06:c4 <----
BOOTPROTO=static
IPV6INIT=no
ONBOOT=yes
IPADDR=x.x.x.x
NETMASK=y.y.y.y

Checking the above outputs we can see that the MAC address on the replaced system board dos not match with the MAC address on the config file.

To fix this modify the "/etc/sysconfig/network-scripts/ifcfg-eth0 and change the HWADDR= to the one seen against fru_macaddress in the ILOM output.

Example :

Modify file "/etc/sysconfig/network-scripts/ifcfg-eth0" and changed "HWADDR=00:10:e0:32:06:c4" to "HWADDR=00:10:e0:32:06:c2".

# cat ifcfg-eth0
# Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
DEVICE=eth0
HWADDR=00:10:e0:32:06:c2 <----
BOOTPROTO=static
IPV6INIT=no
ONBOOT=yes
IPADDR=x.x.x.x
NETMASK=y.y.y.y

c) Once the above change is done, reboot the node . This should resolve the issue.

d) After reboot check the eth0 IP address using

# ifconfig eth0

Confirm that the MAC changes are reflected and IP address is seen.

e) Test connectivity using

# ping IP address of eth0

f) Perform an ssh to the eth0 Ip address and confirm that the system can be accessed via ssh through eth0.

Note : For Oracle VM Server, if the MAC address is used in /etc/sysconfig/network-scripts/ifcfg-xenbr0. please update it as well.
And also the OVS UUID might need to be updated by referring to Doc ID 1531611.1.

Attachments

This solution has no attachment