Exadata DB Machine MotherBoard Replacement Results in Network Interface Link Failure

Asset ID:	1-72-1575919.1
Update Date:	2013-11-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1575919.1 : Exadata DB Machine MotherBoard Replacement Results in Network Interface Link Failure

Applies to:

Exadata Database Machine V2 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Network Interface fails to come online.

Attempting to perform 'service netwowrk start' or 'ifconfig eth0 up' results in network interface being non accessible.

E.g.

% ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: Unknown! -------->> eth0 shows link down !!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000003 (3)
Link detected: no

Changes

MotherBoard Replacement for DB Machine.

Covers Models #

- SUN FIRE X4170 M1 (V2 Exadata)

- SUN FIRE X4170 M2 (X2-2 Exadata)

- SUN FIRE X4170 M3 (X3-2 Exadata)

Cause

Post System Board Replacement Network Interface MAC address Configuration used by the Oracle Enterprise Linux Kernel is incorrect.

Node is still using the OLD MAC address.

With the replacement System Board (motherboard) the newly replaced Network interface information stored in iLOM for the DB Machine differs from what stored in the OEL configuration files stored within the operating system.

Solution

You will need to do all this with Clusterware down on the node. Locate the GRID Infrastructure home and as the root user then run -->

% $GRID_HOME/bin/crsctl disable crs

% $GRID_HOME/bin/crsctl stop

This will stop clusterware along with all DB resources.

We will need to query the DB machine iLOM to ensure we match the actual serial number of each of the Interfaces and its simply a matter of fixing your "/etc/sysconfig/network-scripts/ifcfg-ethX" files to match.

Via Command line query the ilom using --->

% ipmitool sunoem cli "show /SYS/MB/NET0"

(for each NIC NET0/1/2/3)
Keep in mind NET0=eth0, NET1=eth1 etc etc..

E.g. Here I'm query net0 which is eth0 used as the management interface for the CISCO switch.

[root@exadbmel01 sysconfig]# ipmitool sunoem cli "show /SYS/MB/NET0"
Connected. Use ^D to exit.
-> show /SYS/MB/NET0

/SYS/MB/NET0
    Targets:

    Properties:
        type = Network Interface
        ipmi_name = MB/NET0
        fru_name = GIGABIT ETHERNET CONTROLLERS
        fru_manufacturer = INTEL
        fru_part_number = 82576EB
        fru_serial_number = 00:21:28:A5:C8:AE
        fru_extra_2 = 00:21:28:A5:C8:AE
        fault_state = OK
        clear_fault_action = (none)

From your iLOM you can see the 'fru_serial_number' value must match what is stored in /etc/sysconfig/network-scripts/ifcfg-eth0

Compare this to network configuration files under: /etc/sysconfig/network-scripts/ifcfg-ethX

where X is a 0 (ifcfg-eth0=NET0), 1 (ifcfg-eth1=NET1), 2 (NET2), 3 (NET3), or

“ifcfg-bondeth0” if a db node with bonding.

Once you have made the required changes lets restart the network interface stack.

% service network restart

Then finally startup CRS again.

% $GRID_HOME/bin/crsctl enable crs

% $GRID_HOME/bin/crsctl start

References

<NOTE:1336083.1> - How to Replace a Motherboard on a Server that is part of an Exadata Database Machine

Attachments

This solution has no attachment