Exadata X4-2 Hardware
 
Exadata X3-2 Quarter Rack
 
Exadata X3-2 Eighth Rack
 
Exadata X4-2 Quarter Rack
 
Exadata X4-2 Full Rack
 
Exadata X3-2 Half Rack
 
Exadata X3-2 Full Rack
 
Exadata X4-2 Half Rack
 
Zero Data Loss Recovery Appliance X4 Hardware
 
Exadata X3-2 Hardware
 
Exadata X4-2 Eighth Rack
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

Applies to:

Goal

How to Replace a 10G Ethernet Card on an Exadata Database Node (X3-2/X4-2)

Solution

CAP PROBLEM OVERVIEW: 10G Ethernet Card Replacement on an Exadata Storage Cell Database Node (X3-2/X4-2).  For Exadata Database Nodes in X3-8, refer to Note 1469193.1

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE ENGINEER NEED:
Exadata Server Training.

TIME ESTIMATE: 60 minutes

TASK COMPLEXITY: 3-FRU

FIELD ENGINEER INSTRUCTIONS

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

If the system is still up and functioning, the customer should perform an orderly and graceful shutdown of applications and OS.  Then power off the server and remove the AC power cords from the system.

For an Exadata DB node refer to the steps in:
Doc ID 1093890.1 Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

1. Prepare the server for service.

1. Power off the server and disconnect the power cords from the power supplies.
2. Extend the server to the maintenance position in the rack.
3. Attach an anti-static wrist strap.
4. Remove the top cover.

2. Perform the card replacement

Locate and Remove the PCIe card.

1. There are three external PCIe slots in the system. The external PCIe slots are numbered 1, 2, and 3 from left to right when you view the server from the rear. For the Exadata X3-2 and X4-2 systems the 10G Ethernet card is always installed in PCIe slot 2.
   
2. Locate the 10G Ethernet card in PCIe slot 2 and unplug any cables from the PCIe card making note of their locations so that they can be re-installed in the same configuration (label if needed).
3. lift the green-tabbed latch on the rear of the server's chassis next to the PCIe slot to release the PCIe card's rear bracket.
4. Lift the riser release lever with one hand and use your other hand to remove the riser from the motherboard
5. place the riser and card on an anti-static mat.
6. Remove the 10G Ethernet card from the PCIe riser. Hold the riser in one hand and use your other hand to carefully pull the PCIe card connector out of the riser.
7. Disconnect the rear bracket that is attached to the 10G Ethernet card from the rear of the PCIe riser.
8. Place the riser and the 10G Ethernet card on the antistatic mat.

Replace the PCIe card.

1. Remove the replacment 10G Ethernet card from it's anti-static bag.
2. Pull out any SFP modules installed in the original 10G Ethernet card and install them into the replacment 10G Ethernet card making sure to orient them properly so that they seat fully into the card.
3. Insert the rear bracket that is attached to the 10G Ethernet card into the PCIe riser.
4. Hold the riser in one hand and use your other hand to carefully insert the PCIe card connector into the Riser.
5. Install the PCIe riser with the installed PCIe cards into the server.
6. raise the PCIe riser release lever (marked with a green tab) to the open (up) position
7. Making sure to replace the riser into the same position from which it was removed (PCIe slot 2), gently press the riser into the motherboard connector until it seats and press the green-tabbed, riser release lever to the closed (down) position.
8. Close the green-tabbed latch on the rear of the server's chassis next to the applicable PCIe slot to secure the PCIe card's rear bracket to the server's chassis.
9. Reconnect any cables to the PCIe card that were unplugged during the removal procedure making sure to connect them in the same configuration as when they were disconnected.

3.Return the Server to operation

1. Replace the top cover
2. Remove any anti-static measures that were used.
3. Return the server to it's normal operating position within the rack.
4. Re-install the AC power cords and any additional data cables that were removed but not yet replaced.
5. Login to the ILOM and check to see if any faults were logged against the PCIe slot. If so clear the fault by logging into the fault managment shell. -  
   > start /SP/faultmgmt/shell
   Are you sure you want to start /SP/faultmgmt/shell (y/n)? y
   
   faultmgmtsp> fmadm faulty
   ...
   FRU : /SYS/MB/PCIE2
   ...
   faultmgmtsp> fmadm repaired /SYS/MB/PCIE2
   faultmgmtsp> fmadm faulty -a
   No faults found
   faultmgmtsp> exit
    
6. Power on server. Verify that the Power/OK indicator led lights steady on. Coordinate the power on of the host with the customer.

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

The system administrator should verify the system is functioning correctly. Some suggested actions they can take to verify are:
Use the ifconfig command and ensure the UP & RUNNING flags are present.
Example:

# ifconfig eth5
eth5 Link encap:Ethernet HWaddr 00:1B:21:79:8D:8D
inet addr:192.168.23.2 Bcast:192.168.23.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe79:8d8d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11234 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:699829 (683.4 KiB) TX bytes:180 (180.0 b)


Also use ethtool to ensure the link is running at the appropriate speed
For example:

# ethtool eth5
Settings for eth5:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
10000baseT/Full
Advertised auto-negotiation: Yes
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes


Can also verify network using 'ping':
# ping <IP>
Where <IP> = default router or another node on the same network

REFERENCE INFORMATION:

Sun Server X3-2 Documentation
http://docs.oracle.com/cd/E22368_01/index.html
Sun Server X4-2 Documentation
http://docs.oracle.com/cd/E36975_01/index.html

This solution has no attachment