Exadata Database Machine X2-8
 
Sun Server X2-8
 
Exadata X3-8 Hardware
 
Sun Fire X4800 Server
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

In this Document

Goal

Solution

References

Applies to:

Goal

How to Replace a Sun Server X2-8 (Sun Fire X4800 M2), Sun Fire X4800, Exadata X2-8, X3-8 NEM (Network Express Module).

Solution

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED:
No special skills required, Customer Replaceable Unit (CRU) procedure

TIME ESTIMATE: 30 minutes

TASK COMPLEXITY: 0

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

PROBLEM OVERVIEW: A Sun Server X2-8 (Sun Fire X4800 M2), Sun Fire X4800, Exadata X2-8, X3-8 NEM (Network Express Module) needs replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

The Network Express Module can be replaced with system in powered on running state.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Reference Doc:
Sun Fire X4800 Server Service Manual
http://docs.oracle.com/cd/E19140-01/html/821-0282/index.html

Sun Fire X4800 M2 Server Service Manual
http://docs.oracle.com/cd/E20815_01/html/E20840/sfmsm.html#scrolltoc

Network Express Module Designations
http://docs.oracle.com/cd/E20815_01/html/E20840/sfmsm.gjfsa.html#scrolltoc
The server contains two Network Express Modules. The Network Express Modules are designated NEM0, and NEM1, with NEM0 residing in the left slot and NEM1 residing in the right slot.

How to shutdown the Exadata database nodes and storage cells in a rolling fashion so certain hardware tasks can be performed. (Doc ID 1539451.1)


How to Remove a Network Express Module

Network Express Modules (NEM) are supposed to be hot-pluggable, but depending on the OS installed (Linux) can cause the host to reset. That is, the server does not need to be powered off to remove or replace a NEM as long as IO resources accessing the NEM are disabled in the OS.  However, if the system is running Linux, like in the case of Exadata DB nodes, the system must be put into standby power mode as hot-plugging a NEM causes a reset of the HOST.

1. Locate the NEM that you need to remove.
If the NEM is in a fault state, the Amber Service Action Required LED is on.

2. Label and disconnect any cables attached to the NEM.

3. To release the NEM, squeeze and extend the release handles outward in opposite directions.

4. Pull the lower handle downward and lift the upper handle until both are in their fully open position.
This action removes the pawl from the locking slot.

5. Use the handles to pull the unit partially out of the slot.
Pull the unit out far enough to grab it with your hands.

6. Close the release handles by moving them inward again.

7. To remove the NEM, slowly slide it out of the slot with one hand while supporting it with the other hand.

How to Install a Network Express Module

1. If necessary remove the existing NEM or NEM filler panel.

2. Ensure that the release handles on the replacement NEM are in the fully open position.
Extend the release handles outward in opposite directions to their fully open position. Pull the lower handle downward and lift the upper handle.

3. Position the NEM at the opening with the LEDs to the bottom and the release handles on the right hand side.
Correct positioning ensures that the connectors on the NEM is aligned with the connectors on the internal midplane of the server.

4. To install the NEM, use the handles to slide the NEM into the opening until it stops.

Note - In this position, the NEM is not flush with the face of the server, and the connectors on the NEM are not engaged with the connectors on the internal midplane. Do not attempt to force the NEM into the server any farther.

5. Ensure that the pawls on the levers are aligned with the slot in the side wall.

6. To engage and set the NEM connectors with the connectors on the internal midplane, push the end of the release handles inward toward the NEM.

 This action draws the NEM into the slot and engages the connectors on the NEM with the connectors on the internal midplane and the NEM face will be flush with the rear of the server.

7. Continue to push the release handles until the handles lock into place on the right side edge of the NEM.  In this position, the NEM cannot be slid out of the server unless the release handles are unlocked and lifted first.

8. If necessary, install or transfer from removed NEM any 10Gbe transceivers.

9. Attach the necessary cables to the NEM.

10. Once installed the replacement NEM should be checked for firmware level to ensure it matches the other existing NEM installed.  If the replacement NEM is down revved or at a different firmware revision than the other NEM, they must be upgraded to the same level for proper operation.

Note - For component firmware update instructions, refer to the Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for the Sun Server X2-8 (formerly Sun Fire X4800 M2).

11. If the SAS expander is down revved and needs to be updated then power on the server and boot into BIOS so that the NEM is powered on but there is no traffic on the NEMs at all.

Note – You do not need to configure anything in the BIOS setup utility. This action ensures that the NEMs are powered-on but the OS does not boot.

12. In the ILOM BUI go to the Components tab and highlight the NEM needing upgrade by clicking the radio button on the left.

13. From the Actions Drop-down menu, select Update Firmware.  A window will appear asking for download details.

14. Perform the following Actions:

    A. Specify the image location by performing:

       1. Click Browse to select the location of the firmware image you want to install (Example: x4800-sas2exp-5.7.0.0-rom.pkg).

    B. Select a transfer method from the drop-down list.

    C. Click the Update button to upload the file and update the firmware.
        The Update Status display appears, providing details about the update progress. When the update indicates 100%, the firmware upload is complete.  When the update is finished, it displays the message Firmware Update Successful.  Auto-closing windows in 2 seconds, then it closes the display.

15. Repeat above steps 10 through 13 for second NEM if needed.

16. Press ESC in the BIOS setup utility to boot the system.

If you find the SAS expander firmware is down revved below 5.7.0.0 then you should also assume the FPGA is down revved as well.  The NEM FPGA v17 and SAS expander 5.7.0.0 were implemented at the same time in manufacturing so if SAS is down revved then so is FPGA.  You need to acquire an escalation password to access escalation mode.

FPGA v17 comes in SW1.3 release for X4800 and SW1.1 for X4800M2.

Exadata X2-8 images with the proper ILOM containing v17 FPGA are:

image 11.2.2.4.0 contains SW1.3 for X4800

image 11.2.3.1.0 contains SW1.1 for X4800M2

Exadata X3-8 released with correct image at RR so all images contain v17 FPGA.

It's a manual process to update the NEM FPGA:

1. First upgrade the ILOM/BIOS with at least SW1.3 for X4800 or SW1.1 for X4800M2 or later using normal firmware update process.  

2. Then boot the server into BIOS so that the server/NEM is powered on but there is no traffic on the NEMs at all.  The server must be powered on to update the NEM FPGA, it cannot be done with server powered off.

3. Then login with escalation mode to the CMM.  

4. After logging in use the below command to update the NEM FPGA and the PHYROM.  The script updates both automatically.

# /usr/local/bin/g5nemfpgaupdate.sh

No need to AC power cycle the machine.  (As host is turned off at the end of upgrade script, NEM is powered by chassis main 12V, new FPGA/PHYROM will be effective next time host is powered on.)

How to verify the Network Express Module is working properly

Login to CMM to Check the NEM status.

-> show /SYS/NEMx -l all

Note: The "x" represents the NEM number replaced

Sample:

Check ILOM event log to see if any error existed.

-> show /SP/faultmgmt

Example:

-> show /SP/logs/event/list

Check OS FMA info to see if any errors existed.

Example:

Clear any faults found.

References

This solution has no attachment