Oracle Advanced Support Gateway Server X6-2
 
Oracle Advanced Support Gateway Server X5-2
 
Oracle Server X6-2
 
Oracle Server X5-2
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 
Microlearning>Video>ML-VID-VCAP
 

Applies to:

Goal

How to Replace an Oracle Server X5-2, X6-2 Motherboard.

Solution

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?:
Oracle Server X5-2, X6-2 Training

TIME ESTIMATE: 60 minutes

TASK COMPLEXITY: 3

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

PROBLEM OVERVIEW: An Oracle Server X5-2, X6-2 Motherboard needs replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

If the system is still up and functioning, customer should perform an orderly and graceful shutdown of applications and OS.  Then power off the server and remove the AC power cords from the system.

A data backup is not a prerequisite but is a wise precaution. 

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Reference Doc:
Oracle Server X5-2 Remove the Motherboard:
http://docs.oracle.com/cd/E41059_01/html/E48312/napsm.z40017961418774.html#scrolltoc

Oracle Server X6-2 Remove the Motherboard:
http://docs.oracle.com/cd/E62159_01/html/E62171/z40017961418774.html#scrolltoc

1. Log into the ILOM check the fruid container values and sync them if needed.

1. To avoid mismatched fruid values causing a failure after a motherboard replacement the fruid data should be confirmed to have matching data in at least the Primary (DBP) and Backup2 (PS0) containers so that the motherboard will have it's container updated automatically after replacement. Go into restricted mode and use the showpsnc command to check this.  
   -> set SESSION mode=restricted
   
   WARNING: The "Restricted Shell" account is provided solely
   to allow Services to perform diagnostic tasks.
   
   [(restricted_shell) x5-2]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary           | Backup1           | Backup2
   ------------------+-------------------+-------------------+-------------------
   PPN                 33154574+1+1        33154574+1+1        33154574+1+1
   PSN                 1449NM1018          1449NM1018          1449NM1018
   Product Name        ORACLE SERVER X5-2  ORACLE SERVER X5-2  ORACLE SERVER X5-2
   [(restricted_shell) x5-2]# exit
    
2. The above example shows a system with all three containers properly in sync. If the output from the system does not show all of the containers with matching values then you should reset the SP and then re-check the values again. An ILOM reset will attempt to auto-populate the matching values if one container is out of sync.  
   -> reset /SP
   Are you sure you want to reset /SP (y/n)? y
   Performing reset on /SP
    
3. After an ILOM reset if the Primary and Backup2 containers match then proceed with the following steps to replace the motherboard. If these two containers do not match then DO NOT proceed with the replacement yet.
4. If the containers do not match you will need to use the copypsnc command from service or escalation mode to copy the data from the good container so that the Primary and Backup2 containers match (Backup1 is the MB and we are about to replace this so it is not as important at this step). If you are unfamiliar with this process and require assistance please reference the steps for using copypsnc to fix the serial number detailed in the "How to update product serial number on systems which implement TLI functionality (Doc ID 1280913.1)" and contact the TSC if needed. How to access service mode and escalation mode on ILOM 3.x and later platforms (Doc ID 1019946.1)
5. After the fruid data in the Primary and Backup2 containers have been confirmed to match proceed with the following steps.

2. Make sure to back up the ILOM and BIOS configurations before replacing the motherboard.

1. See the ILOM Administrator's Guide for Configuration and Maintenance Firmware Release 4.0.x for instructions:
   
   1. Backup the BIOS configuration https://docs.oracle.com/cd/E81115_01/html/E86149/z40001541481533.html#scrolltoc.
   2. Backup up the ILOM configuration https://docs.oracle.com/cd/E81115_01/html/E86149/z40048b81489311.html#scrolltoc.

3. Prepare the server for service.

1. Power off the server and disconnect the power cords from the power supplies.
2. Extend the server to the maintenance position in the rack.
3. Attach an anti-static wrist strap.

4. Remove the top cover and all of the Fan Modules.

1. Open the Server Fan Door. Slide the fan door latches forward and swing the door up to the open position.
2. Using your thumb and forefinger, grasp each of the fan modules in the finger recesses located in the plastic between the fans.
3. Lift the fan modules straight up and out of the chassis.
4. To open the server top cover, press and hold down the top cover release button and use the recessed area to slide the top cover toward the rear of the server about 0.5 inches (12.7 mm).
5. Lift the cover off the chassis and set it aside.

5. Remove the power supplies.

1. If the cable management arm (CMA) is installed, disconnect both CMA left-side connectors (on the PSU side) and move the CMA out of the way.
   Caution  -  When disconnecting the CMA left-side connectors, use something to support the CMA so that it does not hang down under its own weight and stress the right-side connectors; otherwise, the CMA might be damaged. You must continue to support the CMA until you have reconnected both of the left-side connectors.
2. Grasp the power supply handle and push the power supply latch to the left.
3. Pull the power supply out of the chassis.  Repeat steps b-c for the second power supply.
   Caution  -  When removing the power supplies it is important to label power supplies with the slot numbers from which they were removed (PS0, PS1). This is required because the power supplies must be reinstalled into the slots from which they were removed; otherwise, the server key identity properties (KIP) data might be lost.

6. Remove the PCIe cards and PCIe risers.

1. See Service Manual for instructions http://docs.oracle.com/cd/E41059_01/html/E48320/z40000f91037394.html#scrolltoc.
   

7. Disconnect all the cables from the motherboard.

1. To disconnect the disk backplane power cable from the motherboard, press in on the connector latch and pull the connector out.
2. To eject the disk backplane auxiliary power and signal cable connector, open both side latches.
3. To eject the FIM cable connector, open both side latches.
4. If the server has a DVD drive, do the following:
   1. Disconnect the DVD drive cable from the motherboard.
   2. To remove the DVD drive cable off of the motherboard, carefully guide it through the chassis mid-wall and place it on top of the disk cage so that it is away from the motherboard.  You do not need to disconnect the DVD drive cable from the DVD drive.
5. To remove the SAS cables and the super capacitor cable that were connected to the HBA card, carefully lift them out of the chassis and place them on top of the disk cage so that they are away from the motherboard.
6. To remove the cables that were connected to the switch card, carefully guide them through the chassis mid-wall and put them aside.

8. Remove the server mid-wall.

1. Using a screwdriver (No. 2 Phillips or flathead), loosen the four green captive screws that secure the mid-wall to the server chassis.
2. Lift up the mid-wall slightly to disengage it from the raised mushroom-shaped standoffs that are located on the server chassis sidewall (one on each end of the mid-wall), then lift it out of the server and set it aside.

9. Remove the motherboard from the server chassis.

1. Grasp the metal bracket located just to the rear of the DIMM sockets and slide the motherboard toward the front of the server and lift it slightly to disengage it from the seven mushroom-shaped standoffs that are located on the server chassis under the motherboard.
2. Lift the motherboard out of the server chassis and place it on an antistatic mat next to the replacement motherboard.

10. Remove the motherboard components.

1. Remove the air baffle from the motherboard and set it aside.
2. Remove the internal USB flash drives from the motherboard making note of the original port locations.
3. Remove the DIMMs from the motherboard.
4. Remove the processors from the failed motherboard.
   1. See Service manual for instruction to remove the processor http://docs.oracle.com/cd/E41059_01/html/E48320/z40001d31037145.html#scrolltoc. 

11. Install the motherboard components on the replacement board.

1. Remove the processor socket covers from the replacement motherboard.
   1. Disengage the processor ILM (independent loading mechanism) assembly hinge lever on the right side of the processor socket (viewing the server from the front) by pushing down on the lever and moving it to the side away from the processor, and then rotating the lever upward.
   2. Disengage the processor ILM assembly load lever on the left side of the processor socket (viewing the server from the front) by pushing down on the lever and moving it to the side away from the processor, and then rotating the lever upward.
   3. To lift the processor ILM assembly load plate off of the processor socket, rotate the ILM assembly hinge lever on the right side of the processor toward the closed position (the load plate is lifted up as the hinge lever is lowered) and carefully swing the load plate to the fully open position.
   4. Grasp the top and underside of the processor socket cover with one hand (place your thumb against the underside of the cover), place your other thumb against the underside of the cover, and carefully push the cover out of the processor ILM assembly load plate.  Be careful not to allow the processor socket cover to fall into the processor socket as this could result in damage to the socket.
   5. Repeat steps 1-4 above to remove the second processor socket cover from the replacement motherboard.
2. Install the socket covers on the bad motherboard processor sockets to protect the sockets during transport.
   1. Open one of the processor ILM assemblies on the failed motherboard.
   2. Hold the processor ILM assembly load plate open with one hand and position the processor socket cover over the top of the ILM assembly load plate so that 1) the arrow on the processor socket cover is aligned with the arrow on the top left bottom of the load plate and 2) the fasteners on one side of the cover (the fasteners are located on the underside of the cover) are inside the load plate (it does not matter which side), and use your thumb to press the other side of the processor socket cover into the load plate.  You will hear a clicking sound when the processor socket cover snaps into place.
   3. Close the processor ILM assembly load plate.
   4. Repeat Step 1 through Step 3 above to install the second processor socket cover on the failed motherboard.
3. Install the processors on the replacement motherboard.
   1. See Service Manual for instructions http://docs.oracle.com/cd/E41059_01/html/E48320/z40001d31037155.html#scrolltoc.
      
4. Install the DIMMs onto the replacement motherboard in the corresponding DIMM sockets on the replacement motherboard.  Install the DIMMs only in the sockets (connectors) that correspond to the sockets from which they were removed. Performing a one-to-one replacement of the DIMMs significantly reduces the possibility that the DIMMs will be installed in the wrong slots.
5. Install the internal USB flash drives onto the replacement motherboard.  Ensure to place in the original USB port locations.
6. Install the air baffle on the replacement motherboard.

12. Install the motherboard into the server chassis.

1. Grasp the metal bracket located to the rear of the DIMMs and tilt the front of the motherboard up slightly and push it into the opening in the rear of the server chassis.
2. Lower the motherboard into the server chassis and slide it to the rear until it engages the seven mushroom-shaped standoffs located on the server chassis under the motherboard.
3. Ensure that the indicators, controls, and connectors on the rear of the motherboard fit correctly into the rear of the server chassis.

13. Install the server mid-wall.

1. Lay the SAS cables and super capacitor cable along the left chassis sidewall (viewing the server from the front).  You will connect these cables to the internal HBA card later.
2. Position the mid-wall over the front of the motherboard so that it engages the mushroom-shaped standoffs that are located on the server chassis sidewall (one for each end of the mid-wall).
3. Ensure that SAS cables and super capacitor cable are not pinched by the mid-wall and that they run beside the mid-wall and not under it; otherwise, the cables might be damaged.
4. To secure the mid-wall to the server chassis, use a screwdriver (No. 2 Phillips or flathead) to tighten the four green captive screws.
5. If the server has a switch card, carefully guide card cables through the chassis mid-wall.  You will connect these cables to the switch card later.

14. Reconnect all the cables to the motherboard.

1. If the server has a DVD drive, carefully guide the DVD drive cable through the mid-wall and reconnect it to the motherboard.
2. To install the front indicator module (FIM) cable, push the side latches on the motherboard connector to the open position and push the FIM cable connector in.  The side latches close, locking the connector in place.
3. Reconnect the disk backplane Auxiliary power and signal cable to the motherboard.
4. Reconnect the disk backplane power cable to the motherboard.

15. Reinstall the PCIe cards and PCIe risers.

1. See Service Manual for instructions http://docs.oracle.com/cd/E41059_01/html/E48320/z40000f91037394.html#scrolltoc.
   

16. Reinstall the power supplies.

1. Align the replacement power supply with the empty power supply slot.
   Caution  -  When reinstalling power supplies, it is important to reinstall them into the slots from which they were removed during the motherboard removal procedure; otherwise, the server key identity properties (KIP) data might be lost. When a server requires service, the KIP is used by Oracle to verify that the warranty on the server has not expired.
2. Slide the power supply into the bay until it is fully seated.  You will hear an audible click when the power supply fully seats.  Repeat steps a-b for the second power supply.
3. If you disconnected the two CMA left-side connectors, reconnect the connectors.

17. Reinstall all of the Fan Modules and the top cover.

1. With the fan door open, position the replacement fan module into the server.  The fan modules are keyed to ensure that they are installed in the correct orientation.
2. Press down on the fan module and apply firm pressure to fully seat the fan module.
3. Place the top cover on the chassis.  Place the cover down so that it hangs over the rear of the server by about 13 mm (0.5 inches) and the side latches align with the slots in the sides of the chassis.  There are three latching tabs on the sides of the cover, two on the right side and one on the left side (viewing the server from the front). There is also a latch on the underside of the cover in the front left corner.
   
4. Check both sides of the chassis to ensure that the four corners of the top cover are fully down and flush with the chassis.  If the cover corners are not flush with the chassis, slide the cover towards the rear of the chassis until you can position the cover correctly.  If the top cover is not correctly positioned before you attempt to slide the cover toward the front of the chassis, the internal latch that is located on the underside of the cover might be damaged.
5. Gently slide the cover toward the front of the chassis until it locks into place (with an audible click).  As you slide the cover toward the front of the server, watch the green release button. You will hear an audible click when the green release button pops up, indicating that the cover is locked.
6. Close the server fan door.

18. Return the Server to operation.

1. Remove any anti-static measures that were used.
2. Return the server to it's normal operating position within the rack.
3. Re-install the AC power cords and any data cables that were removed.
4. Power on server. Verify that the Power/OK indicator led lights steady on.

19. Set the system serial number/fruid data if needed.

1. The motherboard is not the primary fruid container in this server so when it is replaced you should not normally need to fix the serial number information.
2. login to the ILOM as root and then enter the restricted shell to check the fruid values. Follow the example below to enter restricted shell and use the showpsnc command   
   -> set SESSION mode=restricted
   
   WARNING: The "Restricted Shell" account is provided solely
   to allow Services to perform diagnostic tasks.
   
   [(restricted_shell) x5-2:~]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary           | Backup1           | Backup2
   ------------------+-------------------+-------------------+-------------------
   PPN                 33154574+1+1        33154574+1+1        33154574+1+1
   PSN                 1449NM1018          0000000000          1449NM1018
   Product Name        ORACLE SERVER X5-2  ORACLE SERVER X5-2  ORACLE SERVER X5-2
   [(restricted_shell) x5-2:~]#
3. When the motherboard is replaced the Backup1 fruid container will likely not match the Primary entry. If it does not you must enter escalation or service mode to fix it (if all three entries match this step is done).
4. Contact the TSC to request an escalation password (service mode will work also if just the copypsnc command ends up needing to be used, if the setpsnc command is needed escalation mode is required. setpsnc is not covered in this procedure).
5. Provide your TSC contact the output from the following ILOM commands- "version", "show /SYS product_serial_number", and "show /SP/clock". If the product_serial_number information does not give good output then provide the showpsnc output that was seen in step b above as well.
6. The TSC will provide an escalation password that is made up of 32 short words. Follow the example below to create a new user with the 'Service' role assigned. The Service role is required to access service or escalation modes. In the following example we will create a user named 'escuser' with the service role.
   -> cd /SP/users
   /SP/users
   -> create escuser
   Creating user...
   Enter new password: ********
   Enter new password again: ********
   Created /SP/users/escuser
   -> set escuser role=aucros
   Set 'role' to 'aucros'
   -> show escuser
   /SP/users/escuser
   Targets:
   ssh
   Properties:
   role = aucros
   password = *****
7. Set the check_physical_presence to false and then exit from the ILOM so that you can login as the newly created user.
   -> set /SP check_physical_presence=false
   Set 'check_physical_presence' to 'false'
   -> show /SP check_physical_presence
   /SP
   Properties:
   check_physical_presence = false
   
   -> exit
8. Login using the escuser login and enter escalation mode using the password that was provided by the TSC.
   X5-2 login: escuser
   Password:
   
   Oracle(R) Integrated Lights Out Manager
   
   Version 3.2.4.34 r95732
   
   Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
   
   Warning: The system appears to be in manufacturing test mode.
   Contact Service immediately.
   
   Hostname: x5-2
   
   -> cd /SP/users/ecsuser/escalation
   -> set SESSION mode=escalation                            
   Password:**** **** **** **** **** *** *** **** **** **** **** **** **** **** **** **** *** *** **** *** **** **** **** *** **** **** *** **** *** *
   Short form password is:  NOSE HAAG MED
   
   [(escalation_mode) X5-2:~]#
9. Use the showpsnc command to confirm the current container values. Confirm that the primary container has a serial number (the value on the PSN line) that matches the system serial number. The system serial number can be checked by comparing to the serial number RFID tag on the front left hand side of the server. After confirming that there is a valid fruid primary use the copypsnc command to write the good data from the primary to the backup1 container on the MB. The following example shows copying from primary to the backup1, but you could also copy from backup2 if needed.
   
   [(escalation_mode) X5-2:~]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary           | Backup1           | Backup2
   ------------------+-------------------+-------------------+-------------------
   PPN                 33154574+1+1        33154574+1+1        33154574+1+1
   PSN                 1449NM1018          0000000000          1449NM1018
   Product Name        ORACLE SERVER X5-2  ORACLE SERVER X5-2  ORACLE SERVER X5-2
   [(escalation_mode) X5-2:~]# copypsnc Primary Backup1
   [(escalation_mode) X5-2:~]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary           | Backup1           | Backup2
   ------------------+-------------------+-------------------+-------------------
   PPN                 33154574+1+1        33154574+1+1        33154574+1+1
   PSN                 1449NM1018          1449NM1018          1449NM1018
   Product Name        ORACLE SERVER X5-2  ORACLE SERVER X5-2  ORACLE SERVER X5-2
   [(escalation_mode) X5-2:~]# exit
   
   
10. At this point if all of the fruid containers match and have the correct serial number data this step is done. If more than one of the fruid containers had non-valid entries then the copypsnc command should be used to copy over the valid data to the other container that is not valid. (ie. "copypsnc Primary Backup2" to copy primary to backup2) After confirming all fruid data is correct reset the ILOM to confirm that the fruid data persists through a reboot and remove the escalation user if needed.
    
    -> reset /SP
    Are you sure you want to reset /SP (y/n)? y
    Performing reset on /SP
    ..........
    
    ***login as the root user again and check the fruid data***
    
    -> set SESSION mode=restricted
    
    WARNING: The "Restricted Shell" account is provided solely
    to allow Services to perform diagnostic tasks.
    
    [(restricted_shell) x5-2]# showpsnc
    Primary: fruid:///SYS/DBP
    Backup 1: fruid:///SYS/MB
    Backup 2: fruid:///SYS/PS0
    
    Element           | Primary           | Backup1           | Backup2
    ------------------+-------------------+-------------------+-------------------
    PPN                 33154574+1+1        33154574+1+1        33154574+1+1
    PSN                 1449NM1018          1449NM1018          1449NM1018
    Product Name        ORACLE SERVER X5-2  ORACLE SERVER X5-2  ORACLE SERVER X5-2
    [(restricted_shell) x5-2]# exit
    
    -> cd /SP/users
    /SP/users
    -> delete escuser
    Are you sure you want to delete /SP/users/escuser (y/n)? y
    Deleted /SP/users/escuser
11. If trouble is encountered during any of the steps of accessing escalation mode and fixing the fruid containers please contact the TSC for assistance.

20. Make sure to restore the ILOM and BIOS configurations after replacing the motherboard.

1. See the ILOM Administrator's Guide for Configuration and Maintenance Firmware Release 4.0.x for instructions:
   
   1. Restore the BIOS configuration https://docs.oracle.com/cd/E81115_01/html/E86149/z40001541481533.html#scrolltoc.
   2. Restore the ILOM configuration https://docs.oracle.com/cd/E81115_01/html/E86149/z40048b81489452.html#scrolltoc.

21. The bios must now be checked to confirm it is set to "Legacy mode" and not "UEFI mode".

1. Change Boot Mode in BIOS to Legacy (Boot Mode = Legacy), as described here: http://docs.oracle.com/cd/E41059_01/html/E48312/napov.goiiw.html
   1. Enter the BIOS setup menu by pressing F2.
   2. BIOS menu -> Boot -> UEFI/BIOS Boot Mode, and select Legacy.
   3. Exit by "Save Changes and Exit"

How to verify the Motherboard is working properly.

     1.  Log into ILOM to confirm if motherboard status is working properly.

Sample

    2.  Check ILOM event log to see if any error related motherboard.

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

Boot up system and verify full functionality.

REFERENCE INFORMATION:

Oracle Server X5-2 Documentation:
http://docs.oracle.com/cd/E41059_01/index.html

Oracle Server X6-2 Documentation:
http://docs.oracle.com/cd/E62159_01/index.html

Oracle Integrated Lights Out Manager (ILOM) 3.2 Documentation:
http://docs.oracle.com/cd/E37444_01/index.html

Otube video:
https://otube.oracle.com/media/How+to+RemoveReplace+a+Motherboard+Assembly+in+an+Oracle+Server+X5-2/0_gzx0l95t

MP4:
How to remove and Replace Motherboard Assembly_(Download the file since not guaranteed to work with some browsers)

This solution has no attachment