Exalytics In-Memory Machine X2-4
 
Exalytics In-Memory Machine X3-4
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

Applies to:

Goal

How to Replace the Motherboard in an Oracle Exalytics X2-4 or X3-4 system

Solution

CAP PROBLEM OVERVIEW: Motherboard replacement

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE ENGINEER NEED:
Oracle Exalytics Server Training.

TIME ESTIMATE: 90 minutes

TASK COMPLEXITY: 4-FRU

FIELD ENGINEER INSTRUCTIONS

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

If the system is still up and functioning, the customer should perform an orderly and graceful shutdown of applications and OS.

A data backup is not a prerequisite but is a wise precaution.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

1.Backup the ILOM configuration.

1. As long as the system's ILOM is functional and not the reason for the motherboard replacement the ILOM configuration should be backed up so that it may be restored to the new replacement board. If the ILOM is not accessible and cannot be backed up the FE will need to re-configure the ILOM manually after the motherboard replacement.
2. To backup the settings using the ILOM web browser expand the "ILOM Administration" section in the left column and then select "Configuration Management". In the main panel you may then backup the ILOM settings.
3. To backup the settings using the ILOM cli you must use scp to copy the file to an available system on the admin network (or your laptop if you connect it directly to the ILOM net management port and configure it for the same network). Use the following commands substituting the appropriate values for the login name, password, ipaddress and file destination. The passphrase should be set to the ILOM's password.
   -> cd /SP/config
   -> set passphrase=ilompassswd
   -> set dump_uri=scp://username:password@IPaddress/tmp/SP.config

2. Prepare the server for service.

1. Power off the server and disconnect the power cords from the power supplies.
2. Extend the server to the maintenance position in the rack.
3. Attach an anti-static wrist strap.
4. Remove the top cover.

3. Remove CPUs and heatsinks from the motherboard

1. Unscrew the two Phillips screws from the heatsink. Turn the screws alternately one and one half turns until they are fully removed.
2. Twist the heatsink slightly to break the grease seal, lift off the heatsink and place it upside down on a flat surface to prevent the thermal grease from contaminating other components.
3. Use an alcohol pad to clean the thermal grease from both the bottom of the heatsink and the top of the CPU.
4. Disengage the CPU release lever by pushing down and moving it to the side away from the CPU, and then rotating the lever upward.
5. Open the load plate and carefully remove the CPU.
6. Place each of the CPUs aside on an antistatic mat.

4. Remove all memory risers

1. For each of the memory risers lift them straight up to remove them from the memory module socket.
2. set each memory riser aside on an antistatic mat. Attention to the removal order should be taken so that the modules can be replaced in the same locations from which they were removed.

5. Remove all the installed PCIe cards.

1. The server has ten PCIe slots.They are numbered 0 through 9 from left to right when you view the server from the rear (the onboard ports/connectors are located between slots 4 and 5)
2. Make a note of the slots where the PCIe cards are installed so that they can be re-installed in the same location.
3. Unplug any data cables from the PCIe cards making note of the locations so that they can be re-installed in the same configuration (label if needed).
4. Disengage the PCIe slot crossbar from it's locked position and rotate it into it's upright position.
5. Carefully remove the PCIe cards from their slots by lifting straight up from the connectors.
6. Place each of the PCIe cards aside on an antistatic mat.

Note: The Exalytics X2-4 and X3-4 systems use a very specific PCIe configuration and the cards must be populated correctly to operate properly. Care should be taken to make sure that PCIe cards are replaced back into their correct slots when any service is performed.

6. Remove the motherboard

1. Loosen the green captive screw in the corner near the fans that secures the motherboard to the chassis.
2. Disconnect all cables that connect the power supply backplane, disk drive backplane, and fan board to the motherboard.
3. Remove the air divider by sliding it out of the chassis.
4. Remove the four bus bar screws securing the motherboard to the power supply backplane.
5. Remove the power supply cover plate that secures the SAS cables.
6. Position the cables off to the side, using the tab on the power supply cover.
7. Grasp the handle on the motherboard and slide it toward the front and then out of the chassis.
8. place the board aside on an antistatic mat.

7. Install the new motherboard

1. Unpack the replacement motherboard and place it on an antistatic mat.
2. Grasp the motherboard by the handle and hold the cables off to the side while sliding the motherboard toward the back of the chassis.
3. Align the rear Locate button with its corresponding hole in the back of the chassis, and ensure that the Locate button is not jammed in the down position.
4. Tighten the captive screw in the corner near the fans that secures the motherboard to the chassis.
5. Reinsert and tighten the four bus bar screws that secure the motherboard to the power supply backplane.
6. Reinstall the air divider by sliding it into the chassis.
7. Reconnect all cables from the power supply backplane, disk drive backplane, and fan board to the motherboard.
8. Reinstall the power supply cover plate that secures the SAS cables.

8. Replace the CPUs and heatsinks on the motherboard.

1. Properly orient the CPU with the socket alignment tabs and carefully place each CPU into it's socket. Ensure that the orientation is correct as damage might result if the CPU pins are not aligned correctly.
2. Lower the load plate. Ensure that the load plate sits flat around the periphery of the CPU.
3. Engage the release lever by rotating it downward and slipping it under the catch.
4. Install the heatsinks. Use the syringe to apply thermal grease to the center of the top of each CPU
5. Orient the heatsink so that the two screws line up with the mounting studs.
6. Carefully position the heatsink on the CPU, aligning it with the mounting posts to reduce movement after it makes initial contact with the layer of thermal grease.
7. Tighten the screws alternately one-half turn until fully seated.

9. Reinstall all memory risers.

1. Align each memory riser with it's riser slot on the motherboard.
2. Push each memory riser module into it's associated CPU memory riser slot until the riser module locks in place.

10. Reinstall all PCIe cards.

1. Make sure to re-install the cards into the same locations from which they were removed.
2. For each of the cards return the PCIe card slot crossbars to their closed and locked position to secure the PCIe cards in place.
3. Reconnect any cables to the PCIe cards that were unplugged during the removal procedure making sure to connect them in the same configuration as when they were disconnected.

NOTE: There are two possible PCIe configurations for the Exalytics X2-4 and X3-4. The first is the original Exalytics X2-4 and the second is for Exalytics X3-4 and X2-4 systems that have had a Flash upgrade kit installed.

Configuration 1 (without F40 Flash cards installed)

• Slot 0  8Gb Dual-Port Fiber Channel PCI-Express card
• Slot 2  Dual-Port 10 Gigabit-Ethernet PCIe 2.0 Copper/Fiber SFP+ card
• Slot 4  SAS-2 8-port internal (2x4), 6 Gb, RAID PCIe Gen 2 host bus adapter
• Slot 6  2-port QDR InfiniBand Host Channel Adapter

Configuration 2 (X3-4 and X2-4 with the F40 Flash card upgrade installed) - note the change in location of the 10G Ethernet card from slot 2 to slot 9

• Slot 0  8Gb Dual-Port Fiber Channel PCI-Express card
• Slot 4  SAS-2 8-port internal (2x4), 6 Gb, RAID PCIe Gen 2 host bus adapter
• Slot 6  2-port QDR InfiniBand Host Channel Adapter
• Slot 9  Dual-Port 10 Gigabit-Ethernet PCIe 2.0 Copper/Fiber SFP+ card
• Slots 1,2,3,5,7,8  Flash Accelerator F40

After servicing the system the PCIe configuration should again match one of these two setups.

11. Return the Server to operation

1. Replace the top cover
2. Remove any anti-static measures that were used.
3. Return the server to it's normal operating position within the rack.
4. Re-install the AC power cords and any data cables that were removed.
5. Power on server. Verify that the Power/OK indicator led lights steady on.

12. Restore ILOM settings from the backup created in the first step.

1. To restore the settings using the ILOM web browser expand the "ILOM Administration" section in the left column and then select "Configuration Management". In the main panel you may then restore the ILOM settings.
2. To restore the settings using the ILOM cli you must use scp to copy the file from an available system on the admin network should be the same system that was used for the backup in step 1. Use the following commands substituting the appropriate values for the login name, password, ipaddress and file destination. The passphrase should be set to the ILOM's password.
   -> cd /SP/config
   -> set passphrase=ilompassswd
   -> set load_uri=scp://username:password@IPaddress/tmp/SP.config
3. If the ILOM was not able to be backed up then the FE will need to manually configure the ILOM settings. Use the commands in the example below to set the network, hostname, system identifier, timezone and date/time. Substitute the appropriate values for the customer system. The system identifier needs to have the serial number from the system chassis as shown in the example, this is located on the front left of the system. The ntp settings only need to be set if the customer is using an ntp server.
   
   -> set /SP/network pendingipaddress=192.168.1.111
   -> set /SP/network pendingipnetmask=255.255.255.0
   -> set /SP/network pendingipgateway=192.168.1.1
   -> set /SP/network pendingipdiscovery=static
   -> set /SP/network commitpending=true
   -> set /SP/network state=enabled
   -> set /SP hostname=exalytics0-sp
   -> set /SP system_identifier="Oracle Exalytics 1234MED00A"
   -> set /SP/clock timezone=EST
   -> set /SP/clock datetime=MMDDhhmmYYYY
   -> set /SP/clients/ntp/server/1 address=ntp_server_ip_address
   -> set /SP/clock usentpserver=enabled
    
   

13. Set the system serial number if needed.

1. login to the ILOM as root and then enter the restricted shell to check the psnc values. Follow the example below to enter restricted shell and use the showpsnc command-

   -> set SESSION mode=restricted
   
   WARNING: The "Restricted Shell" account is provided solely
   to allow Services to perform diagnostic tasks.
   
   [(restricted_shell) exalytics0-sp:~]$showpsnc
   Primary: fruid:///SYS/FB
   Backup 1: fruid:///SYS/DBP
   Backup 2: fruid:///SYS/MB
   
   Element           | Primary           | Backup1           | Backup2           
   ------------------+-------------------+-------------------+-------------------
   PPN                 30142678+1+1        30142678+1+1        30142678+1+1       
   PSN                 1234MED00A          1234MED00A          1234MED00A         
   Product Name        SUN FIRE X4470 M2 SERVER SUN FIRE X4470 M2 SERVER SUN FIRE X4470 M2 SERVER 
   

2. With a motherboard replacement the system should be able to automatically copy the psnc values from the primary and backup1 containers as long as they match. (if more than one part was replaced at the same time this may not happen)
3. As long as all of the psnc values match and are correct as in the above example this step is done. However if there was an issue during the replacement which caused the psnc values not to propagate automatically then this will need to be fixed manually by using the copypsnc command from service or escalation mode. If you are unfamiliar with this process and require assistance please reference the steps for using copypsnc to fix the serial number detailed in the "How to Replace the Fan Board in an Oracle Exalytics X2-4 or X3-4 system (Doc ID 1598232.1)" and contact the TSC if needed.

14. Check BIOS settings.

1. If the Exalytics system is an X3-4 version or has the F40 flash accelerator cards installed then the BIOS needs to have a non-default setting made. If the server does not have F40 cards installed then the server should have the bios defaults set. (the following setting can be applied to both system types so if in doubt make the following change)
2. Access the BIOS setup menu by pressing the F2 button when the system presents the BIOS initialization screen and prompts you. Navigate to the Exit screen and select the Load Optimal Defaults option, then navigate to the PCIPnP screen and change the PCI-E Slot 2 Option ROM: value to Disabled. To save the setting go back to the Exit screen and select Save Changes and Exit.

15. Allow the system to power up and confirm it functions properly.

1. A new Motherboard will have different MAC addresses for the onboard ethernet ports. If the customer is using these ports they should edit the appropriate Linux network configuration files (normally in /etc/sysconfig/network-scripts). Inform the customer that these MAC values have changed with the Motherboard replacement and provide them with the new values. The MAC addresses for the new Motherboard can be found from the ILOM cli in /SYS/MB/NET0 (1,2, and 3) or from the "System Information" - "Networking" screen in the ILOM's WebGUI.

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

Boot up system and verify full functionality

REFERENCE INFORMATION:

Oracle Exalytics In-Memory Machine Documentation Library

http://docs.oracle.com/cd/E41246_01/index.htm

Sun Server X2-4 Documentation

http://docs.oracle.com/cd/E20781_01/index.html

This solution has no attachment