How to Replace an Oracle Database Appliance X3-2/X4-2 Motherboard assembly

Asset ID:	1-71-1555573.1
Update Date:	2017-08-08
Keywords:

Solution Type Technical Instruction Sure

Solution 1555573.1 : How to Replace an Oracle Database Appliance X3-2/X4-2 Motherboard assembly

Applies to:

Oracle Database Appliance X4-2 - Version All Versions and later
Oracle Database Appliance X3-2 - Version All Versions and later
x86_64

Goal

How to Replace an Oracle Database Appliance X3-2/X4-2 Motherboard assembly

Solution

CAP PROBLEM OVERVIEW: MOTHERBOARD ASSEMBLY REPLACEMENT

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE ENGINEER NEED:
Oracle Database Appliance X3-2/X4-2 training.

Note: The removal/insertion CPU tool is new for the Sandy Bridge M3 product lines. If you have not used this new tool before please make yourself familiar before attempting to use on-site. The tool is not intuitive so reference the service manual before attempting this service action.

TIME ESTIMATE: 130 minutes

TASK COMPLEXITY: 3-FRU

FIELD ENGINEER INSTRUCTIONS

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

If the system is still up and functioning, customer should perform an orderly and graceful shutdown of applications and OS. Then power off the server and remove the AC power cords from the system.

Before shutting down, to obtain a backup of the bios configuration, issue this command: /usr/sbin/ubiosconfig export all -f --expert -x /tmp/bios.xml

*** The ubiosconfig command needs to be run from DOM0, not odabase. To determine which environment you're in, issue the "oakcli show env_hw" command.

***This is especially important if the customer has limited the amount of cores licensed. ***

A data backup is not a prerequisite but is a wise precaution.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

1. Prepare the server for service.

Complete the pre-install steps to backup ILOM and BIOS.

For Backup/Restore, refer to the Oracle ILOM 3.1 Configuration and Maintenance Guide

Back Up the Oracle ILOM Configuration Settings pg. 201
Back Up the BIOS Configuration pg. 220

1.Backup the ILOM configuration.

As long as the system's ILOM is functional and not the reason for the motherboard replacement the ILOM configuration should be backed up so that it may be restored to the new replacement board. If the ILOM is not accessible and cannot be backed up the FE will need to re-configure the ILOM manually after the motherboard replacement.
To backup the settings using the ILOM web browser expand the "ILOM Administration" section in the left column and then select "Configuration Management". In the main panel you may then backup the ILOM settings.
To backup the settings using the ILOM cli you must use scp to copy the file to an available system on the admin network (or your laptop if you connect it directly to the ILOM net management port and configure it for the same network). Use the following commands substituting the appropriate values for the login name, password, ipaddress and file destination. The passphrase should be set to the ILOM's password.
-> cd /SP/config
-> set passphrase=ilompassswd
-> set dump_uri=scp://username:password@IPaddress/tmp/SP.config

2.While still logged into the ILOM check the PSNC values and sync them if needed.

To avoid mismatched PSNC values causing a failure after a system board replacement the fruid data should be confirmed to have matching data in at least the Primary (DBP0) and Backup2 (PS0) containers so that the system board will have it's container updated automatically after replacement. Go into restricted mode and use the showpsnc command to check this.
-> set SESSION mode=restricted

WARNING: The "Restricted Shell" account is provided solely
to allow Services to perform diagnostic tasks.

[(restricted_shell) sunx4-2-sp:~]$showpsnc
Primary: fruid:///SYS/DBP0
Backup 1: fruid:///SYS/MB
Backup 2: fruid:///SYS/PS0

Element           | Primary           | Backup1           | Backup2
------------------+-------------------+-------------------+-------------------
PPN                 32180288+1+1      32180288+1+1        32180288+1+1
PSN                 1234NML12V          1234NML12V          1234NML12V
Product Name        SUN SERVER X4-2 SUN SERVER X4-2 SUN SERVER X4-2
[(restricted_shell) sunx4-2-sp:~]$exit
The above example shows a system with all three containers properly in sync. If the output from the system does not show all of the containers with matching values then you should reset the SP and then re-check the values again. An ILOM reset will attempt to auto-populate the matching values if one container is out of sync.
-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP
After an ILOM reset if the Primary and Backup2 containers match then proceed with the following steps to replace the system board. If these two containers do not match then DO NOT proceed with the replacement yet.
If the containers do not match you will need to use the copypsnc command from service or escalation mode to copy the data from the good container so that the Primary and Backup 2 containers match (Backup 1 is the MB and we are about to replace this so it is not as important at this step). If you are unfamiliar with this process and require assistance please reference the steps for using copypsnc to fix the serial number detailed in the "How to Replace a Sun Server X4-2 Storage Disk Backplane (Doc ID 1592273.1)" and contact the TSC if needed.
After the PSNC data in the Primary and Backup2 containers has been confirmed to match proceed with the following steps.

3. Prepare the server for service.

Power off the server and disconnect the power cords from the power supplies.
Extend the server to the maintenance position in the rack.
Attach an anti-static wrist strap.
Remove the top cover.

4. Remove all of the Fan Modules

Open the Server Fan Door. Slide the fan door latches forward and swing the door up to the open position.
Using your thumb and forefinger, grasp the fan module in the finger recesses located in the plastic between the fans.
Lift the fan module straight up and out of the chassis.

5. Remove the Power Supplies

Gain access to the rear of the server where the power supplies are located.
Release the cable management arm (CMA).If one is installed, Press and hold the CMA release tab.
Rotate the cable management arm out of the way so that you can access the power supply.
Disconnect the power cord from each power supply.
Label each power supply with the slot number from which it was removed (PS0, PS1) so that they can be re-installed to the same slots so that the FRU TLI data is not lost.
For each of the power supplies grasp the power supply handle and push the power supply latch to the left.
Pull the power supply out of the chassis.

6. Remove the PCIe risers and PCIe cards.

If any PCIe cards are installed in the risers, disconnect any external or internal cables from the cards. Note the cable locations and their order to ensure they are reinstalled in their proper locations.
For each of the PCIe risers in Slots 1 and 2 lift the green-tabbed latch on the rear of the server's chassis next to the PCIe slot to release the PCIe card's rear bracket.
Lift the riser release lever with one hand and use your other hand to remove the riser from the motherboard
place the riser on an anti-static mat and make a note of the slot in which the PCIe riser was originally installed so that it can be reinstalled into it's proper slot.
For the Slot 3 and 4 PCIe riser open the green-tabbed latch located on the rear of the server's chassis next to the PCIe slot 3 to release the PCIe card holding bracket.
To release the riser from the motherboard connector, lift the riser's green-tabbed release lever to the open position.
Slide the plastic PCIe card retainer, which is mounted on the side of the chassis, forward to release the card(s) installed in the riser
Grasp the riser with both hands and remove it from the server
Disconnect the SAS storage drive (HDD) cables from the internal HBA card installed in PCIe slot 4 Note the cable locations and their order to ensure they are reinstalled in their proper locations.
place the riser on an anti-static mat and make a note of the slot in which the PCIe riser was originally installed so that it can be reinstalled into it's proper slot.

7. Remove the Motherboard from the Chassis.

Disconnect the disk backplane power cable from the motherboard
Disconnect disk backplane LED cable from the motherboard.
Disconnect the front indicator module (FIM) cable from the motherboard.
If the server has a DVD drive, Disconnect the DVD drive cable from the motherboard and carefully lift the DVD drive cable out of the cable trough.
Carefully lift the SAS cables that you disconnected from the internal HBA out of the cable trough.
Remove the server's mid-wall by loosening the four green captive screws that secure the mid-wall to the server's chassis. Then lift the mid-wall slightly up and toward the front of the server to disengage it from the mushroom standoffs that are located on the server's chassis sidewall (one on each end of the mid-wall).
Remove the motherboard from the server's chassis by grasping the middle of the air duct in the front and back and slide the motherboard toward the front of the server then lift it slightly to disengage the six mushroom standoffs that are located on the server's chassis under the motherboard.
Lift the motherboard out of the server's chassis and place it on an anti-static mat, and next to the replacement motherboard.
Remove the following reusable components from the motherboard and install them on the replacement motherboard.
1. DDR3 DIMMs. Perform a one-to-one replacement of the DIMMs from the current motherboard to the new motherboard to reduce the possibility of the DIMMs being installed into the wrong slots. If the DIMMs are not populated correctly the server performance may suffer and some DIMMs might not be used.
2. To remove the DIMMs rotate both DIMM socket ejectors outward as far as they will go.
3. Carefully lift the DIMM straight up to remove it from the socket.
4. Install the DIMMs in the replacement Motherboard by ensuring that the ejector tabs of the DIMM slots are in the open position.
5. Then align the notch in the DIMM with the key in the connector.
6. Push the DDR3 DIMM into the connector until the ejector tabs lock the DIMM in place.

8. Remove the CPU(s) from the current Motherboard. Perform the following steps for each installed CPU.

Gently press down on the top of the heat sink to counteract the pressure of the captive spring-loaded screws that secure the heat sink to the motherboard. Loosen the four Phillips captive screws in the heat sink for the faulty CPU. Using a No. 2 Phillips screwdriver turn the screws counter clockwise alternately one and one half turns until they are fully removed.
To separate the heat sink from the top of the CPU, gently wiggle the heat sink left and right, while pulling upward. A thin layer of thermal compound separates the heat sink and the CPU. This compound also acts as an adhesive. Note –Do not allow the thermal compound to contaminate the work space or other components.
Place the heat sink upside down on a flat surface.
Use an alcohol pad to clean the thermal grease from both the bottom of the heat sink and the top of the CPU. Caution –Handle CPU socket pins with extreme care. CPU and socket pins are very fragile. A light touch can bend the CPU socket pins and damage the board beyond repair.
Disengage both CPU release levers by pushing down and moving it to the side away from the CPU, and then rotating the levers upward starting with the right lever (when viewing server from the front). Repeat with left lever.
Lift the CPU pressure frame up to the fully open position.
This platform requires a custom removal/insertion CPU tool that comes with the CPU replacement FRU. Push the removal/insertion tool button in the top center to unlock the tool. Place the removal/insertion tool on top of the CPU. Properly position the tool over the CPU socket and lower it into place over the CPU socket. Ensure that the tool and CPU key corners are aligned. To properly position the tool over the CPU socket, rotate the tool until the green triangle on the side of the tool is facing the front of the server and it is over the left side of the CPU socket when viewing the server from the front.
Click the tab on the removal/insertion tool to secure the tool to the CPU. Press the release lever on the tool to release the center button and engage the CPU. An audible click indicates that the CPU is engaged.
Lift the CPU out of the socket while it is still attached to the removal/insertion tool. Grasp the tool by its sides and remove it from the server. Turn the tool upside down and verify that it contains the CPU.
Leave the CPU in the removal/insertion tool and proceed to next step to install it into the new Motherboard.

9. Re-install the CPU(s) onto the new Motherboard. Perform the following steps for each CPU.

Ensure that the two CPU socket release levers are in the fully open position and that the CPU pressure frame is in the fully open position.
The CPU removal/insertion tool should have the CPU installed in it from the previous step after removing it from the current motherboard.
Properly position the tool over the CPU socket and lower it into place. Carefully align the CPU over the CPU socket. Ensure that the notches on the sides of the CPU align with the keys on the socket. To properly position the tool in the CPU socket, rotate the tool until the green triangle on the side of the tool is facing the front of the server and it is over the left side of the CPU socket (when viewing server from the front) and lower the tool onto the CPU socket.
Press the center button in the tool down to release the CPU so that it is placed in the CPU socket. Caution – Do not press down on the CPU. Irreparable damage to the CPU or motherboard might occur from excessive downward pressure. Do not forcibly seat the CPU into the socket. Excessive downward pressure might damage the socket pins.
Remove the processor removal/insertion tool.
Visually check the alignment. Close the pressure frame ensure that the pressure frame sits flat around the periphery of the CPU.
Engage the release levers by rotating them downward and slipping them under the catch left first and then the right one.

10.Install the CPU Heat Sink Assembly.

Use the syringe to apply approximately 0.1 ml of thermal grease to the center of the top of the CPU. Do not distribute the grease.
Inspect the heat sink for dust and lint. Clean if necessary.
Carefully position the heat sink on the CPU, aligning it with the mounting posts to reduce movement after it makes initial contact with the layer of thermal grease. Caution – When handling the heat sink, be careful not to get the thermal compound grease on your fingers so as not to transfer the grease to other server components.
Lower the heat sink onto the CPU. Caution – Avoid moving the heat sink after it has contacted the top of the CPU. Too much movement could disturb the layer of thermal compound, causing voids, and leading to ineffective heat dissipation and component damage.
Tighten the Phillips screws alternately one-half turn until they are fully seated.

11. Re-install the Motherboard into the Chassis.

grasp the middle of the air duct in the front and back and tilt the front of the motherboard up slightly and push it into the opening in the rear of the server's chassis.
Lower the motherboard into the server's chassis and slide it to the rear until it engages the six mushroom standoffs located on the server's chassis under the motherboard.
Ensue that the indicators, controls and connectors on the rear of the motherboard fit correctly into the rear of the server's chassis.
Install the server's mid-wall. Ensure that the cables (SAS and DVD cable, if applicable), are running through the opening in the center of the mid-wall.
Position the mid-wall with cables in front of the motherboard so that it engages the mushroom standoffs that are located on the server's chassis sidewall (one for each side of the mid-wall).
Place the FIM cable and the disk backplane power cable along side the mid-wall next to their respective motherboard connectors.
Ensure that FIM cable and the disk backplane power cable are not pinched by the mid-wall and that they run beside the mid-wall and not under it; otherwise, the cables might be damaged.
To secure the mid-wall to the server's chassis, tighten the four green captive screws.
Carefully place the SAS cable(s) that you disconnected from the internal HBA into the cable trough provided by the air duct on the motherboard.
If the server has a DVD drive, carefully place the DVD drive cable into the cable trough provided by the air duct on the motherboard and reconnect the DVD drive cable to the motherboard.
Reconnect the front indicator module (FIM) cable to the motherboard.
Reconnect disk backplane LED cable to the motherboard.
Reconnect the disk backplane power cable to the motherboard.

12. Re-install the PCIe risers and PCIe cards.

For each of the PCIe slot 1 and 2 risers, raise the PCIe riser release lever (marked with a green tab) to the open (up) position
Making sure to replace the riser into the same position from which it was removed gently press the riser into the motherboard connector until it seats and press the green-tabbed, riser release lever to the closed (down) position.
If there is a PCIe card installed in the riser, reconnect any external or internal cables to the card.
Close the green-tabbed latch on the rear of the server's chassis next to the applicable PCIe slot to secure the PCIe card's rear bracket to the server's chassis. If the riser does not have a PCIe card installed, install a PCIe slot filler panel and close the green-tabbed latch to secure filler panel.
For the PCIe slots 3 and 4 riser, reconnect the SAS cable(s) to the internal HBA card.Be sure to connect the SAS cable for storage drives 0 through 3 (HDDs 0-3) to the connector that is farther from the riser in which the HBA card is installed.
Raise the PCIe riser green-tabbed release lever to the open (up) position and gently press the riser into the motherboard connector until it seats.
Ensure that the rear bracket on the internal HBA card in PCIe slot 4 is connected to the slot in the server's chassis side wall. If the bracket is not connected, remove the riser and reposition it so that the rear bracket connects to the side wall, then gently press the riser into the motherboard connector.
Slide the plastic PCIe card retainer that is mounted on the side of the chassis toward the back of the server to secure the card(s) installed in the riser.
Press the green-tabbed release lever on the PCIe riser to the close (down) position.
To secure the PCIe card's rear bracket to the server, close the green-tabbed latch on the rear of the server's chassis.
If there is a PCIe card installed in slot 3 of the riser, reconnect any external or internal cables to the card. If the riser does not have a PCIe card installed in slot 3, install a PcIe filler panel and close the green-tabbed latch to secure the PCIe slot filler panel.

13. Re-install the Power Supplies and Fans

Make sure to reinstall the power supplies into the same slots from which they were removed so that the FRU TLI data is preserved and the serial number information will be auto-updated onto the new system board.
For each power supply align the power supply with the empty power supply bay, then Slide the power supply into the bay until it is fully seated.
Open the fan door and install the fan modules into the server.
Press down on the fan module and apply firm pressure to fully seat the fan module.
Close the fan door.

14. Return the Server to operation

Replace the top cover
Remove any anti-static measures that were used.
Return the server to it's normal operating position within the rack.
Re-install the AC power cords and any data cables that were removed.
Power on server. Verify that the Power/OK indicator led lights steady on.

NOTE: Please ensure that the platform or chassis serial number is correct - refer to How to update the Serial Number on Oracle x64 platforms (Doc ID 1364359.1)

** For ODA X4-2 only, make sure you move the internal usb drive back over to the new motherboard **

Since this is an Engineered System, additional steps to the fruid programming may need to be done, if it does not automatically update on it's own:

A) set the TLI in escalation mode, see example below

# setsvcid
Reading fruid:///SYS/DBP0...
Top Level Product Name ['ODA X3-2']:
Top Level PPN ['31531003+2+1']:
Top Level PSN ['1315FM0001']:

Top Level Product Name:    ODA X3-2
Top Level PPN:             31531003+2+1
Top Level PSN:             1315FM0001
Is the above correct? (y|n) [n]:

B) set the system identifier, this can be done is regular ilom command-line, see example below

-> set /SP system_identifier="Oracle Database Appliance X3-2 <TLI Serial>"

C) Backup/Restore procedure will restore clock settings. If you do not restore the ILOM configuration from a backup, it may be necessary to syncronize the clock settings manually.

To manually set the ILOM clock property values:

-> set /SP/clock datetime=MMDDhhmmYYYY
Example:
-> set /SP/clock datetime=100413002011
Set 'datetime' to '100413002011'

-> show /SP/clock
/SP/clock
Targets:
Properties:
datetime = Tue Oct 4 13:00:06 2011
timezone = GMT (GMT)
usentpserver = disabled

12. Edit the following files in the /etc/sysconfig/network-scripts directory to change the MAC address to the new MAC address. The /etc/udev/rules.d/70-persistent-net.rules file may also need to be edited, along with the /etc/sysconfig/network-scripts/ifcfg* files.

Refer to DOC ID 2041157.1 for instructions on editing the udev rules file.

** NOTE - eth#s do not match up to NET#s **

ifcfg-eth2 <<< /SYS/MB/NET0
ifcfg-eth3 <<< /SYS/MB/NET1
ifcfg-eth4 <<< /SYS/MB/NET2
ifcfg-eth5 <<< /SYS/MB/NET3

For X4-2 systems, if they are using the fiber cards for the public network, use the following coorelations:

ifcfg-eth0 <<< /SYS/MB/NET0
ifcfg-eth1 <<< /SYS/MB/NET1
ifcfg-eth2 <<< this should stay the same (fiber card port)
ifcfg-eth3 <<< this should stay the same (fiber card port)
ifcfg-eth4 <<< /SYS/MB/NET2
ifcfg-eth5 <<< /SYS/MB/NET3
ifcfg-priv1 <<< /SYS/MB/NET0

There are 2 ways to get the MAC address, from ipmitool or from the ILOM commandline.

From ipmitool:

# ipmitool sunoem cli "show /SYS/MB/NET0 fru_macaddress" << do this for NET1,NET2,NET3

From ILOM cmd:

-> show /System/Networking/Ethernet_NICs/Ethernet_NIC_0 mac_addresses << do this for NIC_1, NIC_2, NIC_3

13. Check if ILOM firmware is running at the correct level for the ODA software release:

# oakcli show version -detail
Reading the metadata. It takes a while...
System Version Component Name            Installed Version         Supported Version
-------------- ---------------           ------------------        -----------------
2.7.0.0.0
                Controller                11.05.02.00               Up-to-date
                Expander                  000F                      Up-to-date
                SSD_SHARED                9432                      Up-to-date
                HDD_LOCAL                 A31A                      No-update
                HDD_SHARED                A31A                      Up-to-date
                ILOM                      3.1.2.10 r74387           Up-to-date <<< if the installed version is not equal or greater than the supported version, it will need updating
                BIOS                      17021300                  Up-to-date

To restore bios configuration, use the following command:

# /usr/sbin/ubiosconfig import config -f --expert -y -x /tmp/bios.xml <<<< this will work if the /tm/bios.xml file was created before the motherboard replacement, as in example at beginning of this doc.

To update the ILOM version on systems running 12.1.2.5.0 and below, use the following command:

# oakcli update --patch 12.1.2.2.0 --infra <<<<<<<<<<<<<<< make sure you use the correct version, this is just an example for 12.1.2.2.0

To update the ILOM version on systems running 12.1.2.6.0, do not use oakcli, instead use the ilom web gui. Follow instructions in the ILOM Administrator's Guide for Configuration and Maintenance Guide.

To update the ILOM version on systems running 12.1.2.7.0 and above, use the following command:

# oakcli update -patch 12.1.2.7.0 --local --server <<<<<<<<<<<<<<< make sure you use the correct version, this is just an example for 12.1.2.7.0

*the oakcli update commands above will reboot the node that needs the ILOM/BIOS updated

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

Boot up system and verify full functionality

REFERENCE INFORMATION:
Oracle Database Appliance Documentation

http://docs.oracle.com/cd/E22693_01/index.htm

Attachments

This solution has no attachment