How to Replace a SPARC T8-2 Motherboard [VCAP]

Asset ID:	1-71-2291116.1
Update Date:	2018-03-16
Keywords:

Solution Type Technical Instruction Sure

Solution 2291116.1 : How to Replace a SPARC T8-2 Motherboard [VCAP]

Applies to:

SPARC T8-2 - Version All Versions and later
Information in this document applies to any platform.

Goal

How to Replace a SPARC T8-2 Motherboard

Solution

**************************************************************************************
To report errors or request improvements on this procedure, please add a comment on Doc ID: 2291116.1
**************************************************************************************

ESD Caution:

Circuit boards and drives contain electronic components that are extremely sensitive to static electricity. Ordinary amounts of static electricity from clothing or the work environment can destroy the components located on these boards. Do not touch the components along their connector edges.
Use a Antistatic Wrist strap. Attach one end of the strap to your wrist and the other end to the chassis, depending on what type of strap you use, with the adhesive end or the metal plug.
Use an Antistatic Mat. Place ESD-sensitive components such as motherboards, memory, and other PCBs on an antistatic mat

Contamination Caution:

Dust particles of packaging material are number one cause of datacenter contamination. Make sure to remove all packaging material, up to the ESD safe packaging material, while still being outside the datacenter.

DISPATCH INSTRUCTIONS

WHAT SKILLS ARE REQUIRED?: SPARC T8-2 Product Training required, to be serviced by qualified Oracle Service personnel, requires the ability to follow steps similar to what is in the Product Service manual.

TIME ESTIMATE: 90 minutes

TASK COMPLEXITY: 1

REMOVAL/REPLACEMENT INSTRUCTIONS:

PROBLEM OVERVIEW: SPARC T8-2 Motherboard Replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

Note:

A data backup is not a prerequisite but is a wise precaution.
If the system is still up and functioning, the Customer should perform an orderly and graceful shutdown of the applications and operating system to get the OpenBoot PROM prompt. Run the printenv command and make a note of any OpenBoot PROM variables that have been modified.
The LDOM configuration (if any) needs to be saved before motherboard replacement to avoid loss of LDOM configuration, refer to doc 1019720.1 for details.
Then power off the server and remove the AC power cords from the system. For ALL scenarios where an AC power down or AC power cycle is required for a T8-x server, please always use the steps in doc 1571054.1 prior to physically removing AC power cables from the server.

WHAT ACTIONS ARE REQUIRED?:

Damage Alert:

Perform a visual inspection of the replacement part to make sure that there are no damaged components, connectors, bent pins, damaged packages during shipping, etc. If the part is damaged, don't install it into the system, order a new part. Handle with caution and package carefully the return FRU just as the new FRU was packaged, to avoid any damages during shipping.

NOTE:

Until Further notice this T8 Part must be submitted through the CPAS process. The FE may not always see the CPAS note, so please make sure you alert the FE to add it in the task too. Once you have the CPAS# please add it into the SR notes for all to see.
See this link for more information about the CPAS process: https://stbeehive.oracle.com/teamcollab/overview/CPAS
Further details: Mandatory NCAT/CPAS for Specific SPARC T8 Series Servers FRU's/CRU's (Doc ID 2297742.1)

Replace the Motherboard

1. Log into the ILOM and check the fruid container values and sync them if needed.

a. To avoid mismatched fruid values causing a failure after a motherboard replacement the fruid data should be confirmed to have matching data in at least the Primary (DBP) and Backup1 (SPM) containers so that the motherboard will have it's container updated automatically after replacement. Go into restricted mode and use the showpsnc command to check this.

-> set SESSION mode=restricted

WARNING: The "Restricted Shell" account is provided solely
to allow Services to perform diagnostic tasks.

[(restricted_shell) t8-2-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///persist/psnc_backup1.xml
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
----------------+----------------------+-----------------------+-------------------
PPN 35133190+1+1 35133190+1+1 35133190+1+1
PSN 1735NN80U2 1735NN80U2 1735NN80U2
MACADDR 00:10:E0:D5:7D:3A 00:10:E0:D5:7D:3A 00:10:E0:D5:7D:3A
HOSTID 86d57d3a 86d57d3a 86d57d3a
Product Name SPARC T8-2 SPARC T8-2 SPARC T8-2
RFID SN 341A583DE50000000009F58B 341A583DE50000000009F58B 341A583DE50000000009F58B
[(restricted_shell) t8-2-bur09-a-sp:~]#

b. The above example shows a system with all three containers properly in sync. If the output from the system does not show all of the containers with matching values then you should reset the SP and then re-check the values again. An ILOM reset will attempt to auto-populate the matching values if one container is out of sync.

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP

c. After an ILOM reset if the Primary and Backup1 containers match then proceed with the following steps to replace the motherboard. If these two containers do not match then DO NOT proceed with the replacement yet.
d. If the containers do not match you will need to use the copypsnc command from service or escalation mode to copy the data from the good container so that the Primary and Backup1 containers match (Backup2 is the motherboard and we are about to replace this so it is not as important at this step).

If you are unfamiliar with this process and require assistance please reference the steps for using copypsnc to fix the serial number detailed in the "How to update product serial number on systems which implement TLI functionality (Doc ID 1280913.1)" and contact the TSC if needed. How to access service mode and escalation mode on ILOM 3.x and later platforms (Doc ID 1019946.1).

e. After the fruid data in the Primary and Backup1 containers have been confirmed to match proceed with the following steps.

2. Prepare for servicing:

a. Attach an antistatic wrist strap.
b. Power off the server and unplug power cords from the power supplies.
c. Extend the server to maintenance position.
d. Remove the top cover.

Caution:

Components inside the chassis might be hot. Use caution when servicing components inside the chassis.

Note:

When replacing the motherboard, you will need to remove the SPM and SCC PROM from the old motherboard and install these components on the new motherboard. The SP contains the Oracle ILOM system configuration data, and the SCC PROM contains the system host ID and MAC address. Transferring these components preserves the system-specific information stored on these modules. Whenever you replace the motherboard or the SPM, you must update the firmware so the portions of firmware in the SPM and on the motherboard are consistent.

3. Remove all PCIe cards.

Note:

Always remove transceivers from a PCIe card(s) before removing the card from the server.
Keep track of which slot each PCIe card was in so you can return them to their original positions.

4. Remove the SP (Service Processor) and SCC PROM from the motherboard so you can reinstall it on the new motherboard.

5. Remove all eight memory risers.

Note:

The eUSB does not need to be swapped over from the old motherboard to the new motherboard.

6. Remove the System Remind button assembly (air divider) by lifting it up and away from the power supplies.

7. Disconnect all cables connected to the motherboard by completing the following tasks:

Caution:

Some of these data cables are delicate. Please use caution while disconnecting them and ensure they are safely out of the way when removing and installing the motherboard.

a. Disconnect two longer cables that connect the motherboard to the hard disk drive backplane (Push down a metal tab on each connector and pull up).
b. Disconnect three shorter cables from the motherboard (Two cables go to the drive backplane. The other is a ribbon cable to the power supply).
c. Disconnect the fan board power cable and the ribbon cable from the motherboard.

8. Remove the power supply cover (Remove screw from side of the power supply and ribbon cable to make removing the cover easier. There are two slots on the power supply backplane cover that you must guide around pins on the inside of the power supply cage).

a. Lift the cover up a little to clear the first part of the slots.
    b. Push the cover a little towards the front of the chassis.
    c. Push the tooth at the bottom of the cover to clear the edge of the power supply cage.
    d. Lift the cover out of the chassis (Notice the two cables that are now exposed. Be prepared to move those cables out of the way when you lift the motherboard).

9. Remove the four bus bar screws securing the motherboard to the power supply backplane.

10. Position the HDD end of the cables off to the side using the tab on the top of the plastic power supply cover.

11. Remove the motherboard by completing the following tasks:
a. Loosen the captive screw in the corner near the fans that secures the motherboard to the chassis.
    b. Grasp the handle on the motherboard and slide it toward the front of the chassis (Tilt up the end of the motherboard that is near the front of the chassis).
    c. Lift the motherboard out of the chassis (Ensure that remaining cables do not get caught on edges of the motherboard).

12. Unpack the replacement motherboard and place it on an antistatic mat.

13. On the replacement motherboard, install the service processor (SP) and System Configuration PROM that you removed from the old motherboard.

14. Grasping the motherboard by the handle, place it into the chassis (Ensure that remaining cables do not get caught on edges of the motherboard. Set the motherboard towards the front of the chassis, then slide it toward the rear of the chassis).

15. Tighten the captive screw (in the corner near the fans) that secures the motherboard to the chassis.

16. Reinsert and tighten the four bus bar screws that secure the motherboard to the power supply backplane.

Note:

Using a No. 2 screwdriver, tighten the bus bar screws until the power supply backplane and the motherboard securely fasten to the bus bars.

17. Replace the power supply backplane cover.

a. Align the power supply backplane cover (Ensure that the tooth at the bottom of the cover is clear of the power supply cage. There are two slots on the power supply backplane cover that you must guide around pins on the inside of the power supply cage).
    b. Fit the two slots on the cover around the two pins.
    c. Lift up the cover a little to guide the two pins into the other part of the slots.
    d. Attach the screw to fasten the power supply backplane cover in place.

18. Push the power supplies back into place.

19. Reattach all cables to the motherboard.

Caution:

Some of these data cables are delicate. Please use caution while connecting them and ensure they are safely out of the way when removing and installing the motherboard.

a. In the center rear of the motherboard, connect the fan board power cable and the ribbon cable to the motherboard.
b. Near the drives, connect two shorter cables to the motherboard. One cable goes to the drive backplane. The other is a ribbon cable to the power supply.
c. Near the drives connect two longer cables between the motherboard and the drive backplane.

20. Reinstall the System Remind button assembly (air divider) by sliding it into the chassis.

Caution:

After replacing the motherboard, inspect the dividing wall gasket, and then install the plastic dividing wall securely. This dividing wall maintains a pressurized seal between the server cooling zones. Without this pressurized seal, the power supply fans will not be able to draw enough air to cool the drives properly.

21. Reinstall all eight memory risers.

22. Reconnect all cables from the power supply backplane, drive backplane, and fan board to their original locations on the motherboard.

23. Reinstall all PCIe cards.

Note:

Always install transceivers for a PCIe card(s) after installing the card(s) in the server.
Remember to return each PCIe card to their original positions.

24. Install the top cover.

25. Return the server to the normal rack position.

26. Reinstall the power cords to the power supplies.

27. Prior to powering on the server, connect a terminal or a terminal emulator (PC or workstation) to the service processor SER MGT port.

If the service processor detects the host firmware on the replacement motherboard is not compatible with the existing service processor firmware, further action will be suspended and the following message will be displayed:

Unrecognized Chassis: This module is installed in an unknown or unsupported chassis. You must upgrade the firmware to a newer version that supports this chassis.

If you see the preceding message, continue to Step 28. Otherwise, skip to Step 29.

Note:

Whenever you replace the SP or the motherboard, update the firmware on the server so the portions of firmware in the two components remain consistent.

28. Download the system firmware.

a. If needed, configure the SP network port to enable the firmware image to be downloaded. Refer to the Oracle ILOM documentation for network configuration instructions.
b. Download the system firmware. Follow the firmware download instructions in the Oracle ILOM documentation.

Note:

You can load any supported system firmware version, including the firmware revision that had been installed prior to the replacement of the motherboard.
When replacing a motherboard on T3, T4, T5, T7 and T8 systems, certain procedures will need to be followed if on board hardware raid volumes are configured. Re-activate any RAID volumes that existed prior to replacing the motherboard. Only perform this task if your system had RAID volumes prior to replacing the motherboard. For details please consult doc 1387771.1.
The LDOM configuration (if any) needs to be restored after motherboard replacement to avoid loss of LDOM configuration, refer to doc 1019720.1 for details.

29. Power on server. Verify that the Power/OK indicator led lights steady on.

30. Set the system serial number/fruid data if needed.

a. The motherboard is not the primary fruid container in this server so when it is replaced you should not normally need to fix the serial number information (TLI).
b. login to the ILOM as root and then enter the "restricted shell" to check the fruid values. Follow the example below to enter restricted shell and use the showpsnc command:

-> set SESSION mode=restricted

WARNING: The "Restricted Shell" account is provided solely
to allow Services to perform diagnostic tasks.

[(restricted_shell) t8-2-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///persist/psnc_backup1.xml
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
------------+----------------------+-----------------------+-------------------
PPN 35133190+1+1 35133190+1+1 35133190+1+1
PSN 1735NN80U2 1735NN80U2 1735NN80U2
MACADDR 00:10:E0:D5:7D:3A 00:10:E0:D5:7D:3A 00:10:E0:D5:7D:3A
HOSTID 86d57d3a 86d57d3a 86d57d3a
Product Name SPARC T8-2 SPARC T8-2 SPARC T8-2
RFID SN 341A583DE50000000009F58B 341A583DE50000000009F58B 341A583DE50000000009F58B
[(restricted_shell) t8-2-bur09-a-sp:~]#

c. When the motherboard is replaced the Backup2 fruid container will likely not match the Primary entry. If it does not you must enter escalation or service mode to fix it (if all three entries match this step is done).
d. Contact the TSC to request an escalation password (service mode will work also if just the copypsnc command ends up needing to be used, if the setpsnc command is needed escalation mode is required. setpsnc is not covered in this procedure).
e. Provide your TSC contact the output from the following ILOM commands- "version", "show /SYS product_serial_number", and "show /SP/clock". If the product_serial_number information does not give good output then provide the showpsnc output that was seen in step b above as well.
f. The TSC will provide an escalation password that is made up of 32 short words. Follow the example below to create a new user with the 'Service' role assigned. The Service role is required to access service or escalation modes. In the following example we will create a user named 'escuser' with the service role.

-> cd /SP/users
/SP/users
-> create escuser
Creating user...
Enter new password: ********
Enter new password again: ********
Created /SP/users/escuser
-> set escuser role=aucros
Set 'role' to 'aucros'
-> show escuser
/SP/users/escuser
Targets:
ssh
Properties:
role = aucros
password = *****

g. Set the check_physical_presence to false and then exit from the ILOM so that you can login as the newly created user.

-> set /SP check_physical_presence=false
Set 'check_physical_presence' to 'false'
-> show /SP check_physical_presence
/SP
Properties:
check_physical_presence = false

-> exit

h. Login using the escuser login and enter escalation mode using the password that was provided by the TSC.

t8-2-bur09-a-sp login: escuser
Password:

Oracle(R) Integrated Lights Out Manager

Version 4.0.1.2 r121112

Warning: The system appears to be in manufacturing test mode.
Contact Service immediately.

Hostname: t8-2-bur09-a-sp

-> cd /SP/users/escuser/escalation
-> set SESSION mode=escalation
Password:**** **** **** **** **** *** *** **** **** **** **** **** **** **** **** **** *** *** **** *** **** **** **** *** **** **** *** **** *** *
Short form password is: NOSE HAAG MED

[(escalation_mode) t8-2-bur09-a-sp:~]#

i. Use the showpsnc command to confirm the current container values. Confirm that the primary container has a serial number (the value on the PSN line) that matches the system serial number. The system serial number can be checked by comparing to the serial number RFID tag on the front left hand side of the server. After confirming that there is a valid fruid primary use the copypsnc command to write the good data from the primary to the backup2 container on the MB. The following example shows copying from primary to the backup2, but you could also copy from backup1 if needed.

[(restricted_shell) t8-2-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///persist/psnc_backup1.xml
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
-----------------+---------------------+----------------------+-------------------
PPN 35133190+1+1 35133190+1+1 35133190+1+1
PSN 1735NN80U2 1735NN80U2 0000000000
MACADDR 00:10:E0:D5:7D:3A 00:10:E0:D5:7D:3A 00:10:E0:D5:7D:3A
HOSTID 86d57d3a 86d57d3a 86d57d3a
Product Name SPARC T8-2 SPARC T8-2 SPARC T8-2
RFID SN 341A583DE580000000028000 341A583DE580000000028000 341A583DE580000000028000

[(restricted_shell) t8-2-bur09-a-sp:~]# copypsnc Primary Backup2

[(restricted_shell) t8-2-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///persist/psnc_backup1.xml
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
-----------------+---------------------+-----------------------+-------------------
PPN 35133190+1+1 35133190+1+1   35133190+1+1
PSN 1735NN80U2 1735NN80U2   1735NN80U2
MACADDR 00:10:E0:D5:7D:3A 00:10:E0:D5:7D:3A  00:10:E0:D5:7D:3A
HOSTID 86d57d3a 86d57d3a   86d57d3a
Product Name SPARC T8-2 SPARC T8-2 SPARC T8-2
RFID SN 341A583DE580000000028000 341A583DE580000000028000 341A583DE580000000028000

[(restricted_shell) t8-2-bur09-a-sp:~]# exit

j. At this point if all of the fruid containers match and have the correct serial number data this step is done. If more than one of the fruid containers had non-valid entries then the copypsnc command should be used to copy over the valid data to the other container that is not valid. (ie. "copypsnc Primary Backup1") After confirming all fruid data is correct reset the ILOM to confirm that the fruid data persists through a reboot and remove the escalation user if needed.

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP
..........

***login as the root user again and check the fruid data***

-> set SESSION mode=restricted

WARNING: The "Restricted Shell" account is provided solely to allow Services to perform diagnostic tasks.

[(restricted_shell) t8-2-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///persist/psnc_backup1.xml
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
----------------+-----------------------+--------------------+-------------------
PPN 35133190+1+1 35133190+1+1 35133190+1+1
PSN 1735NN80U2 1735NN80U2   1735NN80U2
MACADDR 00:10:E0:D5:7D:3A 00:10:E0:D5:7D:3A  00:10:E0:D5:7D:3A
HOSTID 86d57d3a 86d57d3a   86d57d3a
Product Name SPARC T8-2 SPARC T8-2 SPARC T8-2
RFID SN 341A583DE580000000028000 341A583DE580000000028000 341A583DE580000000028000

[(restricted_shell) t8-2-bur09-a-sp:~]# exit

-> cd /SP/users
/SP/users
-> delete escuser
Are you sure you want to delete /SP/users/escuser (y/n)? y
Deleted /SP/users/escuser

k. If trouble is encountered during any of the steps of accessing escalation mode and fixing the fruid containers please contact the TSC for assistance.

How to verify the Motherboard is working properly

1. Log into ILOM to confirm motherboards 'fault_state' status.

Sample:

-> show /SYS/MB

/SYS/MB
Targets:
0V9_SAS0_OBPS
.

.
V_NCSI_PWR
XGBE

Properties:
type = Motherboard
ipmi_name = MB
fru_description = ASY,MB,T8-2
fru_manufacturer = Oracle Corporation
fru_part_number = 7346999
fru_rev_level = 01
fru_serial_number = 465769T+17221G00L1
fault_state = OK
clear_fault_action = (none)

Commands:
cd
set
show

2. Check ILOM event log to see if any error related motherboard.

-> show /SP/faultmgmt
-> show /SP/logs/event/list

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE FE/ADMINISTRATOR NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

Boot system and monitor boot sequence for errors. Test functionality of system:

Run the Solaris "fmadm faulty" and SP/ILOM "show faulty" command to verify that the fault has been cleared.
Perform one of the following tasks based on your verification results:
- If the previous steps did not clear the fault, refer to doc 1004229.1 for information about the tools and methods you can use to diagnose and clear component faults.
- If the previous steps indicate that no faults have been detected, the component has been replaced successfully. No further action is required
Restart software applications per applicable administration guides to resume system operation.

PARTS NOTE:
https://support.oracle.com/handbook_partner/Systems/SPARC_T8_2/components.html#SystemBoard

REFERENCE INFORMATION:
SPARC T8-2 Service Manual: https://docs.oracle.com/cd/E79179_01/html/E80511/index.html

Save

References

<NOTE:1571054.1> - Performing an AC power cycle on the T3/T4/T5/S7/T7/T8 Servers

Attachments

This solution has no attachment