How to Replace a SPARC T7-1 Motherboard [VCAP]

Asset ID:	1-71-1954706.1
Update Date:	2018-05-10
Keywords:

Solution Type Technical Instruction Sure

Solution 1954706.1 : How to Replace a SPARC T7-1 Motherboard [VCAP]

Applies to:

SPARC T7-1 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Goal

How to replace a SPARC T7-1 Motherboard

Solution

**************************************************************************************
To report errors or request improvements on this procedure, please add a comment on Doc ID: 1954706.1
**************************************************************************************

NOTE: For ALL scenarios where an AC power down or AC power cycle is required for a T7-x server, please always use the steps in doc 1571054.1 prior to physically removing AC power cables from the server.

ESD Caution:

Circuit boards and drives contain electronic components that are extremely sensitive to static electricity. Ordinary amounts of static electricity from clothing or the work environment can destroy the components located on these boards. Do not touch the components along their connector edges.
Use a Antistatic Wrist strap. Attach one end of the strap to your wrist and the other end to the chassis, depending on what type of strap you use, with the adhesive end or the metal plug.
Use an Antistatic Mat. Place ESD-sensitive components such as motherboards, memory, and other PCBs on an antistatic mat.

Contamination Caution:

Dust particles of packaging material are number one cause of datacenter contamination. Make sure to remove all packaging material, up to the ESD safe packaging material, while still being outside the datacenter.

DISPATCH INSTRUCTIONS

WHAT SKILLS ARE REQUIRED?: SPARC T7-1 Product Training required, to be serviced by qualified Oracle Service personnel, requires the ability to follow steps similar to what is in the Product Service manual.

Time Estimate: 90 minutes

TASK COMPLEXITY: 1

REMOVAL/REPLACEMENT INSTRUCTIONS:

PROBLEM OVERVIEW: SPARC T7-1 Motherboard Replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:

NOTE: Customer should perform an orderly and graceful shutdown of applications and OS to get the OpenBoot PROM prompt. Run the printenv command and make a note of any OpenBoot PROM variables that have been modified. For ALL scenarios where an AC power down or AC power cycle is required for a T7-x server, please always use the steps in doc 1571054.1 prior to physically removing AC power cables from the server.

DAMAGE ALERT: Perform a visual inspection of the replacement part to make sure that there are no damaged components, connectors, bent pins, damaged packages during shipping, etc). If the part is damaged, don't install it into the system, order a new part. Handle with caution and package carefully the return FRU just as the new FRU was packaged, to avoid any damages during shipping.

Note - The LDOM configuration (if any) needs to be saved before motherboard replacement to avoid loss of LDOM configuration, refer to doc 1019720.1 for details.

Note - A data backup is not a prerequisite but is a wise precaution.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Replace the Motherboard

1. Log into the ILOM and check the fruid container values and sync them if needed.

a. To avoid mismatched fruid values causing a failure after a motherboard replacement the fruid data should be confirmed to have matching data in at least the Primary (DBP) and Backup1 (SPM) containers so that the motherboard will have it's container updated automatically after replacement. Go into restricted mode and use the showpsnc command to check this.

           -> set SESSION mode=restricted

           WARNING: The "Restricted Shell" account is provided solely to allow Services to perform diagnostic tasks.

           [(restricted_shell) t7-1-bur09-a-sp:~]# showpsnc
           Primary: fruid:///SYS/DBP
           Backup 1: file:///persist/psnc_backup1.xml
           Backup 2: fruid:///SYS/MB

           Element           | Primary           | Backup1           | Backup2
          ------------------+-------------------+-------------------+-------------------
           PPN                 33316362+1+1        33316362+1+1 33316362+1+1
           PSN                 AK00327467 AK00327467          AK00327467
           MACADDR 00:10:E0:70:74:EE   00:10:E0:70:74:EE 00:10:E0:70:74:EE
           HOSTID 867074ee 867074ee 867074ee
           Product Name SPARC T7-1 SPARC T7-1          SPARC T7-1
           [(restricted_shell) t7-1-bur09-a-sp:~]# exit

b. The above example shows a system with all three containers properly in sync. If the output from the system does not show all of the containers with matching values then you should reset the SP and then re-check the values again. An ILOM reset will attempt to auto-populate the matching values if one container is out of sync.

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP

c. After an ILOM reset if the Primary and Backup1 containers match then proceed with the following steps to replace the motherboard. If these two containers do not match then DO NOT proceed with the replacement yet.
d. If the containers do not match you will need to use the copypsnc command from service or escalation mode to copy the data from the good container so that the Primary and Backup1 containers match (Backup2 is the motherboard and we are about to replace this so it is not as important at this step).

If you are unfamiliar with this process and require assistance please reference the steps for using copypsnc to fix the serial number detailed in the "How to update product serial number on systems which implement TLI functionality (Doc ID 1280913.1)" and contact the TSC if needed. How to access service mode and escalation mode on ILOM 3.x and later platforms (Doc ID 1019946.1).
e. After the fruid data in the Primary and Backup1 containers have been confirmed to match proceed with the following steps.

2. Prepare the server for service.

a. Power off the server and disconnect the power cords from the power supplies.
b. Extend the server to the maintenance position in the rack.
c. Attach an anti-static wrist strap.

Caution - Components inside the chassis might be hot. Use caution when servicing components inside the chassis.

Note - When replacing the motherboard, you will need to remove the SPM, SCC PROM and eUSB drive from the old motherboard and install these components on the new motherboard. The SP contains the Oracle ILOM system configuration data, and the SCC PROM contains the system host ID and MAC address. Transferring these components preserves the system-specific information stored on these modules. Whenever you replace the motherboard or the SPM, you must update the firmware so the portions of firmware in the SPM and on the motherboard are consistent.

3. Remove the top cover and open the fan door to remove all of the fan modules.

a. Open the fan Door. Slide the green fan door latches forward and swing the door up to the open position.
b. Loosen the green screw on each fan module. The screws should only be finger tight. If you have difficulty loosening a screw by hand, use a Phillips No. 2 screwdriver.
c. Using your thumb and forefinger, grasp each of the fan modules in the finger recesses located in the plastic between the fans and lift the fan modules straight up and out of the chassis.
d. To open the server top cover, press and hold down the top cover release button and use the recessed area to slide the top cover toward the rear of the server about 0.5 inches (12.7 mm).
e. Lift the cover off the chassis and set it aside.

4. Open the clear plastic air duct assembly cover by lifting the edge of the cover closest to the rear of the server. The cover can stand in its upright position but perform the next steps if you need to remove the cover:

a. Open the fan cover.
b. Pull open the plastic tabs to release the clear plastic air duct assembly covers hinged edge and remove it from the server.

5. Remove all PCIe cards.

Note - Always remove transceivers from a PCIe card(s) before removing the card from the server.

Note - Keep track of which slot each PCIe card was in so you can return them to their original positions.

6. If you are replacing the motherboard, remove the following components and place them on an ESD mat:

a. Both memory risers (if installed) and All DIMMs on motherboard.

Note - See "How to Replace a SPARC T7-1 Memory Mezzanine (Riser) (Doc ID 1964501.1)" on how to remove and install memory risers.

Note - Keep track of which slot each DIMM came from and return them to their original position.

Note - Keep track of which side each memory riser came from and return them to their original position.

b. SCC PROM
c. SPM (Service Processor Module)

d. eUSB

Note - The eUSB is below the MRs. Remove the eUSB drive from old motherboard. You will reinstall the eUSB drive on the new motherboard

7. Disconnect all the cables from the motherboard.

a. Disconnect the ribbon cables from the motherboard that go to the left and right LED front indicator modules (The left and right LED indicator modules do not need to be removed to replace the motherboard).
b. Disconnect the cable from the server intrusion switch.
c. Disconnect the DVD drive cable from the motherboard (Carefully guide the DVD drive cable through the chassis mid-wall. Place the cable on top of the disk cage so that it is away from the motherboard. You do not need to disconnect the DVD drive cable from the DVD drive).
d. Disconnect the signal and power cables from the motherboard that go to the drive backplane and get them away from the motherboard.

Caution - Some of these data cables are delicate. Please use caution while disconnecting them and ensure they are safely out of the way when removing and installing the motherboard.

8. Disconnect the mid-wall from the chassis.
a. Remove the screw on each side of the chassis that secures the mid-wall to the chassis.
b. Loosen the four green captive screws that secure the mid-wall to the bottom of the chassis. Use a #2 Phillips screwdriver to loosen the captive screws.

9. Lift the mid-wall out of the chassis.

10. Release the power supplies and pull them slightly out of the server (The power supplies do not need to be removed from the chassis to lift out the motherboard).

11. Lift the motherboard out of the chassis (You can use the bar by the rear I/O panel and the metal handle in front of the cable channel as handles to lift the motherboard).

Note - Angle the motherboard up as you lift it out of the chassis.

12. Place the motherboard assembly on an ESD mat.

13. Install new motherboard into the server chassis.

a. Grasp the motherboard by the handle and place it into the chassis (You can use the metal handle in front of the cable channel and the bar by the rear I/O panel as handles to lift the motherboard).

Caution: Ensure that the remaining cables do not get caught on edges of the motherboard or under it.

b. Tilt the motherboard to the right side so it gets under the power supply.
c. Level the motherboard in the server chassis.
d. Slide the motherboard to the rear of the server to engage the raised standoffs.

14. Insert the mid-wall into the chassis.

15. Fasten the mid-wall to the chassis.

a. Tighten the four green captive screws that secure the mid-wall to the bottom of the chassis (Use a No. 2 Phillips screwdriver to tighten the captive screws).
b. Fasten a screw on each side of the chassis to secure the mid-wall to the chassis.

16. Insert the power supplies back into their connectors on the motherboard (Make sure they line up correctly prior to inserting to prevent damage to the connector).

17. Reconnect all the cables to the motherboard.

a. Connect the cable from the server intrusion switch.
b. Connect the ribbon cables to the motherboard from the left and right LED indicator modules.
c. Connect the signal and power cables from the drive backplane to the motherboard.

Caution - Some of these data cables are delicate. Please use caution while connecting them and ensure they are safely out of the way when removing and installing the motherboard.

d. Connect the DVD drive cable to the motherboard (Thread the DVD cable through the chassis mid-wall to reach the motherboard connector).

18. Install all the fan modules.

19. If you replaced the motherboard, install the following components saved from original motherboard:

a. Both memory risers (if applicable) and All DIMMs.

Note - See "How to Replace a SPARC T7-1 Memory Mezzanine (Riser) (Doc ID 1964501.1)" on how to remove and install memory risers.

Note - Keep track of which slot each DIMM came from and return them to their original position.

Note - Keep track of which side each memory riser came from and return them to their original position.

b. SCC PROM.
c. Service processor module (SPM).

20. Install all PCIe cards saved from original motherboard.

Note - Always install transceivers for a PCIe card(s) after installing the card(s) in the server.

Note - Remember to return each PCIe card to their original positions.

21. Close the clear plastic air duct assembly cover by rotating it down over the motherboard / two memory risers and then slightly pressing in the two tabs to secure it. See following if you need to reattach the cover:

a. Open the fan cover.
b. Align the clear plastic air duct assembly cover on its hinge edge.
c. Attach the hinged plastic tabs to the server by pulling them open and letting them seat into the holes so that the clear plastic air duct assembly cover can hinge up and down.
d. Lower the clear plastic air duct assembly cover (making sure the two pairs of side tabs clear the sides of the server).

22. Install the top cover on the system

23. Return the Server to operation.

a. Remove any anti-static measures that were used.
b. Return the server to it's normal operating position within the rack.
c. Re-install the AC power cords and any data cables that were removed.

24. Prior to powering on the server, connect a terminal or a terminal emulator (PC or workstation) to the SP SER MGT port. If the SP detects that the new host firmware component is incompatible with SP firmware component, further action will be suspended and the following message will be delivered over the SER MGT port:

Unrecognized Chassis: This module is installed in an unknown or unsupported chassis. You must upgrade the firmware to a newer version that supports this chassis.

If you see this message, go on to Step #25.

Note - Whenever you replace the SPM or the motherboard, update the firmware on the server so the portions of firmware in the two components remain consistent.

25. Download the system firmware.

a. If needed, configure the SP network port to enable the firmware image to be downloaded. Refer to the Oracle ILOM documentation for network configuration instructions.
b. Download the system firmware. Follow the firmware download instructions in the Oracle ILOM documentation

Note - You can load any supported system firmware version, including the firmware revision that had been installed prior to the replacement of the motherboard.

Note - When replacing a motherboard on T3, T4, T5, and T7 systems, certain procedures will need to be followed if on board hardware raid volumes are configured. Re-activate any RAID volumes that existed prior to replacing the motherboard. Only perform this task if your system had RAID volumes prior to replacing the motherboard. For details please consult doc 1387771.1.

Note - The LDOM configuration (if any) needs to be restored after motherboard replacement to avoid loss of LDOM configuration, refer to doc 1019720.1 for details.

26. Power on server. Verify that the Power/OK indicator led lights steady on.

27. Set the system serial number/fruid data if needed.

a. The motherboard is not the primary fruid container in this server so when it is replaced you should not normally need to fix the serial number information (TLI).
b. login to the ILOM as root and then enter the "restricted shell" to check the fruid values. Follow the example below to enter restricted shell and use the showpsnc command:

-> set SESSION mode=restricted

WARNING: The "Restricted Shell" account is provided solely to allow Services to perform diagnostic tasks.

[(restricted_shell) t7-1-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///persist/psnc_backup1.xml
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
------------------+-------------------+-------------------+-------------------
PPN 33316362+1+1 33316362+1+1 33316362+1+1
PSN AK00327467 AK00327467 0000000000
MACADDR 00:10:E0:70:74:EE 00:10:E0:70:74:EE 00:10:E0:70:74:EE
HOSTID 867074ee 867074ee 867074ee
Product Name SPARC T7-1 SPARC T7-1 SPARC T7-1
[(restricted_shell) t7-1-bur09-a-sp:~]#

c. When the motherboard is replaced the Backup2 fruid container will likely not match the Primary entry. If it does not you must enter escalation or service mode to fix it (if all three entries match this step is done).
d. Contact the TSC to request an escalation password (service mode will work also if just the copypsnc command ends up needing to be used, if the setpsnc command is needed escalation mode is required. setpsnc is not covered in this procedure).
e. Provide your TSC contact the output from the following ILOM commands- "version", "show /SYS product_serial_number", and "show /SP/clock". If the product_serial_number information does not give good output then provide the showpsnc output that was seen in step b above as well.
f. The TSC will provide an escalation password that is made up of 32 short words. Follow the example below to create a new user with the 'Service' role assigned. The Service role is required to access service or escalation modes. In the following example we will create a user named 'escuser' with the service role.

-> cd /SP/users
/SP/users
-> create escuser
Creating user...
Enter new password: ********
Enter new password again: ********
Created /SP/users/escuser
-> set escuser role=aucros
Set 'role' to 'aucros'
-> show escuser
/SP/users/escuser
Targets:
ssh
Properties:
role = aucros
password = *****

g. Set the check_physical_presence to false and then exit from the ILOM so that you can login as the newly created user.

-> set /SP check_physical_presence=false
Set 'check_physical_presence' to 'false'
-> show /SP check_physical_presence
/SP
Properties:
check_physical_presence = false

-> exit

h. Login using the escuser login and enter escalation mode using the password that was provided by the TSC.

t7-1-bur09-a-sp login: escuser
Password:

Oracle(R) Integrated Lights Out Manager

Version 3.2.4.34 r95732

Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.

Warning: The system appears to be in manufacturing test mode.
Contact Service immediately.

Hostname: t7-1-bur09-a-sp

-> cd /SP/users/ecsuser/escalation
-> set SESSION mode=escalation
Password:**** **** **** **** **** *** *** **** **** **** **** **** **** **** **** **** *** *** **** *** **** **** **** *** **** **** *** **** *** *
Short form password is: NOSE HAAG MED

[(escalation_mode) t7-1-bur09-a-sp:~]#

i. Use the showpsnc command to confirm the current container values. Confirm that the primary container has a serial number (the value on the PSN line) that matches the system serial number. The system serial number can be checked by comparing to the serial number RFID tag on the front left hand side of the server. After confirming that there is a valid fruid primary use the copypsnc command to write the good data from the primary to the backup2 container on the MB. The following example shows copying from primary to the backup2, but you could also copy from backup1 if needed.

[(escalation mode) t7-1-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///persist/psnc_backup1.xml
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
------------------+-------------------+-------------------+-------------------
PPN 33316362+1+1 33316362+1+1 33316362+1+1
PSN AK00327467 AK00327467 0000000000
MACADDR 00:10:E0:70:74:EE 00:10:E0:70:74:EE 00:10:E0:70:74:EE
HOSTID 867074ee 867074ee 867074ee
Product Name SPARC T7-1 SPARC T7-1 SPARC T7-1

[(escalation mode) t7-1-bur09-a-sp:~]# copypsnc Primary Backup2

[(escalation mode) t7-1-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///persist/psnc_backup1.xml
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
------------------+-------------------+-------------------+-------------------
PPN 33316362+1+1 33316362+1+1 33316362+1+1
PSN AK00327467 AK00327467 AK00327467
MACADDR 00:10:E0:70:74:EE 00:10:E0:70:74:EE 00:10:E0:70:74:EE
HOSTID 867074ee 867074ee 867074ee
Product Name SPARC T7-1 SPARC T7-1 SPARC T7-1
[(escalation mode) t7-1-bur09-a-sp:~]# exit

j. At this point if all of the fruid containers match and have the correct serial number data this step is done. If more than one of the fruid containers had non-valid entries then the copypsnc command should be used to copy over the valid data to the other container that is not valid. (ie. "copypsnc Primary Backup1") After confirming all fruid data is correct reset the ILOM to confirm that the fruid data persists through a reboot and remove the escalation user if needed.

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP
..........

***login as the root user again and check the fruid data***

-> set SESSION mode=restricted

WARNING: The "Restricted Shell" account is provided solely to allow Services to perform diagnostic tasks.

[(restricted_shell) t7-1-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///persist/psnc_backup1.xml
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
-------------------+------------------------+-------------------------+-------------------
PPN 33316362+1+1 33316362+1+1 33316362+1+1
PSN AK00327467 AK00327467 AK00327467
MACADDR 00:10:E0:70:74:EE 00:10:E0:70:74:EE 00:10:E0:70:74:EE
HOSTID 867074ee 867074ee 867074ee
Product Name SPARC T7-1 SPARC T7-1 SPARC T7-1
[(restricted_shell) t7-1-bur09-a-sp:~]# exit

-> cd /SP/users
/SP/users
-> delete escuser
Are you sure you want to delete /SP/users/escuser (y/n)? y
Deleted /SP/users/escuser

k. If trouble is encountered during any of the steps of accessing escalation mode and fixing the fruid containers please contact the TSC for assistance.

How to verify the Motherboard is working properly

1. Log into ILOM to confirm if motherboard status is working properly.

Sample:

-> show /SYS/MB

/SYS/MB
Targets:
1V05_IOH_OBPS
1V6_IOH_OBPS
3V3_MAIN_OBPS
3V3_STBY_OBPS
BAT
BOB_VCORE_OBPS0
BOB_VCORE_OBPS1
BOB_VDDIO_OBPS0
BOB_VDDIO_OBPS1
BOB_VMEMA_OBPS0
BOB_VMEMA_OBPS1
BOB_VMEMB_OBPS0
BOB_VMEMB_OBPS1
BOB_VTERM_OBPS0
BOB_VTERM_OBPS1
CM
FM0
FM1
FM2
FM3
FPGA
IOH
NET0
NET1
NET2
NET3
PCIE1
PCIE2
PCIE3
PCIE4
PCIE5
PCIE6
SAS
SASHBA
SCC
SPM
T_0V9_SAS_OBPS
T_OUTLET0
T_OUTLET1
USB_CTRL
VCORE_IOH_OBPS0
VCORE_IOH_OBPS1
VMA_PWR_MGR
VMA_VDD_IO
VMA_VDD_TERM
VMA_VMEMA
VMA_VMEMA_OBPS
VMA_VMEMB
VMA_VMEMB_OBPS
VMA_VPP
VMA_VTTA
VMA_VTTB
VMA_V_VCORE
VMA_V_VIN
VMB_PWR_MGR
VMB_VDD_IO
VMB_VDD_TERM
VMB_VMEMA
VMB_VMEMA_OBPS
VMB_VMEMB
VMB_VMEMB_OBPS
VMB_VPP
VMB_VTTA
VMB_VTTB
VMB_V_VCORE
VMB_V_VIN
V_+0V8_XGBE
V_+0V9_SAS
V_+0V9_SAS_OBPS
V_+0V67_XGBE
V_+1V0_VUSB
V_+1V1_USB_HUB
V_+1V2_XGBE
V_+1V8_SAS
V_+1V35_SAS
V_+2V5_STBYA
V_+2V5_STBYB
V_+2V5_XGBE
V_+3V3_VDD
V_+5V0_VREG1
V_+5V0_VREG2
V_+12_CM
XGBE0
XGBE1
YUBA_A
YUBA_B

Properties:

type = Motherboard
ipmi_name = MB
fru_description = ASY,MB,T7-1
fru_manufacturer = Oracle Corporation
fru_part_number = 7300942
fru_rev_level = 02
fru_serial_number = 465769T+1504N200NV
fault_state = OK
clear_fault_action = (none)

Commands:
cd
set
show

2. Check ILOM event log to see if any error related motherboard.

-> show /SP/faultmgmt
-> show /SP/logs/event/list

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE FE/CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:
Boot system and monitor boot sequence for errors. Test functionality of system:
1. Run the Solaris "fmadm faulty" and SP/ILOM "show faulty" command to verify that the fault has been cleared.
2. Perform one of the following tasks based on your verification results:
   * If the previous steps did not clear the fault, refer to doc 1004229.1 for information about the tools and methods you can use to diagnose and clear
     component faults.
   * If the previous steps indicate that no faults have been detected, the component has been replaced successfully. No further action is required
3. Restart software applications per applicable administration guides to resume system operation.

PARTS NOTE:
https://support.oracle.com/handbook_partner/Systems/SPARC_T7_1/components.html#SystemBoard

REFERENCE INFORMATION:
SPARC T7-1 Service Manual: http://docs.oracle.com/cd/E54976_01/html/E54980/index.html

Save

References

<NOTE:1571054.1> - Performing an AC power cycle on the T3/T4/T5/S7/T7/T8 Servers
<NOTE:1280913.1> - How to update System, Chassis, and Product level Key Identity Properties on ILOM based systems which implement Top Level Identifier (TLI) functionality
<NOTE:1019946.1> - How to access service mode and escalation mode on ILOM 3.x and later platforms

Attachments

This solution has no attachment