How to Replace a Netra SPARC S7-2 Motherboard and SP (Embedded Service Processor) [VCAP]

Asset ID:	1-71-2159205.1
Update Date:	2018-01-31
Keywords:

Solution Type Technical Instruction Sure

Solution 2159205.1 : How to Replace a Netra SPARC S7-2 Motherboard and SP (Embedded Service Processor) [VCAP]

Applies to:

Netra SPARC S7-2 - Version All Versions and later
Information in this document applies to any platform.

Goal

How to Replace a Netra SPARC S7-2 Motherboard and SP(Embedded Service Processor)

***************************************************************************************************************************
To report errors or request improvements on this procedure,please go to http://mosemp.us.oracle.com and put a comment on Doc ID: 2159205.1
***************************************************************************************************************************

Solution

How to Replace a Netra SPARC S7-2 Motherboard and SP(Embedded Service Processor)

ESD Caution:

Circuit boards and drives contain electronic components that are extremely sensitive to static electricity. Ordinary amounts of static electricity from clothing or the work environment can destroy the components located on these boards. Do not touch the components along their connector edges.
Use a Antistatic Wrist strap. Attach one end of the strap to your wrist and the other end to the chassis, depending on what type of strap you use, with the adhesive end or the metal plug.
Use an Antistatic Mat. Place ESD-sensitive components such as motherboards, memory, and other PCBs on an antistatic mat.

Contamination Caution:

Dust particles of packaging material are number one cause of datacenter contamination. Make sure to remove all packaging material, up to the ESD safe packaging material, while still being outside the datacenter.

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER NEED:
Netra SPARC S7-2 Product Training required, to be serviced by qualified Oracle Service personnel, requires the ability to follow steps similar to what is in the Product Service manual.

Time Estimate: 120 minutes

TASK COMPLEXITY: 2

FIELD ENGINEER INSTRUCTIONS

PROBLEM OVERVIEW: Netra SPARC S7-2 Motherboard Replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:

DAMAGE ALERT: Perform a visual inspection of the replacement part to make sure that there are no damaged components, connectors, bent pins, damaged packages during shipping, etc). If the part is damaged, don't install it into the system, order a new part. Handle with caution and package carefully the return FRU just as the new FRU was packaged, to avoid any damages during shipping.

NOTE: The LDOM configuration (if any) needs to be saved before motherboard replacement to avoid loss of LDOM configuration, refer to doc 1019720.1 for details.

NOTE: When replacing the service processor (SP), you need to backup/restore the configuration settings maintained in the SP. Before replacing the SP, save the configuration using the Oracle ILOM backup utility. Refer to the Oracle Integrated Lights Out Manager (ILOM) 3.2.x Documentation for instructions on Backing up and Restoring the Oracle ILOM configuration.

Note - A data backup is not a prerequisite but is a wise precaution.

Customer should perform an orderly and graceful shutdown of applications and OS to get the OpenBoot PROM prompt. Run the printenv command and make a note of any OpenBoot PROM variables that have been modified. Then power off the server and remove the AC power cords from the system.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Replace the Motherboard

1. Log into the ILOM and check the fruid container values and sync them if needed.

a. To avoid mismatched fruid values causing a failure after a motherboard replacement the fruid data should be confirmed to have matching data in at least the Primary (DBP) and Backup1 (PS0) containers so that the motherboard will have it's container updated automatically after replacement. Go into restricted mode and use the showpsnc command to check this.

           -> set SESSION mode=restricted

           WARNING: The "Restricted Shell" account is provided solely to allow Services to perform diagnostic tasks.

           [(restricted_shell) s7-2l-bur09-a-sp:~]$ showpsnc
           Primary: fruid:///SYS/DBP
           Backup 1: file:///SYS/PS0
           Backup 2: fruid:///SYS/MB

           Element           | Primary | Backup1 | Backup2
          ------------------+----------------------+-----------------------+-------------------
           PPN 34235727+1+1       34235727+1+1 34235727+1+1
           PSN AK00370269 AK00370269 AK00370269
           MACADDR 00:10:E0:B3:0C:28 00:10:E0:B3:0C:28   00:10:E0:B3:0C:28
           HOSTID 86b30c28    86b30c28    86b30c28
           Product Name SPARC S7-2L SPARC S7-2L   SPARC S7-2L
           [(restricted_shell) s7-2l-bur09-a-sp:~]$ exit

b. The above example shows a system with all three containers properly in sync. If the output from the system does not show all of the containers with matching values then you should reset the SP and then re-check the values again. An ILOM reset will attempt to auto-populate the matching values if one container is out of sync.

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP

c. After an ILOM reset if the Primary and Backup containers match then proceed with the following steps to replace the motherboard. If these two containers do not match then DO NOT proceed with the replacement yet.
d. If the containers do not match you will need to use the copypsnc command from service or escalation mode to copy the data from the good container so that the Primary and Backup1 containers match (Backup2 is the motherboard and we are about to replace this so it is not as important at this step).

If you are unfamiliar with this process and require assistance please reference the steps for using copypsnc to fix the serial number detailed in the "How to update product serial number on systems which implement TLI functionality (Doc ID 1280913.1)" and contact the TSC if needed. How to access service mode and escalation mode on ILOM 3.x and later platforms (Doc ID 1019946.1).
e. After the fruid data in the Primary and Backup1 containers have been confirmed to match proceed with the following steps.

2. Back up the Oracle ILOM variables per doc: https://docs.oracle.com/cd/E37444_01/html/E37446/z400371a1482122.html.

3. Prepare the server for service.

a. Power off the server and disconnect the power cords from the power supplies.
b. Extend the server to the maintenance position in the rack.
c. Attach an anti-static wrist strap.

Caution - Components inside the chassis might be hot. Use caution when servicing components inside the chassis.

Note - When replacing the motherboard, you will need to remove the SCC PROM from the old motherboard and install the component on the new motherboard. The SCC PROM contains the system host ID and MAC address. Transferring these components preserves the system-specific information stored on these modules.

4. Remove the top cover.

a. To open the server top cover, press and hold down the top cover release button and use the recessed area to slide the top cover toward the rear of the server about 0.5 inches (12.7 mm).
b. Lift the cover off the chassis and set it aside.

5. Open the front bezel.

6. Remove all Fan Modules.

7. Remove the Fan Tray.

8. Disconnect the disk backplane cables & rotate the Drive Cage to the service position.

9. Remove the USB cable from the PSU duct's channel & then remove the PSU duct.

10. Loosen the cables from the main air duct assembly.

11. Loosen the left thumb screw bracket, slide it forward to remove, & then remove the air duct.

12. Remove all PCIe cards.

Note - Always remove transceivers from a PCIe card(s) before removing the card from the server.
Note - Keep track of which slot each PCIe card was in so you can return them to their original positions

Removing PCIe cards in slots 1 through 6

a. Unplug all data cables from the PCIe card
b. Rotate the PCIe card locking mechanism, and then lift up on the PCIe card to disengage it from the motherboard connectors

Removing PCIe card in slot 7

a. PCIe slot 7 is located on the mother board between the CMP0 DIMMs and the left side of the chassis. It contains the on board SAS disk controller.
b. Remove screw from top of standoff
c. Lift PCIe card with SAS cables off mother board

13. Remove the bus bar screws.

14. Remove NVMe cables & install on replacement motherboard

15. Disconnect the ribbon cables from the left front LED indicator module and right front LED/USB indicator module

16. Disconnect the ribbon cables to the PDB & install on replacement motherboard

17. Disconnect the ribbon cables to the disk backplane & install on replacement motherboard

18. Disconnect the power cable from the motherboard to the front storage drive backplane & install on replacement motherboard

19. Remove the right thumb screw bracket.

20. Remove the fan cage bracket.

21. Remove the mother board

   a. Unscrew the 2 captive screws on the front of the motherboard.
     b. Grasp the two motherboard handles, and slide the motherboard forward off of the alignment pins.
     c. Tilt the motherboard left, and slowly lift the motherboard up and out of the server.
   d. Place the motherboard assembly on an antistatic mat, and next to the replacement motherboard.

22. Remove the eUSB drive from the original motherboard and install it on the replacement motherboard.

a. Loosen the screw that attaches the eUSB drive to the motherboard.
b. Lift the eUSB drive up to disconnect it from the motherboard.
c. Attach it to the replacement system board & tighten the screw.

23. Remove the SEEPROM from the original motherboard and install it on the replacement motherboard

24. If you are replacing the original motherboard, then remove each DIMM on motherboard & then insert it on new motherboard in the same position.

Note - Install the DIMMs only in the sockets (connectors) that correspond to the sockets from which they were removed. Performing a one-to-one replacement of the DIMMs significantly reduces the possibility that the DIMMs will be installed in the wrong slots. If you do not reinstall the DIMMs in the same sockets, server performance might suffer and some DIMMs might not be used by the server.

This is the end of the removal process

Start of the install process

1. Insert the motherboard into the server chassis

a. Lift the motherboard by the handles, tilt it to the left & lower it into position. Slowly move the motherboard rearward until the connectors slightly protrude through the rear of the server.
b. Lower the motherboard onto the bottom alignment pins. Slowly slide the motherboard until it drops, engaging the bottom alignment pins. Continue to slide the motherboard forward until it seats, but be sure to line up the plastic LED tube with the hole.
c. Tighten the 2 captive screws on the front of the motherboard.
d. Ensure that it does NOT have jumper J1_SP installed (located in rear near Ser Mgmt connector see attachment).

2. Install the fan cage bracket.

3. Install the right thumb screw bracket.

4. Connect the ribbon cables from the left front LED indicator module and right front LED/USB indicator module

5. Install the bus bar screws.

6. Insert PCIe Cards in Slots 1-6

a. Insert the PCIe card into the correct slot [1], and rotate the PCIe locking mechanism to secure the PCIe card in place
b. Reconnect the cables to the PCIe card that you unplugged during the removal procedure

7. Installing PCIe card in slot 7

a. PCIe slot 7 is located on the mother board between the CMP0 DIMMs and the left side of the chassis. It contains the on board SAS disk controller.
b. Insert screw into top of standoff
c. Insert PCIe card with SAS cables onto mother board

8. Place the main air duct assembly in position.

9. Place the left thumb screw bracket in position, slide it back, then tighten the thumb screw.

10. Fasten the disk cables to the Air Duct.

11. Install the PSU air duct & then route the USB cable in it's channel.

12. Rotate the Drive Cage to the service position & connect the disk backplane cables.

13. Install Fan Tray.

14. Install the fan modules.

15. Close the front bezel.

16. Install the top cover.

17. Return the Server to operation.

a. Remove any anti-static measures that were used.
b. Return the server to it's normal operating position within the rack.
c. Re-install the AC power cords and any data cables that were removed.

18. Prior to powering on the server, connect a terminal or a terminal emulator (PC or workstation) to the SER MGT port.

a. If needed, configure the SP network port to enable the firmware image to be downloaded. Refer to the Oracle ILOM documentation for network configuration instructions.
b. Download the system firmware. Follow the firmware download instructions in the Oracle ILOM documentation

Note - The LDOM configuration (if any) needs to be restored after motherboard replacement to avoid loss of LDOM configuration, refer to doc 1019720.1 for details.

19. Power on server. Verify that the Power/OK indicator led lights steady on.

20. Set the system serial number/fruid data if needed.

a. The motherboard is not the primary fruid container in this server so when it is replaced you should not normally need to fix the serial number information (TLI).
b. login to the ILOM as root and then enter the "restricted shell" to check the fruid values. Follow the example below to enter restricted shell and use the showpsnc command:

-> set SESSION mode=restricted

WARNING: The "Restricted Shell" account is provided solely to allow Services to perform diagnostic tasks.

[(restricted_shell) s7-2l-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///SYS/PS0
Backup 2: fruid:///SYS/MB

Element         | Primary    | Backup1    | Backup2
------------------+-------------------+-------------------+-------------------
PPN    34235727+1+1 34235727+1+1 34235727+1+1
PSN    AK00370269     AK00370269 0000000000
MACADDR    00:10:E0:B3:0C:28 00:10:E0:B3:0C:28  00:10:E0:B3:0C:28
HOSTID 86b30c28    86b30c28 86b30c28
Product Name SPARC S7-2L SPARC S7-2L   SPARC S7-2L
[(restricted_shell) s7-2l-bur09-a-sp:~]#/div>

c. When the motherboard is replaced the Backup2 fruid container will likely not match the Primary entry. If it does not you must enter escalation or service mode to fix it (if all three entries match this step is done).
d. Contact the TSC to request an escalation password (service mode will work also if just the copypsnc command ends up needing to be used, if the setpsnc command is needed escalation mode is required. setpsnc is not covered in this procedure).
e. Provide your TSC contact the output from the following ILOM commands- "version", "show /SYS product_serial_number", and "show /SP/clock". If the product_serial_number information does not give good output then provide the showpsnc output that was seen in step b above as well.
f. The TSC will provide an escalation password that is made up of 32 short words. Follow the example below to create a new user with the 'Service' role assigned. The Service role is required to access service or escalation modes. In the following example we will create a user named 'escuser' with the service role.

-> cd /SP/users
/SP/users
-> create escuser
Creating user...
Enter new password: ********
Enter new password again: ********
Created /SP/users/escuser
-> set escuser role=aucros
Set 'role' to 'aucros'
-> show escuser
/SP/users/escuser
Targets:
ssh
Properties:
role = aucros
password = *****

g. Set the check_physical_presence to false and then exit from the ILOM so that you can login as the newly created user.

-> set /SP check_physical_presence=false
Set 'check_physical_presence' to 'false'
-> show /SP check_physical_presence
/SP
Properties:
check_physical_presence = false

-> exit

h. Login using the escuser login and enter escalation mode using the password that was provided by the TSC.

s7-2l-bur09-a-sp login: escuser
Password:

Oracle(R) Integrated Lights Out Manager

Version 3.2.4.34 r95732

Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.

Warning: The system appears to be in manufacturing test mode.
Contact Service immediately.

Hostname: s7-2l-bur09-a-sp

-> cd /SP/users/ecsuser/escalation
-> set SESSION mode=escalation
Password:**** **** **** **** **** *** *** **** **** **** **** **** **** **** **** **** *** *** **** *** **** **** **** *** **** **** *** **** *** *
Short form password is: NOSE HAAG MED

[(escalation_mode) s7-2l-bur09-a-sp:~]#

i. Use the showpsnc command to confirm the current container values. Confirm that the primary container has a serial number (the value on the PSN line) that matches the system serial number. The system serial number can be checked by comparing to the serial number RFID tag on the front left hand side of the server. After confirming that there is a valid fruid primary use the copypsnc command to write the good data from the primary to the backup2 container on the MB. The following example shows copying from primary to the backup2, but you could also copy from backup1 if needed.

[(escalation mode) s7-2l-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///SYS/PS0
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
------------------+-------------------+-------------------+-------------------
PPN 34235727+1+1 34235727+1+1 34235727+1+1
PSN AK00370269 AK00370269 0000000000
MACADDR 00:10:E0:B3:0C:28 00:10:E0:B3:0C:28 00:10:E0:B3:0C:28
HOSTID 86b30c28 86b30c28 86b30c28
Product Name SPARC S7-2L SPARC S7-2L SPARC S7-2L

[(escalation mode) s7-2l-bur09-a-sp:~]# copypsnc Primary Backup2

[(escalation mode) s7-2l-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///SYS/PS0
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
------------------+-------------------+-------------------+-------------------
PPN 34235727+1+1 34235727+1+1   34235727+1+1
PSN AK00370269 AK00370269 AK00370269
MACADDR 00:10:E0:B3:0C:28 00:10:E0:B3:0C:28 00:10:E0:B3:0C:28
   HOSTID 86b30c28 86b30c28 86b30c28
   Product Name SPARC S7-2L SPARC S7-2L   SPARC S7-2L
[(escalation mode) s7-2l-bur09-a-sp:~]# exit

j. At this point if all of the fruid containers match and have the correct serial number data this step is done. If more than one of the fruid containers had non-valid entries then the copypsnc command should be used to copy over the valid data to the other container that is not valid. (ie. "copypsnc Primary Backup1") After confirming all fruid data is correct reset the ILOM to confirm that the fruid data persists through a reboot and remove the escalation user if needed.

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP
..........

***login as the root user again and check the fruid data***

-> set SESSION mode=restricted

WARNING: The "Restricted Shell" account is provided solely to allow Services to perform diagnostic tasks.

[(restricted_shell) s7-2l-bur09-a-sp:~]# showpsnc
Primary: fruid:///SYS/DBP
Backup 1: file:///SYS/PS0
Backup 2: fruid:///SYS/MB

Element | Primary | Backup1 | Backup2
-------------------+------------------------+-------------------------+-------------------
PPN 34235727+1+1 34235727+1+1 34235727+1+1
PSN AK00370269 AK00370269 AK00370269
MACADDR 00:10:E0:B3:0C:28 00:10:E0:B3:0C:28 00:10:E0:B3:0C:28
HOSTID 86b30c28 86b30c28 86b30c28
Product Name SPARC S7-2L SPARC S7-2L SPARC S7-2L
[(restricted_shell) s7-2l-bur09-a-sp:~]# exit

-> cd /SP/users
/SP/users
-> delete escuser
Are you sure you want to delete /SP/users/escuser (y/n)? y
Deleted /SP/users/escuser

k. If trouble is encountered during any of the steps of accessing escalation mode and fixing the fruid containers please contact the TSC for assistance.

How to verify the Motherboard is working properly

1. Log into ILOM to confirm if motherboard status is working properly.

Sample:

-> show /SYS/MB

/SYS/MB
Targets:
1V05_IOH_OBPS
1V6_IOH_OBPS
3V3_MAIN_OBPS
3V3_STBY_OBPS
BAT
BOB_VCORE_OBPS0
BOB_VCORE_OBPS1
BOB_VDDIO_OBPS0
BOB_VDDIO_OBPS1
BOB_VMEMA_OBPS0
BOB_VMEMA_OBPS1
BOB_VMEMB_OBPS0
BOB_VMEMB_OBPS1
BOB_VTERM_OBPS0
BOB_VTERM_OBPS1
CM
FM0
FM1
FM2
FM3
FPGA
IOH
NET0
NET1
NET2
NET3
PCIE1
PCIE2
PCIE3
PCIE4
PCIE5
PCIE6
SAS
SASHBA
SCC
SPM
T_0V9_SAS_OBPS
T_OUTLET0
T_OUTLET1
USB_CTRL
VCORE_IOH_OBPS0
VCORE_IOH_OBPS1
VMA_PWR_MGR
VMA_VDD_IO
VMA_VDD_TERM
VMA_VMEMA
VMA_VMEMA_OBPS
VMA_VMEMB
VMA_VMEMB_OBPS
VMA_VPP
VMA_VTTA
VMA_VTTB
VMA_V_VCORE
VMA_V_VIN
VMB_PWR_MGR
VMB_VDD_IO
VMB_VDD_TERM
VMB_VMEMA
VMB_VMEMA_OBPS
VMB_VMEMB
VMB_VMEMB_OBPS
VMB_VPP
VMB_VTTA
VMB_VTTB
VMB_V_VCORE
VMB_V_VIN
V_+0V8_XGBE
V_+0V9_SAS
V_+0V9_SAS_OBPS
V_+0V67_XGBE
V_+1V0_VUSB
V_+1V1_USB_HUB
V_+1V2_XGBE
V_+1V8_SAS
V_+1V35_SAS
V_+2V5_STBYA
V_+2V5_STBYB
V_+2V5_XGBE
V_+3V3_VDD
V_+5V0_VREG1
V_+5V0_VREG2
V_+12_CM
XGBE0
XGBE1
YUBA_A
YUBA_B

Properties:

type = Motherboard
ipmi_name = MB
fru_description = ASY,MB,T7-1
fru_manufacturer = Oracle Corporation
fru_part_number = 7300942
fru_rev_level = 02
fru_serial_number = 465769T+1504N200NV
fault_state = OK
clear_fault_action = (none)

Commands:
cd
set
show

2. Check ILOM event log to see if any error related motherboard.

-> show /SP/faultmgmt
-> show /SP/logs/event/list

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE FE/CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:
Boot system and monitor boot sequence for errors. Test functionality of system:
1. Run the Solaris "fmadm faulty" and SP/ILOM "show faulty" command to verify that the fault has been cleared.
2. Perform one of the following tasks based on your verification results:
* If the previous steps did not clear the fault, refer to doc 1004229.1 for information about the tools and methods you can use to diagnose and clear component faults.
* If the previous steps indicate that no faults have been detected, the component has been replaced successfully. No further action is required
3. Restart software applications per applicable administration guides to resume system operation.

PARTS NOTE:

https://support.oracle.com/handbook_partner/Systems/Netra_S7_2/components.html#SystemBoard

REFERENCE INFORMATION:
Netra SPARC S7-2 Service Manual: http://docs.oracle.com/cd/E72798_01/pdf/E73207.pdf

Save

References

<NOTE:1280913.1> - How to update System, Chassis, and Product level Key Identity Properties on ILOM based systems which implement Top Level Identifier (TLI) functionality
<NOTE:1019946.1> - How to access service mode and escalation mode on ILOM 3.x and later platforms

Attachments

This solution has no attachment