Oracle ZFS Storage ZS5-2
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: DISK-CAP VCAP
 

In this Document

Goal

Solution

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: FRU CAP

Applies to:

Goal

How to Remove and Replace a Motherboard Assembly in an Oracle ZS5-2 ZFS Storage Appliance

Scope:

This procedure is applicable to the following products: ZS5-2 cluster and non cluster versions

Solution

CAP PROBLEM OVERVIEW:  Motherboard replacement

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE ENGINEER NEED:  ZS5-2 ZFS Storage Appliance Training.

TIME ESTIMATE: 60 minutes

TASK COMPLEXITY: 3

FIELD ENGINEER INSTRUCTIONS

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?

Before Replacing the Motherboard

Run the following command and copy down the information prior replacing the motherboard

Replace the Motherboard

Physical motherboard replacement is Netra Server X5-2 - http://docs.oracle.com/cd/E53596_01/html/E53601/z40000ee165586.html#scrolltoc

If the system is still up and functioning, customer should perform an orderly and graceful shutdown of applications and OS. A data backup is not a prerequisite but is a wise precaution.

Removing the motherboard is a cold-service operation. You must power off the server and and remove the AC power cords from the system before you remove the motherboard

1. Consider your first step.

• If you have not prepared for service, do so now. See “Preparing for Service”.
• If you are removing the motherboard as part of another component's removal or installation procedure, go to Step 2.

2. Review this information and complete these prerequisites in this order.

• Remove the Filter on page 89
• Remove a Fan Module on page 122
• Remove all fan modules.
• Remove the Fan Tray on page 80
• Rotate the Drive Cage to the Service Position on page 81
• Remove HBA Super Cap
• Remove the Air Duct on page 83
• Remove the PSU Duct on page 87
• Remove PCIe card on page 129
• Remove the USB Flash Drive on page 140
• Remove a DIMM on page 174
• Remove all DIMMs.
• Remove a CPU on page 186
• Remove both CPUs

3. Disconnect all remaining cables from the motherboard and move them out of the way.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Servicing the Motherboard Assembly:  http://docs.oracle.com/cd/E53596_01/html/E53601/z40000ee165586.html#scrolltoc

1.  Log into the ILOM check the fruid container values and sync them if needed.

a)  To avoid mismatched fruid values causing a failure after a motherboard replacement the fruid data should be confirmed to have matching data in at least the Primary (DBP) and Backup2 (PDB) containers so that the motherboard will have it's container updated automatically after replacement.

Go into restricted mode and use the showpsnc command to check this.

b)  The above example shows a system with all three containers properly in sync. If the output from the system does not show all of the containers with matching values then you should reset the SP and then re-check the values again. An ILOM reset will attempt to auto-populate the matching values if one container is out of sync.

c)  After an ILOM reset if the Primary and Backup2 containers match then proceed with the following steps to replace the motherboard.

d)  If these two containers do not match then DO NOT proceed with the replacement yet. If the containers do not match you will need to use the copypsnc command from service or escalation mode to copy the data from the good container so that the Primary and Backup2 containers match (Backup1 is the MB and we are about to replace this so it is not as important at this step).

If you are unfamiliar with this process and require assistance please reference the steps for using copypsnc to fix the serial number detailed in the "How to update product serial number on systems which implement TLI functionality (Doc ID 1280913.1)" and contact the TSC if needed.

How to access service mode and escalation mode on ILOM 3.x and later platforms (Doc ID 1019946.1)

e)  After the fruid data in the Primary and Backup2 containers have been confirmed to match proceed with the following steps.

2.  Prepare the server for service.

Power off the server and disconnect the power cords from the power supplies.
Remove the server from the rack.
4-Post, 19-inch Hardmount Rack Instructions
Support the weight of the server for removal.
Loosen the center captive screw on the left and right rear support brackets.
Remove the two screws securing each hardmount bracket at the front of the rack.
Lift the server from the rack and set it aside on an antistatic mat.

2-Post, 19-inch Hardmount Rack
Support the weight of the server for removal.
Remove the two screws that secure each rear plate to the back of the post.
Loosen the two screws that secure each rear plate to the eyelets of the side bracket.
Slide each rear plate out of the eyelets and set them aside.
Remove the two screws from each side bracket at the front of the posts.
Lift the server from the rack and set it aside on an antistatic mat.

Attach an anti-static wrist strap.

3.  Remove the Front Air Filter, the server Top Cover, all of the Fan Modules, the Fan Tray, rotate the Disk Device Cage.

On the front air filter pinch the right and left release levers inwards, pivot the air filter forward and lift it out and away from the chassis.
Fully loosen the two captive screws at the rear of the top cover.
Slide the top cover backward about .50 inch (13mm), lift top cover straight up and off from chassis.
Note: If the top cover will not slide back, you may have to loosen or remove the top two screws on both the left and right rack support backet. These top screws cause the support backet to press against the top cover.

Remove the Fan Modules from the fan tray by pulling up on the green handle.
Unlock the Fan Tray from the system
Disconnect the cables attached to the disk backplane and the cable attached to the USB board.
Fully loosen the four captive screws securing the drive cage to the chassis (use a No. 2 Phillips screwdriver).
Rotate the drive cage to the vertical position.

4.  Disconnect the RAID HBA super capacitor, the air baffle, the PSU air duct and PCIe cards.

Grasp both ends of the HBA super capacitor cable connector, press in on the release latch, and then unplug the connector.
Move all cables away from the top of the air baffle.
Fully loosen the thumbscrew on the left bracket for the air baffle.
Slide the left (L) thumbscrew bracket forward and off of the mounting pins and set the thumbscrew bracket aside.
Lift the air duct straight up and off the alignment pins. Set the air duct aside.
Lift the PSU air duct up and out of the chassis. Set aside the PSU duct.
Disconnect cables from each of the installed PCIe cards, make sure you are aware of which cables are connected to each card and the ports to which they are attached.
Swing the retainers to their fully open position and lift the PCIe cards out of each socket.
Place each of the PCIe cards on an antistatic mat.
Remove each of the cables from the motherboard, noting their routing path and connections.

5.  Remove the motherboard (MB)

Disconnect the bus cable between the PDB and the MB at the MB.
Disconnect all remaining cables from the MB and move them out of the way.
Remove the four screws securing the PDB bus bars to the MB.
Fully loosen the two captive screws at the front of the MB.
Grasp the two MB handles and slide the MB forward off of the alignment pins.
Tilt the MB left and slowly lift the MB up and out of the chassis.
Set the MB aside on an antistatic mat.

6.  Swap the motherboard components over to the new motherboard (USB Flash drive, DIMMs, CPUs)

Prepare the new MB for installation.
On old MB, grasp the USB flash drive and pull it straight up and out of the chassis, insert USB flashdrive into the same port on the new MB.
Performing a one-to-one replacement of the DIMMs significantly reduces the possibility that the DIMMs will be installed in the wrong slots.
For each DIMM, Press down and out on the release levers at both ends of the DIMM slot, lift the DIMM up and out the slot.
Install the DIMMs onto the replacement MB in the corresponding DIMM sockets on the replacement MB, pressing firmly so that both release levers click closed. Install the DIMMs only in the sockets (connectors) that correspond to the sockets from which they were removed.
Caution - When you remove and install a CPU, a special tool is required for safe handling of the CPU. The CPU tool is provided in the Motherboard shipping package.
Remove both of the processors from the failed MB.
Gently press down on the top of the heatsink to counteract the pressure of the captive spring-loaded screws that secure the heatsink to the motherboard and loosen the four Phillips captive screws in the heatsink for the failed processor.
Using a No. 2 Phillips screwdriver, turn the screws counterclockwise alternately one turn until they are fully removed.
To separate the heatsink from the top of the processor, gently twist the heatsink left and right, while pulling upward, and then lift off the heatsink and place it upside down on a flat surface. A thin layer of thermal grease separates the heatsink and the processor. This grease acts as an adhesive.
Use an alcohol pad to clean the thermal grease from the underside of the heatsink. Be very careful not to get the thermal grease on your fingers.
Disengage the processor release lever on the right side of the processor socket (viewing the server from the front) by pushing down on the lever and moving it to the side away from the processor, and then rotating the lever upward.
Unhook the pressure frame release levers from the catches of the CPU socket and raise them upward to approximately 35 degrees.
Release the pressure frame by slightly lowering the right release lever and lifting the pressure frame up to the fully open position.
To remove the processor from the processor socket, use the CPU removal and replacement tool and perform the following steps:
Position the tool over the CPU socket and lower it onto the CPU. The triangle on the side of the tool aligns with pin 1 of the CPU.
Locate the button in the center of the top of the processor removal and replacement tool and press it to the down position.
Press the release lever on the tool to release the center button and engage the processor. An audible click indicates that the processor is engaged.
Grasp the tool by the sides and remove it from the server.
Turn the tool upside down and verify that it contains the processor.
While holding the processor tool up side down, press the center button on the tool to release the processor.
Carefully grasp the processor by the front and back edges, lift it out of the tool and place it with the circuit side down (the installed orientation) onto the antistatic mat.
Carefully clean the thermal grease off the top of the processor.
Install both of the processors on the new motherboard.
To install the processor in the processor socket, use the CPU removal and replacement tool and perform the following steps:
Press the release button on the CPU tool to open the securing clips.
Turn the tool upside down and lower the CPU into the tool, pin side up. The triangle on the side of the tool aligns with pin 1 of the CPU.
Press the hold lever outward to engage the CPU. You will hear a click, and the release button retracts.
Position the tool over the CPU socket and lower it into place. The triangle on the side of the tool aligns with pin 1 of the CPU socket.
Caution - The tool must be level and flat onto the socket, not offset or angled.

Press the release button on the tool to release the CPU into the socket.
Caution - Do not attempt to forcibly seat the CPU.

Remove the tool. The processor should lie flat in the socket, and be properly aligned.
Lower the pressure frame to the closed position. The pressure frame should sit flat over the processor.
Lower the release levers and hook them under the catches of the CPU socket.
Use the syringe to apply approximately 0.1 ml of thermal compound to the top center of the CPU.
Note - Do not distribute the compound. The spring pressure of the heat sink will evenly distribute the compound.

Carefully align the heat sink screws to the mounting posts. The “System Front” label must face the front of the server.
Caution - Moving the heat sink too much unevenly distributes the thermal compound and can lead to CPU damage.

Tighten the four captive screws of the CPU heat sink. Alternate between the screws in an X pattern, turning each one-half revolution, until they are fully seated.

7.  Install the motherboard back into the chassis

Position the MB for installation into the chassis. The heatsinks are toward the front of the chassis, and the ports are at the rear.
Lift the MB by the handles and slowly move the MB rearward until the connectors slightly protrude through the rear of the chassis.
Lower the MB onto the bottom alignment pins.
Slowly slide the MB until it drops, engaging the bottom alignment pins.
Continue to slide the MB forward until it seats. The rear of the MB must be flush with the rear of the chassis. Additionally, the four holes in the MB must align with the four holes in the bus bars.
Tighten the two captive screws near the disk tray.
Install and tighten the four bus bar screws associated with the PDB.
Reconnect the bus cable between the PDB and the MB.
Reconnect the remaining cables you previously disconnected.

8.  Reinstall the remaining components, cables, air baffle and PSU air duct:

Reinstall the PCIe cards
Swing the retainers to their fully open position and insert each PCIe card back to the original socket.
The NVME switch card is installed in slot 5 only, the Oracle Storage 12 Gbs PCIe RAID HBA card is installed in slot 6.

Swing the retainers to its fully closed position.
Connect and route the cables to each of the PCIe cards as needed.
Plug each cable into its connector until you hear an audible click.
Reinstall PSU air duct, lower the PSU duct down over the PDB and the bus bars.
Reinstall the main air duct.
Move any loose cables to the gap between the DIMMs and the chassis walls.
Align the air duct to where it installs into the chassis.
Lower the air duct onto the alignment pins.
Set the left (L) thumbscrew bracket over the alignment pins and slide it rearward. Tighten the thumbscrew.
Route the USB cable into the cable channel in the PSU air duct
Reinstall the Fan Tray
Align the fan tray to where it installs into the chassis. Two arrows on the top of power supply bay show where the fan tray aligns.
Slowly lower the fan tray into the chassis. Verify that the fan tray is properly seated and the top surfaces of the fan tray are flush with the chassis.
Close the two fan tray locks.
Reinstall the fan modules.
Align the fan module to the location where it installs into the fan tray. The green band is on the top of the fan module and the arrow points to the rear of the chassis.
Lower the fan module into the fan tray.
Press the fan module down until it clicks securely in the fan tray.
Repeat steps 1 through 3 for the remaining fan modules.
Lower the drive cage to the horizontal position. Confirm that the alignment holes at the rear of the drive cage are over the alignment pins underneath.
Tighten the four screws securing the drive cage to the chassis.
Connect the cables to the drive backplane, and the cable attached to the USB board. Plug each cable into its connector until you hear an audible click.
Complete the routing of each cable into the appropriate cable clamps.
Reconnect the RAID HBA super capacitor's cable.
Reinstall the air filter
Set the bottom edge of the filter into the lip at the front of the chassis. The left and right release levers are facing forward.
Pivot the filter to vertical. The release levers snap the filter into place.
Install the top cover
Position the top cover on the chassis, with the front edge of the top cover 0.5 in. (13 mm) rearward of the front edge of the chassis.
Slide the top cover forward into place.
Secure the top cover with the two captive screws at the rear.
Reinstall or tighted the rack support bracket top screws.

9.  Return the Server to operation.

Remove any anti-static measures that were used.
Return the server to it's normal operating position within the rack.
Re-install the power cords and any data cables that were removed.
Power on server. Verify that the Power/OK indicator led lights steady on.

10.  Verify the system serial number/fruid data if needed.

The motherboard is not the primary fruid container in this server so when it is replaced you will normally will not need to fix the serial number information.
Login to the ILOM as root and then enter the restricted shell to check the fruid values. Follow the example at the beginning of these instructions to enter restricted shell and use the showpsnc command.
When the motherboard is replaced the Primary fruid container will likely match the Backup entries. If it does not you must enter escalation or service mode to fix it (if all three entries match this step is done).
Contact the TSC to request an escalation password (service mode will work also if just the copypsnc command ends up needing to be used, if the setpsnc command is needed escalation mode is required. setpsnc is not covered in this procedure).
Provide your TSC contact the output from the following ILOM commands- "version", "show /SYS product_serial_number", and "show /SP/clock".  If the product_serial_number information does not give good output then provide the showpsnc output that was seen in step b above as well.

The TSC will provide an escalation password that is made up of 32 short words.

Follow the example below to create a new user with the 'Service' role assigned. The Service role is required to access service or escalation modes.

In the following example we will create a user named 'escuser' with the service role.

Set the check_physical_presence to false and then exit from the ILOM so that you can login as the newly created user.

Login using the escuser login and enter escalation mode using the password that was provided by the TSC.

Use the showpsnc command to confirm the current container values. Confirm that one of the backup containers has a serial number (the value on the PSN line) that matches the system serial number.

The system serial number can be checked by comparing to the serial number RFID tag on the front left hand side of the server.

After confirming that there is a valid fruid backup use the copypsnc command to write the good data from the primary to the backup1 container on the MB.

The following example shows copying from primary to the backup1 but you could also copy from backup2 if needed.

[ NOTE: Example outputs below are from the underlying base platform (X5-2) server ]

At this point if all of the fruid containers match and have the correct serial number data this step is done.

If more than one of the fruid containers had non-valid entries then the copypsnc command should be used to copy over the valid data to the other container that is not valid. (ie. "copypsnc Primary Backup2" to copy primary to backup2)

After confirming all fruid data is correct reset the ILOM to confirm that the fruid data persists through a reboot and remove the escalation user if needed.

Login as the root user again and check the fruid data

If trouble is encountered during any of the steps of accessing escalation mode and fixing the fruid containers please contact the TSC for assistance.

11.  Check ILOM event log to see if any new errors are reported.

After Replacement of the Motherboard

To set the /SP/network after motherboard replacement

You will momentarily lose access to the SP, wait and reconnect

To boot the system in to BIOS

If you forget to set the boot_device prior starting the /SYS you can Control-E to access the BIOS menu while the system is performing a power-on self test

ZS5-2 BIOS settings

Open a terminal to the ILOM.

Reboot the system to enter BIOS.

Use the arrow keys to select “Exit” arrow down to “Restore Defaults”. Press Enter to “Load Optimized Defaults”.

A popup screen appears asking to approve loading optimal defaults. Press Enter.

In the event of field motherboard replacement or ILOM factory reset, the following options must be configured (in addition to network config).

Auto Power-On

Automatically powers on host after service processor cold boot

set /SP/policy HOST_AUTO_POWER_ON=“enabled”

Password Policy (new)

New ILOM now enforces password policy for password changes over IPMI interfaces.  AK synchronization of root passwords between host and SP could fail if host password does not comply to ILOM policy.

Set ILOM password policy to minimum:

set /SP/preferences/password_policy policy=1.

(Note: the ‘.’ after ‘1’ is significant)

Advanced

CPU Power Management Configuration

CPU Power Limit     [Disabled]

I/O

Add In Cards

Slot 1

IO Enable             [Enabled]

OpROM Enable     [Disabled]

Repeat for Slots 2, 3, 4, 5 & 6

Boot Menu

UEFI/BIOS Boot Mode    [Legacy]

Persistent Boot Mode     [Enabled]

Return to the main menu and arrow over to Save and Exit.

Resync SP Password with Host

1.  After the system reboots, press Esc then ( to go to the SP login.
2.  Enter > cd /SP/users/root
3.  Enter > set password            Enter the Host password or you can set a new password at this time.

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

Boot up system and verify full functionality.

REFERENCE INFORMATION:

Oracle ZS5-2 ZFS Storage Appliance Documentation - https://www.oracle.com/storage/nas/index.html

Oracle Integrated Lights Out Manager (ILOM) 3.2 Documentation : http://docs.oracle.com/cd/E37444_01/index.html

This solution has no attachment