How to Replace an PCA X3-2, X4-2, X5-2, X6-2 and X7-2 compute node motherboard

Asset ID:	1-71-1633452.1
Update Date:	2018-04-26
Keywords:

Solution Type Technical Instruction Sure

Solution 1633452.1 : How to Replace an PCA X3-2, X4-2, X5-2, X6-2 and X7-2 compute node motherboard

Applies to:

Oracle Virtual Compute Appliance X4-2 Hardware - Version All Versions to All Versions [Release All Releases]
Private Cloud Appliance - Version 1.0.1 and later
Private Cloud Appliance X6-2 Server Upgrade - Version All Versions and later
Oracle Virtual Compute Appliance X3-2 Hardware - Version All Versions to All Versions [Release All Releases]
Private Cloud Appliance X5-2 Hardware - Version All Versions and later
Information in this document applies to any platform.

Goal

How to Replace an PCA X3-2, X4-2, X5-2, X6-2, and X7-2 compute node motherboard

Solution

CAP PROBLEM OVERVIEW: MOTHERBOARD ASSEMBLY REPLACEMENT

NOTE: This CAP has been modified. The OVM shell CLI commands to clean up the db have been removed from this CAP and are hidden text over in (Doc ID 2281894.1)
If needed for reference, the original CAP is a MS Word document found below under the "Attachment" section.

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE ENGINEER NEED:
The engineer must be OVCA and/or PCA trained.
TIME ESTIMATE: 135 minutes
TASK COMPLEXITY: 4

FIELD ENGINEER INSTRUCTIONS

- PROBLEM OVERVIEW

This document points to existing MOS documentation containing the steps required to successfully replace a system board in all versions of PCA.

- WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?

If the system is still up and functioning, customer should perform an orderly and graceful shutdown of applications and OS and then power the node off.
If needed, see: Steps to Gracefully Shutdown and Power Off a Node in Oracle Private Cloud Appliance Prior to Maintenance (Doc ID 2256834.1)

The general steps are broken into three sections:

- Pre-motherboard FRU replacement steps. The customer is responsible for these steps:
[PCA] How to Prepare a Compute Node for Motherboard Replacement (Doc ID 2281894.1)

- Physical replacement of the motherboard using the appropriate CAP:
   How to Replace a Sun Server X3-2(X4170M3) Motherboard assembly (Doc ID 1495251.1)
   How to Replace a Sun Server X4-2 Motherboard assembly (Doc ID 1592250.1)
   How to Remove and Replace a Motherboard Assembly in an Oracle Server X5-2 and X6-2 (Doc ID 1992420.1)
   How to Replace an Oracle Server X7-2 Motherboard Assembly (Doc ID 2346485.1)

- Post-motherboard FRU replacement steps.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Pre-motherboard FRU replacement

The customer should prepare for replacement and shutdown the node. The customer facing docs are:
[PCA] How to Prepare a Compute Node for Motherboard Replacement (Doc ID 2281894.1)
Steps to Gracefully Shutdown and Power Off a Node in Oracle Private Cloud Appliance Prior to Maintenance (Doc ID 2256834.1)

Replace the motherboard

VERY IMPORTANT
Before you start the replacement steps, confirm with the customer they have completed all of the Pre-motherboard FRU replacement in the section above. If those steps are not done first, database corruption may result, and you will need to follow the 'How to Deprovision' steps listed below.

VERY IMPORTANT
Before you begin, please note it is extremely important that you do not connect the ethernet cable in the CMA cable bundle to the new motherboard until instructed. Doing so will likely corrupt the dhcp database with the wrong MAC address and you will need to follow the 'How to Deprovision' steps listed below.

VERY IMPORTANT
In the event there is a network link issue after motherboard replacement to not attempt to troubleshoot by temporarily swapping network cables around. The network ports are pre-wired to specific RU locations. If a node or laptop is plugged into the wrong switch port it will cause dhcp database corruption. If this happens, you will need to follow the 'How to Deprovision' steps below.
If you suspect an issue with the node network port, attempt laptop to node back-to-back testing using steps 6a and 6b for guidance to temporarily set a static IP address. If you can connect to the node this way, there should not be an issue with the node's net0 port. Assure setting are returned to expected values if this method is used.

IMPORTANT
If the dhcp database becomes corrupted from any situations listed above, you will need to work with the customer (and collab with EEST if needed) to clean everything up using this time consuming multi-step internal MOS doc: How to Deprovision a OVCA X3-2, OVCA X4-2, PCA X5-2 or PCA X6-2 Compute Node (Doc ID 2241032.1)

1. Replace the Motherboard using the appropriate CAP:
Replace a Sun Server X3-2(X4170M3) Motherboard assembly (Doc ID 1495251.1)
    Replace a Sun Server X4-2 Motherboard assembly (Doc ID 1592250.1)
    How to Remove and Replace a Motherboard Assembly in an Oracle Server X5-2 X6-2. (Doc ID 1992420.1)
    How to Replace an Oracle Server X7-2 Motherboard Assembly (Doc ID 2346485.1)

2. This completes the Motherboard replacement steps.

Post-motherboard FRU replacement steps

3. Using your laptop, connect a serial cable from your laptop to the SER MGT port on the node and connect a network cable from your laptop to the NET MGT port on the node.

4. Re-install the AC power cords but DO NOT CONNECT THE NETWORK CABLE yet.

5. After the SP boots, login and check that the firmware version is at the recommended revision.

To check:
a. -> show /System/BIOS system_bios_version
b. -> version

If needed, refer to this MOS notes for all PCA firmware versions:
Firmware Upgrade Policy for Private Cloud Appliance (PCA) (Doc ID 1610373.1)

6. If the firmware doesn’t match, you will need to manually upgrade it. One method is to temporarily configure the SP and your laptop so you can use the SP GUI to perform the FW upgrade:

a. On the SP:
      -> cd /SP/network
-> set pendingipaddress=192.168.4.253
      -> set pendingipnetmask=255.255.255.0
-> set pendingipdiscovery=static
      -> set commitpending=true
b. On your laptop configure the IP address to 192.168.4.254 and netmask 255.255.255.0.
c. Using your browser, browse to 192.168.4.253
d. Login to the SP root/changeme
e. On the left side of the screen, click on “ILOM Administration”
f. In the same area, click on “Maintenance”
g. On the “Firmware Upgrade” tab, click “Enter Upgrade Mode”
h. Click okay, then click on the “Browse…” button and supply the path to the firmware you obtained in step #5.
i. Complete the fw update.
j. After the SP has reboot, log in and continue with the next step.

7. It is recommended to run PC check to verify there are no issues after the MB has been replaced.
a. From the SP GUI, in the left pane, click on "Host Management"
b. Click on "Diagnostics"
c. Select "Run Diagnostics on Boot"
d. Reboot and run the diagnostics
e. After the PC-check runs without any issues, return to the diagnostics screen and set Run Diagnostics on Boot to "Disabled".

NOTE: DO NOT SKIP ANY OF THE STEPs IN STEP8. If you connect the ethernet cable when sideband is not enabled, the dhcp database will get corrupted with the wrong MAC address.
If that happened, see the IMPORTANT note at the top of the page for corrective actions.

8. Make the following changes to the SP:

a. Depending on the base rack mgmt nodes (or check another compute note) set the system identifier:
    -> set /SP system_identifier="Oracle Virtual Compute Appliance X3-2 <top level rack SN>"
    -> set /SP system_identifier="Oracle Virtual Compute Appliance X4-2 <top level rack SN>"
    -> set /SP system_identifier="Oracle Virtual Compute Appliance X5-2 <top level rack SN>"
b. Set root password
    -> set /SP/users/root password=Welcome1
c. Enable sideband management. These steps will not work if there is no network cable plugged into the NET MGT port. If successful, you will see “DHCP request in progress” after you type the commitpending command.
    -> cd /SP/network
    -> set pendingmanagementport=NET0
    -> set pendingipdiscovery=dhcp
    -> set commitpending=true
d. Confirm both "ipdiscovery" and "management port" are set as shown below before you continue.

-> show /SP/network

commitpending = (Cannot show property)
dhcp_clientid = none
dhcp_server_ip = 192.168.4.4
ipaddress = 0.0.0.0
ipdiscovery = dhcp <<<<<<<<<<<<<<<
ipgateway = 0.0.0.0
ipnetmask = 0.0.0.0
macaddress = 00:10:E0:3B:74:C3
managementport = NET0 <<<<<<<<<<<<<<<
outofbandmacaddress = 00:10:E0:3B:74:C2
pendingipaddress = 0.0.0.0
pendingipdiscovery = dhcp
pendingipgateway = 0.0.0.0
pendingipnetmask = 0.0.0.0
pendingmanagementport = NET0
sidebandmacaddress = 00:10:E0:3B:74:C3

9. Configure the node to boot into the BIOS, power cycle it, then start the serial console and check the following three BIOS settings: System Date/Time, Boot List, and OSA support.

a. set /HOST boot_device=bios
b. -> stop /SYS
    Are you sure you want to stop /SYS (y/n)? y
c. -> start /SYS
    Are you sure you want to start /SYS (y/n)? y
Starting /SYS
d. -> Start /SP/console
    Are you sure you want to start /SP/console (y/n)? y
    Serial console started. To stop, type ESC (

    You will see similar to the following

    Version 2.14.1219. Copyright (C) 2011 American Megatrends, Inc.
    BIOS Date: 12/07/2012 15:42:44 Ver: 17030300
    Press F2 to run Setup (CTRL+E on serial keyboard)
    Press F8 for BBS Popup (CTRL+P on serial keyboard)
    Press F12 for network boot (CTRL+N on serial keyboard)
    Selected Boot Mode = Legacy BIOS

    Wait for the node to boot into the BIOS setup screen.

e. On the BIOS Main screen, set the system date and time
f. On the BIOS Boot screen, make sure the “RAID:PCIE4…PCI RAID Adapter” is on the top of the boot list and the four PXE:NETx devices are after it.
g. For x3-2, x4-2, and x5-2 nodes: On the BIOS Boot/OSA Configuration screen, set “OSA Internal Support” to [Disabled]. x6-2 nodes ship from the factory with no OSA device installed.

Note: The OSA USB stick may or may not be in the chassis. It may have been removed as explained in
Upgrade: How to check the Oracle System Assistant (OSA) BIOS status on PCA nodes. (Doc ID 2281943.1)
The PCA software does not use the OSA device. It must be disabled if it in the chassis. The above MOS note explains why.

h. Select Save&Exit to exit from the BIOS setup.

10. After the BIOS has been saved, remove the power cables. Removing the power cables is done not only so the node will power off, but so the SP will reboot when you get to step 13.

11. Remove the laptop cables you connected to the nodes NET MGT and SER MGT ports.

NOTE: BEFORE YOU CONNECT THE ETHERNET CABLE IN STEP 12, MAKE SURE YOU DID NOT SKIP ANY STEPS IN STEP 8.

12. Reconnect all of the original network cables from the CMA. The two IB cables, and the one to NET0.

13. Reconnect the power cables.

14. Push the node back into the rack and wait for the SP to boot.

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

Return the node to service as documented in Doc ID 2256834.1 and verify functionality.

End

References

<NOTE:2241032.1> - How to Deprovision a OVCA X3-2, OVCA X4-2, PCA X5-2 or PCA X6-2 Compute Node

Attachments

This solution has no attachment