How to Replace an Exadata X7-2 Compute Node Server PCIe Riser Card

Asset ID:	1-71-2359358.1
Update Date:	2018-05-10
Keywords:

Solution Type Technical Instruction Sure

Solution 2359358.1 : How to Replace an Exadata X7-2 Compute Node Server PCIe Riser Card

Applies to:

Exadata X7-2 Hardware - Version All Versions to All Versions [Release All Releases]
Zero Data Loss Recovery Appliance X7 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Goal

How to Replace an Exadata X7-2 Compute Node Server PCIe Riser Card.

Solution

DISPATCH INSTRUCTIONS:

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?: Exadata X7-2 Training

TIME ESTIMATE: 60 minutes

TASK COMPLEXITY: 2

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

PROBLEM OVERVIEW: An Exadata X7-2 Compute Node PCIe Riser needs replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:

IMPORTANT NOTE TO TSC ENGINEER: CUT & PASTE the “CUSTOMER ACTIVITY” sections of the Pre-Replacement and Post-Replacement steps into a SR Note and ensure the customer is aware to do these steps prior to the scheduled field engineer activity, and during and after the replacement activity.

CUSTOMER ACTIVITY:

Shutdown of the database node is required prior to the part replacement:

If running Linux or Solaris native - follow Steps 1 to 9 of MOS Note:
How to shutdown the Exadata database nodes and storage cells in a rolling fashion so certain hardware tasks can be performed. (Doc ID 1539451.1)

If running OVM then follow MOS Note:
How to Shutdown and Startup Exadata database nodes running OVM (Doc ID 2367609.1)

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE?:

Prepare the Server for Service

The customer should have already prepared the server and powered it off. If not, provide them the instructions in the previous section.

1. Extend the server to the maintenance position
2. Disconnect the power cords from the power supplies.
3. Attach an anti-static wrist strap to your wrist and to a metal area on the chassis or the rack.
4. Remove the server top cover. Use a Torx T10 screwdriver to unlock the release button latch.

Caution - Ensure that all power is removed from the server before removing or installing the front indicator module. You must disconnect the power cables from the system before performing these procedures.

Caution - These procedures require that you handle components that are sensitive to electrostatic discharge. This sensitivity can cause the components to fail. To avoid damage, ensure that you follow anti-static practices as described in Electrostatic Discharge Safety.

Removing a PCIe Riser

Important: Oracle Server X7-2 Documentation outlines important PCIe Riser Location and Differences.
Exadata Compute Nodes have PCIe cards in the following slots:
Slot 1 - 10/25Gb SFP28 Dual-port Network Interface Card (Broadcom Whitney+) (DB nodes in 1/4 rack and larger configurations)
Slot 2 - 40Gb Dual-port CX3 InfiniBand Card
Slot 3 - Optional 10Gb Base-T Quad-port Copper TP Network Interface Card (DB nodes in 1/4 rack and larger configurations)
or Slot 3 - 10/25Gb SFP28 Dual-port Network Interface Card (Broadcom Whitney+) (DB nodes in Base rack configurations)
Slot 4 - 16-port 12Gb SAS3 RAID HBA

Removing a PCIe Riser From PCIe Slot 1 or 2

1. Select the PCIe riser that you want to remove.
2. If a PCIe card is installed in the riser, disconnect any external cables from the card.
3. Lift the green-tabbed latch on the rear of the server chassis next to the applicable PCIe slot to release the rear bracket on the PCIe card.

Note - If the riser does not have a PCIe card installed, then lift the latch to release the PCIe slot filler panel.

4. Lift the green-tabbed riser release lever on the PCIe riser with one hand and use your other hand to remove the riser from the motherboard connector.
5. Remove the PCIe card from the riser.
6. Place the riser on an anti-static mat.

Removing a PCIe Riser From PCIe Slots 3 and 4

Note - This PCIe riser is actually installed in PCIe slot 3, but it supports an M.2 mezzanine and up to two PCIe cards. The upper slot, referred to as slot 3, can be used for any supported PCIe card, and, therefore, is optionally populated. The lower slot, referred to as slot 4, is dedicated to the internal HBA card.

1. If there is a PCIe card installed in PCIe slot 3, disconnect any external or internal cables.

Note - Do not disconnect the SAS cables from the internal host bus adapter card in slot 4 until after you have removed the riser from the server.

2. Open the green-tabbed latch located on the rear of the server chassis next to PCIe slot 3 to release the rear bracket on the PCIe card.

Note - If the riser does not have a PCIe card installed in slot 3, then lift the latch to release the PCIe slot 3 filler panel.

3. To release the riser from the motherboard connector, lift the green-tabbed release lever on the PCIe riser to the open position.
4. Grasp the riser with both hands and remove it from the server.
5. Disconnect the M.2 SATA cable between the PCIe riser and motherboard SATA connector.
6. Disconnect the SAS cables from the internal HBA card installed in PCIe slot 4.
7. Disconnect the Super Capacitor cable from the internal HBA card in slot 4.
8. Remove the PCIe cards from the riser.
9. Place the riser on an anti-static mat.

Installing a PCIe Riser Into PCIe Slot 1 or 2

Note - PCIe slot 1 is nonfunctional in single-processor DB nodes in Base rack configurations.

Note - The PCIe riser in slots 3 and 4 is different than the PCIe risers in slots 1 and 2. Do not attempt to install a PCIe riser from slot 1 or 2 into PCIe slots 3 and 4 and vice versa.

1. Determine into which slot you will install the PCIe riser.
2. Install the PCIe card into the riser
3. Raise the green-tabbed release lever on the PCIe riser to the open (up) position.
4. Gently press the riser into the motherboard connector until it is seated, and then press the green-tabbed release lever on the PCIe riser to the closed (down) position.
5. Close the green-tabbed latch on the rear of the server chassis next to the applicable PCIe slot to secure the rear bracket on the PCIe card to the server chassis.

Note - If the riser does not have a PCIe card installed, install a PCIe slot filler panel and close the green-tabbed latch to secure the filler panel.

6. If there were any external cables connected to the PCIe card, reconnect them.

Installing a PCIe Riser Into PCIe Slot 3 or 4

Note - The PCIe riser in slots 3 and 4 is different than the PCIe risers in slots 1 and 2. Do not attempt to install a PCIe riser from slot 1 or 2 into PCIe slots 3 and 4 and vice versa.

1. Install the PCIe card(s) into the PCIe riser.
2. Reconnect the SAS cables and the super capacitor cable to the internal HBA card in slot 4.
To ensure SAS cables are properly connected to the internal HBA card
3. Reconnect the M.2 SATA cable between the PCIe riser and the motherboard SATA connector.
4. Raise the green-tabbed release lever on the PCIe riser to the open (up) position, and then gently press the riser into the motherboard connector until it is seated.
5. Ensure that the rear bracket on the internal HBA card in PCIe slot 4 is connected to the slot in the server chassis side wall.
If the bracket is not connected, remove the riser and reposition it so that the rear bracket connects to the side wall, and then gently press the riser into the motherboard connector.
6. Press the green-tabbed release lever on the PCIe riser to the closed (down) position.
7. To secure the rear bracket on the PCIe card to the server, close the green-tabbed latch on the rear of the server chassis.

Note - If the riser does not have a PCIe card installed in slot 3, install a PCIe slot filler panel and close the green-tabbed latch to secure the PCIe slot filler panel.

8. If there is a PCIe card in PCIe slot 3, reconnect any external cables.

Return the Server to Operation

1. Install the server top cover. Use a Torx T10 screwdriver to lock the release button latch.
2. Reconnect the power cords to the server power supply and connect any other cables to their original locations.
3. Return the server to the normal rack position.
4. Once the power cords have been re-attached and the ILOM has booted you will see a slow blink on the green LED for the server. Power on the
server by pressing the power button on the front of the unit.
5. Connect to the server console via the ILOM and monitor the boot.
By default the ILOM serial console displays the primary console output.
In the event of unexpected boot behavior, it is advisable to connect to both ILOM serial and ILOM graphics consoles at the same time and monitor.

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE?:

FIELD SERVICE ENGINEER and CUSTOMER ACTIVITY:

1. Verify all expected hardware is visible to the server and the fault is cleared. Assistance from the customer for server login access will be required.

2. Verify there are no outstanding faults in ILOM:

# ipmitool sunoem cli 'show faulty'
Connected. Use ^D to exit.
-> show faulty
Target | Property | Value
-------------------+-----------------------+-----------------------------------
-> Session closed
Disconnected
#

3. Verify there are no outstanding alerts in the Database Node:

# dbmcli -e list alerthistory

4. Re-enable and restart the Database services:

If running Linux or Solaris native - follow Steps 11 to 14 of MOS Note:
How to shutdown the Exadata database nodes and storage cells in a rolling fashion so certain hardware tasks can be performed. (Doc ID 1539451.1)

If running OVM then follow MOS Note:
How to Shutdown and Startup Exadata compute nodes running OVM (Doc ID 2367609.1)

PARTS NOTE:

7331100 [F] 2-Slot Riser Assembly (for slots 3 and 4)
7343599 [F] 1-Slot Riser Assembly (for slot 1 and slot 2)

REFERENCE INFORMATION:

Oracle Exadata Database Machine Maintenance Guide: https://docs.oracle.com/cd/E80920_01/DBMMN/maintaining-exadata-database-servers.htm#DBMMN22020

Oracle Server X7-2 Documentation http://docs.oracle.com/cd/E72435_01/index.html

How to shutdown the Exadata database nodes and storage cells in a rolling fashion so certain hardware tasks can be performed. (Doc ID 1539451.1)

How to Shutdown and Startup Exadata compute nodes running OVM (Doc ID 2367609.1)

Attachments

This solution has no attachment